Kaizen 123- Data Synchronization from third party application

Kaizen 123- Data Synchronization from third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

          Zoho Developer Community




                                    Zoho Desk Resources

                                    • Desk Community Learning Series


                                    • Digest


                                    • Functions


                                    • Meetups


                                    • Kbase


                                    • Resources


                                    • Glossary


                                    • Desk Marketplace


                                    • MVP Corner


                                    • Word of the Day



                                        Zoho Marketing Automation


                                                Manage your brands on social media



                                                      Zoho TeamInbox Resources

                                                        Zoho DataPrep Resources



                                                          Zoho CRM Plus Resources

                                                            Zoho Books Resources


                                                              Zoho Subscriptions Resources

                                                                Zoho Projects Resources


                                                                  Zoho Sprints Resources


                                                                    Qntrl Resources


                                                                      Zoho Creator Resources



                                                                          Zoho Campaigns Resources


                                                                            Zoho CRM Resources

                                                                            • CRM Community Learning Series

                                                                              CRM Community Learning Series


                                                                            • Kaizen

                                                                              Kaizen

                                                                            • Functions

                                                                              Functions

                                                                            • Meetups

                                                                              Meetups

                                                                            • Kbase

                                                                              Kbase

                                                                            • Resources

                                                                              Resources

                                                                            • Digest

                                                                              Digest

                                                                            • CRM Marketplace

                                                                              CRM Marketplace

                                                                            • MVP Corner

                                                                              MVP Corner





                                                                                Design. Discuss. Deliver.

                                                                                Create visually engaging stories with Zoho Show.

                                                                                Get Started Now


                                                                                  Zoho Show Resources


                                                                                    Zoho Writer Writer

                                                                                    Get Started. Write Away!

                                                                                    Writer is a powerful online word processor, designed for collaborative work.

                                                                                      Zoho CRM コンテンツ






                                                                                        Nederlandse Hulpbronnen


                                                                                            ご検討中の方





                                                                                                  • Recent Topics

                                                                                                  • Blueprint - 'On hold' state with an automatic transition?

                                                                                                    I think I'm missing something here so I'm hoping if I explain what I'm trying to achieve someone might be able to give me a way around it. We sometimes get request far in advance but we don't want to action them unless it is 7 days from when they are
                                                                                                  • HOW TO VIEW INDIVIDUAL COST OF NEWLY PURCHASED GOODS AFTER ALLOCATING LANDED COSTS

                                                                                                    Hello, I have been able to allocate landed costs to the purchase cost of the new products. however, what i need to see now is the actual cost price (original cost plus landed cost), of only my newly purchased products to enable me set a selling price
                                                                                                  • Client Script: $Client.refresh({ triggerOnLoad: true }); not triggering onLoad Client Scripts

                                                                                                    Hey friends! I'm trying to store a temporary var, refresh the page for the user, then check that temporary var and do some actions. Theoretically using the title's code: $Client.refresh({ triggerOnLoad: true }); should refresh the page and trigger on
                                                                                                  • Calculate months and years between 2 dates on subform

                                                                                                    I am looking for a function syntax for an employment candidate to calculate the number of years and months (decimal format. eg 1.2 years) they are employed. I have their start date entered, but if the end date is blank, that tells me they are still employed
                                                                                                  • Combine related grouping values into categories in CRM analytical components

                                                                                                    Hello everyone, Analyzing large datasets can be challenging when dealing with numerous individual data points. It's often difficult to extract meaningful insights when information is scattered and ungrouped. To address this, we're adding options to create
                                                                                                  • How Kiosk Studio can simplify sales for bank employees | Kiosk Studio Session #4

                                                                                                    Hello everyone, Banks can boost revenue by cross-selling to their current customers. For example, they can sell credit cards, personal loans, and more to existing account holders. To do this, bank employees move all around the CRM, open and close records,
                                                                                                  • Tip #19 - Create checkbox tracker in your spreadsheet

                                                                                                    Hello Sheet users! We are here with yet another tip to help you get the most out of Zoho Sheet. Spreadsheets can be used to handle a variety of tasks, but ever tried using checkboxes to track the progress of your action items dynamically? Here's a sample
                                                                                                  • Integrate Oracle with ZOHO CRM

                                                                                                                         Please let me know if its possible to Integrate Oracle with ZOHO CRM. If yes then help me in doing it. 
                                                                                                  • UUIDs

                                                                                                    Has anyone coded a Universal Unique Identifier (UUID) generator in Deluge?
                                                                                                  • Introducing signer groups: Streamline signature collection and make it even faster

                                                                                                    Hello everyone, We're excited to introduce signer groups, a feature designed to make your signing process quicker, more efficient, and collaborative. With signer groups, you can send an envelope to a group of people, and any member of the group can open
                                                                                                  • Calendar - Recurring Event - End of Month and Last Weekday of Month

                                                                                                    How do I set a calendar event to recur on the last day of every month? How do I set a calendar event to recur on the last weekday of the month?
                                                                                                  • Duplicate Zoho Invoices and Sales Receipts

                                                                                                    We have been running into an issue where upon saving an invoice or a sales receipt, we get a duplicate: same information saved twice but with a different invoice id/number and sales receipt id/number. I have logged a ticket but so far no response. It
                                                                                                  • Introducing revamped Zoho Creator Developer Console—a powerful platform for developing and distributing apps

                                                                                                    Hello everyone! We're thrilled to announce the launch of the revamped Zoho Creator Developer Console—a dedicated platform designed specifically for Creator developers and Partners to build, test, and distribute apps to your clients. Developer Console
                                                                                                  • Phone App of CRM doesn't find contacts

                                                                                                    I open the crm to and do a contact search but nothing comes up. If I do a full zia search it will find an old note that is attached to a contact and I can open the contact. BUT I still can't enter notes of calls, set appointments, etc. It requests I go
                                                                                                  • [Product Update] Deprecation of 'Inbound Transport Details' module in Zoho Analytics - Amazon Seller Central integration

                                                                                                    Dear Customers, Please be informed that the Amazon Seller Central will be deprecating the 'Inbound Transport Details' module APIs effective December 20, 2024. As a result, this module will no longer be supported in the Amazon Seller Central - Zoho Analytics
                                                                                                  • Is there any support for Drivers license and other forms of ID scanning capabilities?

                                                                                                    When scanning a drivers license barcode for data input to a Form, the scanning tool receives all the raw data but there doesnt seem to be a function to limit or remove the unnecessary fragmrnts, like a prefix. Is there any support or info in the help
                                                                                                  • Exciting Update: Multi WhatsApp Business Account (WABA) Support Now Available in SalesIQ!

                                                                                                    We’re pleased to share an important update that will enhance the way you manage your WhatsApp Business accounts (WABAs) within SalesIQ. With the launch of Multi WABA support, you can now connect and manage multiple brands more effectively, each under
                                                                                                  • Gravity Forms plugin not passing some fields

                                                                                                    I use the gravity form zoho plugin to push data from my lead form into my lead page in Zoho CRM. Everything was working file for about 6 months. Suddenly on Oct 1st, some of the fields are no longer getting passed to Zoho. The fields with the problem
                                                                                                  • Can't delete bank transactions (i changed from 14 days trail to free just now)

                                                                                                    Hi, I manually added one bank transaction When i try to delete it, it say below: What should I do?
                                                                                                  • Feature Request: Search in the PC client. Some thoughts about the search.

                                                                                                    Hi all. I'm really excited to start using Zoho Notebook, but I'm missing some of the search capabilities on my desktop. There are also some thoughts on improving search in general. Search is very important to me, without it it is difficult for me to use
                                                                                                  • Leave Policy for Brazil

                                                                                                    Hi, Brazil asked us to configure Zoho People to apply the following policy: · To block starting vacations 2 business days before holidays or weekends; Employees cannot start their vacations 2 days before holidays or weekends. Example: If December 25th
                                                                                                  • Adding date field to each individual Items when creating Invoices?

                                                                                                    When adding items into an invoice I need to be able to have a date for each item. Example below: Date Item Description Qty Rate Discount(%) Tax Amount 31/07/13 Brown Sugar Performance 1.00 3,000.00 0.00 10% $3300 Is this possible or will it be in the
                                                                                                  • Auto-Create OneDrive Folder Structure Upon Lead Creation

                                                                                                    Hello, New to Zoho and looking for help on a critical process automation I'm looking to implement. My company currently utilizes OneDrive for file management and the folder structure is Proposals -> Client Name -> Address (where I need to initially create
                                                                                                  • Profile Page View Customization

                                                                                                    I need to change the fields, sections from the profile view of an emplyoyee.
                                                                                                  • What do the Image Sizes mean in Zoho CRM Email Templates?

                                                                                                    Below you can see the image options in email templates: Banner, Fit, Small, Medium, Original, Custom. Can someone from Zoho please share with me: What each is/means? How each will look on desktop AND mobile? How to edit "custom"?  If I choose "Custom"
                                                                                                  • Translate any published presentation to the language of your choice

                                                                                                    As part of our constant effort to enrich your presentation experience, Zoho Show has rolled out a new feature for translating published presentations. Consider the following scenario: Zylker IT services, a multinational corporation, has announced product
                                                                                                  • Blueprint: multi-select lookup field not available in the criteria option

                                                                                                    I read this old forum post which stated that multi-select lookup fields are now selectable as an option in a Blueprint transition configuration: https://help.zoho.com/portal/en/community/topic/blueprint-multi-select-lookup-field-not-available-within-blueprint-transition
                                                                                                  • Apply Credit Note Automatically

                                                                                                    We need the ability to apply open Credit Notes toward invoices generated by recurring invoice as the first, priority payment. This should be an option that we can enable/disable in the recurring profile and/or the Credit Note. Other invoicing systems can do this. I'm not sure why Zoho Invoice doesn't have it. Here's an example for a Recurring Invoice... If a customer has open Credit Notes, and a saved credit card set for auto-pay, once an invoice goes out, the credit balance gets automatically applied
                                                                                                  • Workflow for "Expenses" module?

                                                                                                    Hi there, over the last 2 years, Zoho Expense has seen tremendous growth and we are happy with it. But, sometimes it is frustrating to see things are being implemented halfheartedly, or so it seems. For example, There is the possibility to create workflows
                                                                                                  • Record Template - Conditionally printing sections

                                                                                                    Is there a way to conditionally print a section of a Record Template? More specifically I am printing records from a Form "Invoice". That Form has 3 SubForms. I'd like to print the content of those SubForms using a Record Template but only if they have at least one line item. If they have no line items, I'd like to hide the headers for that section on the printout.
                                                                                                  • User can choose the PDF report

                                                                                                    Hi. I would like to find out if a user (Creator or customer portal) to choose from the different PDF customised reports that have been built?
                                                                                                  • Query table pull last 12 months

                                                                                                    I am tying to pull the following criteria and the date is always what causes me the issue. I want to pull people (pco_id) who have entries of "event_id" being these 2 events and whos "kind" is Regular or Guest and where the event_starts_at (date column)
                                                                                                  • PLEASE FIX Search options and consider a Global seach option

                                                                                                    A recent update has removed the ability to search for addresses and phone numbers under contacts. We cannot find where this moved to (If it is still available). Please put these options back as we cannot locate specific projects anymore. Also please consider allowing for a Global search. This would really improve the search engine. For example: If I search for "Sally Jones" then all invoices , estimates, vendors etc.. would populate.. Please let me know if you need any more information. Thank You....
                                                                                                  • Customize your calendar based on personal preferences

                                                                                                    Greetings, We're happy to introduce a few new capabilities to the Activities module's Calendar View! Now you can tailor your calendar's appearance and notification settings to suit your needs. In the past, the Calendar View lacked customization options
                                                                                                  • BUG ALERT: Client Script + Commands -> $Page contextual data is not updated

                                                                                                    When using the new Client Script Commands feature, there is an issue with the Client Script $Page contextual data not accurately being updated each time a Command is run. Assuming a Client Script Command called "Client Script Command Bug" with the following
                                                                                                  • Show iFrame of related List inside of Blueprint Transition

                                                                                                    Hey, is it possible to show an iFrame of a related list like this inside of a Blueprint transition?
                                                                                                  • Lookup Fields not Converting

                                                                                                    I manage holiday properties. I have a lookup to the Accounts (Properties) in the Leads module. The lookup is connected to the property address field. When I convert it the lookup field does not update in Deals, although the property address does. There
                                                                                                  • 2024: A Year of Transformation with Zoho Forms

                                                                                                    As we close the curtain on another exciting year, it’s time to reflect on the strides Zoho Forms has taken in 2024. From empowering businesses with advanced tools to simplifying workflows and enhancing user experiences, our updates this year were all
                                                                                                  • Stop selling out of stock Items.

                                                                                                    Hi I have been using Zohobooks for a around 8 month now. I am not involved in selling process but my staff cant stop selling product which they do not hold in stock, this is a big headache for me as physical count never matches what is shown on the books. 
                                                                                                  • Bigin API Token Request ("invalid_client")

                                                                                                    Hi people, I tried to connect to the API without success, I've read all of the documentation multiple time and tried just about everything. I tried to do it with Python Request module and with Postman, passing the information through both the URL parameter
                                                                                                  • Next Page