Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.






                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • How to list emails in a folder, e.g. Inbox, on multiple pages when using Zoho mail webpage?

                                                                                                                  Something as shown in the figure. There are totally 50 emails in Sent folder. If "Mail per page" equals 20, then the Sent folder is split into 3 pages. When I wander through Sent folder, I can just select a specific page to jump to. BTW, it seems that
                                                                                                                • Zoho Calendar soft bounce on @hotmail.com and @yahoo.com email addresses

                                                                                                                  Hello, our Zoho calendar recently does not send the calendar invites to emails with hotmail and yahoo domains and comes back with a "soft bounce". other domains like Gmail works fine. Also sending "email" to the same emails to the above domains work well
                                                                                                                • ERROR CODE :512 - 5.4.4 DNS error:NXDOMAIN.

                                                                                                                  Suddenly we cant send mail, we are getting this error for all outbound mail to multiple domains.
                                                                                                                • Unveiling Cadences: Redefining CRM interactions with automated sequential follow-ups

                                                                                                                  Last modified on 01/04/2024: Cadences is now available for all Zoho CRM users in all data centres (DCs). Note that it was previously an early access feature, available only upon request, and was also known as Cadences Studio. As of April 1, 2024, it's
                                                                                                                • customer data security

                                                                                                                  We are exploring ways to enhance our within Zoho CRM. Our Goal: We want to fully integrate RingCentral with Zoho CRM to enable click-to-call functionality for our sales team. However, to comply with data privacy regulations and protect customer contact
                                                                                                                • Sync CRM inventory data with Zoho Books

                                                                                                                  I just switched everything over to ZoHo books, but I am trying to find out why the CRM Estimates, Invoices, and Sales Orders created in ZoHo CRM are not then duplicated in ZoHo Books? I had Quickbooks before, and had to do everything twice, I thought
                                                                                                                • mask Customer phone number and agents cant see customer phone number

                                                                                                                  Is there any way we can integrate Zoom Phone with Zoho CRM while ensuring that customer phone numbers remain masked? We need a solution where agents can make outbound calls but cannot see customer phone numbers. Please let us know if there is any solution
                                                                                                                • Email Reminders on Shared Calendars

                                                                                                                  How do we turn off the setting that emails reminders to everyone who has accepted or declined a calendar invite? If 8 of us have been invited to the same meeting, we receive 8 notifications for every step of the process, from invitation to decision.
                                                                                                                • How to refresh a ticket view ?

                                                                                                                  I am doing a widget where I send a rest api call to make a new draft to the ticket I am viewing. The issue is sometimes it refresh a ticket view and I can see inserted draft right away, but sometimes I do not see it even if it is inserted correctly and
                                                                                                                • Ugh! - Text Box (Single Line) Not Enough - Text Box (Multi-line) Unavailable in PDF!

                                                                                                                  I provide services, I do not sell items. In each estimate I send I provide a customized job description. A two or three sentence summary of the job to be performed. I need to be able to include this job description on each estimate I send as it's a critical
                                                                                                                • Supervisor Rules - Zoho Desk

                                                                                                                  Hi, I have set up a Supervisor Rule in Zoho Desk to send an email alert when a ticket has been on hold for 48 hours. Is there a way to change it so that the alert only sends once and not on an hourly basis? Thank you Laura
                                                                                                                • ResponseCode 421, 4.7.0 [TSS04] Messages from 136.143.188.51 temporarily deferred due to user complaints

                                                                                                                  Had email bounce. Let me know if you can fix this. Thanks. Michael
                                                                                                                • New Zoho triggers Google Dangerous flag due toabnormal charcters

                                                                                                                  Just signed up and doing my first email test. I sent it to my google email account but it got flagged as Dangerous" due abnormal characters. My DNS setup looks ok. Page snips attached Help Please Thanks, Rick DC PowerWorld
                                                                                                                • Is there a API to fetch tasks in a Board/Section

                                                                                                                  I am writing a scheduled function that retrieves all the tasks and send an reminder on cliq. I cannot seem to find a API to fetch tasks (by user / board / section) What are the way to fetch tasks?
                                                                                                                • Having trouble fetching contents of Zoho Connect Feeds using the API, requesting alternative API documentation.

                                                                                                                  I'm trying to retrieve feed/post data from Zoho Connect using the API but facing challenges with the current documentation. What I've tried: OAuth authentication is working correctly (getting 200 OK responses) Tested multiple endpoints: /pulse/nativeapi/v2/feeds,
                                                                                                                • Adding an Account Name to Tasks/Reminders

                                                                                                                  Does anyone know how to add the related account name to a task?  When we look at the list of activities and when the reminders pop up, there is no way of quickly seeing who the account is. 
                                                                                                                • Triggering Zoho Flow on Workdrive File Label

                                                                                                                  Right now Im trying to have a zoho flow trigger on the labeling/classification of a file in a folder. Looking at the trigger options they arent great for something like this. File event occurred is probably the most applicable, but the events it has arent
                                                                                                                • SendMail to multiple recipients

                                                                                                                  Hi, I'm trying to send an email to a list of recipients.  Right now the "to" field is directed to a string variable. (List variables won't work here). In the string variable, how can I make it work? trying "user@app.com;user2@app.com" or "user@app.com; user2@app.com" just failed to send the emails. Ravid
                                                                                                                • Populate drop down field from another form's subform

                                                                                                                  Hello, I found how to do that, but not in case of a subform. I have a Product form that has a subform for unit and prices. A product might have more than one unit. For example, the product "Brocoli" can be sold in unit at 3$ or in box of 10 at 25 $. Both
                                                                                                                • Usar o Inventory ou módulo customizado no CRM para Gestão de Estoque ?

                                                                                                                  Minha maior dor hoje em usar o zoho é a gestão do meu estoque. Sou uma empresa de varejo e essa gestão é fundamental pra mim. Obviamente preciso que esse estoque seja visível no CRM, Inicialmente fiz através de módulos personalizados no próprio Zoho CRM,
                                                                                                                • Signup forms behaviour : Same email & multiple submissions

                                                                                                                  My use case is that I have a signup form (FormA) that I use in several places on my website, with a hidden field so I can see where the contact has been made from. I also have a couple of other signup forms (FormB and FormC) that slight differences. All
                                                                                                                • getting error in project users api

                                                                                                                  Hello, I'm getting a "Given URL is wrong" error when trying to use the Zoho Projects V3 API endpoint for adding users to a project. The URL I'm using is https://projectsapi.zoho.com/api/v3/portal/{portalid}/projects/{projectid}/projectusers/ and it's
                                                                                                                • Change total display format in weekly time logs

                                                                                                                  Hi! Would it be possible to display the total of the value entered in the weekly time log in the same format that the user input? This could be an option in the general settings -> display daily timesheet total in XX.XX format or XX:XX.
                                                                                                                • How to display historical ticket information of the total time spent in each status

                                                                                                                  Hi All, Hoping someone can help me, as I am new to Zoho Analytics, and I am a little stuck. I am looking to create a bar chart that looks back over tickets raised in the previous month and displays how much time was spent in each status (With Customer,
                                                                                                                • Zoho Community Weekend Maintenance: 13–15 Sep 2025

                                                                                                                  Hi everyone, We wanted to give you a heads-up that Zoho Community will undergo scheduled maintenance this weekend. During this period, some community features will be temporarily unavailable, while others will be in read-only mode. Maintenance Window:
                                                                                                                • Agent Performance Report

                                                                                                                  From data to decisions: A deep dive into ticketing system reports An agent performance report in a ticketing system provides a comprehensive view of how support agents manage customer tickets. It measures efficiency and quality by tracking key performance
                                                                                                                • Pourquoi dans zohobooks version gratuite on ne peut ajouter notre stock d'ouverture??

                                                                                                                  Pourquoi dans zohobooks version gratuite on ne peut ajouter notre stock d'ouverture ??
                                                                                                                • How can I adjust column width in Zoho Books?

                                                                                                                  One issue I keep running into is as I show or hide columns in reports, the column widths get weird. Some columns have text cut off while others can take a fourth of the page for just a few characters. I checked report layout guides and my settings, but
                                                                                                                • Invalid value passed for file_name

                                                                                                                  System generated file name does not send file anymore - what is the problem?
                                                                                                                • Custom Function for Estimates

                                                                                                                  Hey everyone, I was wondering if there was a way to automate the Subject of an estimate whenever one is created or edited: * the green box using following infos: * Customer Name and Estimate Date. My Goal is to change the Subject to have this format "<MyFirm>-Estimate
                                                                                                                • Search in module lists has detiorated

                                                                                                                  Every module has a problem with the search function :-/
                                                                                                                • YouTube Live #1: AI-powered agreement management with Zia and Zoho Sign

                                                                                                                  Hi there! We're excited to announce Zoho Sign’s first YouTube live series, where you can catch the latest updates and interact with our Zoho Sign experts, pose questions, and discover lesser-known features. We're starting off by riding the AI wave in
                                                                                                                • Search in module lists has detiorated

                                                                                                                  Every module has a problem with the search function :-/
                                                                                                                • Bigin: filter Contacts by Company fields

                                                                                                                  Hello, I was wondering if there's a way to filter the contacts based on a field belonging to their company. I.e.: - filter contacts by Company Annual Revenue field - filter contacts by Company Employee No. field In case this is not possibile, what workaround
                                                                                                                • Has Zoho changed the way it searches Items?

                                                                                                                  Right now all of our searches have broken and we can no longer search using the SKU or alias. It was fine last night and we came in this morning to broken.....this is impacting our operations now.
                                                                                                                • Refunds do not export from Shopify, Amazon and Esty to Zoho. And then do not go from Zoho inventory to Quickbooks.

                                                                                                                  I have a huge hole in my accounts from refunds and the lack of synchronisation between shopify , Amazon and Etsy to Zoho ( i.e when I process a refund on shopify/ Amazon or Etsy it does not come through to Zoho) and then if I process a manual credit note/
                                                                                                                • Zoho Calendar not working since a few days

                                                                                                                  Hey there, first off a minor thing, since I just tried to enable the Calendar after reading this in another topic (there was no setting for this though) : For some reason my current session is showing me based in New York - I'm in Germany, not using a
                                                                                                                • Monthly timesheet, consolidation of time by project

                                                                                                                  I have time logs for various jobs for project. Is it possible to consolidate the time spent for each job, when I am generating a timesheet for a month? I am getting the entries of jobs done on each day when I generate a timesheet for a month For example
                                                                                                                • Sending emails from an outlook account

                                                                                                                  Hi, I need to know if it's possible to send automatic emails from an Outlook account configured in Zoho CRM and, if so, how I can accomplish that. To give you some context, I set up a domain and created a function that generates PDF files to be sent later
                                                                                                                • Struggling with stock management in Zoho CRM – is Zoho Inventory the solution?

                                                                                                                  My biggest pain point today with Zoho is inventory management. I run a retail business and reliable stock management is absolutely critical. Obviously, I need this inventory to be visible inside the CRM. At first, I tried handling it through custom modules
                                                                                                                • Next Page