Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere







                            Zoho Developer Community




                                                  • Desk Community Learning Series


                                                  • Digest


                                                  • Functions


                                                  • Meetups


                                                  • Kbase


                                                  • Resources


                                                  • Glossary


                                                  • Desk Marketplace


                                                  • MVP Corner


                                                  • Word of the Day


                                                  • Ask the Experts



                                                            • Sticky Posts

                                                            • Kaizen #217 - Actions APIs : Tasks

                                                              Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                            • Kaizen #216 - Actions APIs : Email Notifications

                                                              Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are
                                                            • Kaizen #152 - Client Script Support for the new Canvas Record Forms

                                                              Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
                                                            • Kaizen #142: How to Navigate to Another Page in Zoho CRM using Client Script

                                                              Hello everyone! Welcome back to another exciting Kaizen post. In this post, let us see how you can you navigate to different Pages using Client Script. In this Kaizen post, Need to Navigate to different Pages Client Script ZDKs related to navigation A.
                                                            • Kaizen #210 - Answering your Questions | Event Management System using ZDK CLI

                                                              Hello Everyone, Welcome back to yet another post in the Kaizen Series! As you already may know, for the Kaizen #200 milestone, we asked for your feedback and many of you suggested topics for us to discuss. We have been writing on these topics over the


                                                            Manage your brands on social media



                                                                  Zoho TeamInbox Resources



                                                                      Zoho CRM Plus Resources

                                                                        Zoho Books Resources


                                                                          Zoho Subscriptions Resources

                                                                            Zoho Projects Resources


                                                                              Zoho Sprints Resources


                                                                                Qntrl Resources


                                                                                  Zoho Creator Resources



                                                                                      Zoho CRM Resources

                                                                                      • CRM Community Learning Series

                                                                                        CRM Community Learning Series


                                                                                      • Kaizen

                                                                                        Kaizen

                                                                                      • Functions

                                                                                        Functions

                                                                                      • Meetups

                                                                                        Meetups

                                                                                      • Kbase

                                                                                        Kbase

                                                                                      • Resources

                                                                                        Resources

                                                                                      • Digest

                                                                                        Digest

                                                                                      • CRM Marketplace

                                                                                        CRM Marketplace

                                                                                      • MVP Corner

                                                                                        MVP Corner









                                                                                          Design. Discuss. Deliver.

                                                                                          Create visually engaging stories with Zoho Show.

                                                                                          Get Started Now


                                                                                            Zoho Show Resources

                                                                                              Zoho Writer

                                                                                              Get Started. Write Away!

                                                                                              Writer is a powerful online word processor, designed for collaborative work.

                                                                                                Zoho CRM コンテンツ



                                                                                                  Nederlandse Hulpbronnen


                                                                                                      ご検討中の方




                                                                                                              • Recent Topics

                                                                                                              • How can I convert an existing contact into a lead?

                                                                                                                I imported many contacts into ZOHO CRM. In order to select a few of them as leads, I want to mark or convert a few hundred as Leads. Who can I do that?
                                                                                                              • Introducing Dedicated Modules for Plans, Addons, and Coupons in Zoho Billing

                                                                                                                We’ve enhanced the way you manage Plans, Addons, and Coupons in Zoho Billing. Previously, all three grouped together under Subscription Items. Now, each one has its own dedicated module, giving you a cleaner and more intuitive experience. This update
                                                                                                              • Price book functionality enhancement

                                                                                                                A common use of price books is to create a price book for a given customer level or contract with a specific company. Given that this is done at a company/customer level it would be great to see a way to associate a price book to a given customer and
                                                                                                              • Contact data removes Account data when creating a quote

                                                                                                                Hi, Our customer has address fields in their quote layout which should be the address of the Account. They prefill the information, adding the account name - the address data is populated as per what is in the account - great. However when they then add
                                                                                                              • Can we have 24x7 live support for Zohobooks ?

                                                                                                                Can we have 24x7 live support for Zohobooks ?
                                                                                                              • Important update about notification security in Business Messaging SDK

                                                                                                                As part of our ongoing effort to strengthen the security of notification data, we have enhanced how notifications are processed and protected in our applications. These improvements ensure that all notification related data follows the most up-to-date
                                                                                                              • 年内最後のユーザー向けイベント:5名限定! 課題解決型ワークショップイベント Zoho ワークアウト開催のお知らせ (12/18)

                                                                                                                ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 12月開催のZoho ワークアウトについてお知らせします。 今回はZoomにて、オンライン開催します。 参加登録はこちら(無料) https://us02web.zoom.us/meeting/register/QHn6kJAcRs-znJ1l5jk0ww ━━━━━━━━━━━━━━━━━━━━━━━━ Zoho ワークアウトとは? Zoho ユーザー同士で交流しながら、サービスに関する疑問や不明点の解消を目的とした「Zoho ワークアウト」を開催します。
                                                                                                              • Sync Zoho Marketing Automation with Zoho Forms

                                                                                                                I can't find a section where the integration between Zoho Marketing Automation and Zoho form is explained. And how form subscription works with opt-in and consents.
                                                                                                              • What is the difference between Zoho Campaigns and marketing hub?

                                                                                                                Zoho Campaigns is a permission-based email marketing tool for sending marketing emails and mass emails. Marketing Automation is a multichannel marketing automation tool that caters to organizations looking for tools to engage their users across multiple
                                                                                                              • Webhook not firing.

                                                                                                                I created a webhook using the Web UI, it looks very nice and the testing worked without an issue, but when i save/ update a ticket, the webhook is not firing. Here are the details of the web-hook i get from using the API "modifiedTime": "2019-10-22T09:23:37.380Z",
                                                                                                              • Duplicate customers being created in Desk

                                                                                                                Hi I've trying to work out why I've getting duplicated customers being created in my desk. I have an external booking system that generates an email when I get a customer booking a job. A booking email gets sent to Desk where I manage the booking and
                                                                                                              • Creating a custom CSV file using deluge script/

                                                                                                                I have an application I have developed and the client wants us to place an export file in csv onto an ftp server daily. Now I don't see au options in creator to change the separator to anything else. The client wants the separator to be the pipe symbol "|"  I think i would be able to create schedule with some code to create the appropriate data in a string using deluge script but I haven't seen any functionality that would allow me to deposit that data as a file anywhere or attach it to an email
                                                                                                              • Can we add custom fields to portal community profiles?

                                                                                                                How do we add custom fields to our profile pages in our portal community? If we have the ability to add custom fields, will we be able to access those fields via API? We want to use our Desk community in our help portal as our primary community and would
                                                                                                              • Customer ticket creation via Microsoft Teams

                                                                                                                Hi all, I'm looking to see if someone could point me in the right direction. I'd love to make it so my customers/ end users can make tickets, see responses and respond within microsoft teams. As Admin and an Agent i've installed the zoho assist app within
                                                                                                              • E-Invoicing in Belgium with Zoho Billing

                                                                                                                Starting January 1, 2026, Belgium is introducing mandatory electronic invoices (e-invoicing) for all B2B transactions between VAT-registered businesses. Invoices and credits notes must be exchanged in a prescribed digital format. How E-Invoicing works
                                                                                                              • delayed: host mx2.zoho.com (136.143.183.44): Network error: Unexpectedly disconnected (STARTTLS)

                                                                                                                Hi Team, I'm not receiving any email. Seems to be an issue. Sender sent me the error message they received: "delayed: host mx2.zoho.com (136.143.183.44): Network error: Unexpectedly disconnected (STARTTLS)" Please advise.
                                                                                                              • Interview booked through Invite but no Notifications

                                                                                                                We have a workflow that was developed through a developer/partner that was tested and worked. Today, we pushed a candidate through the process and invited them to an in-office interview. They were sent the booking link (as usual and as tested before successfully)
                                                                                                              • Additional Address - Company Name

                                                                                                                It would be very helpful to have a "Company Name" field in the Additional Address shipping section.  In a situation where you drop-ship orders to a different address or different company entirely, it may be necessary to list the company name of the receiver. I understand that the Attention field can be used for that purpose, but that's not really the intended purpose of that field.
                                                                                                              • Incorrect Email Notifications for Product Reviews

                                                                                                                Dear Zoho Commerce Support Team, I am writing to report a technical issue that occurs frequently on our platform. Problem Description: We regularly receive email notifications informing us of new product reviews awaiting approval. However, when we access
                                                                                                              • In App Auto Refresh/Update Features

                                                                                                                Hi,    I am trying to use Zoho Creator for Restaurant management. While using the android apps, I reliased the apps would not auto refresh if there is new entries i.e new kitchen order ticket (KOT) from other users.   The apps does received notification but would not auto refresh, users required to refresh the apps manually in order to see the new KOT in the apps.    I am wondering why this features is not implemented? Or is this feature being considered to be implemented in the future? With the
                                                                                                              • How to Rank Tables by Row Count in Descending Order

                                                                                                                I am trying to understand the consume of lines that grow up so fast in the last week. Is there any way to create a pivot table or query to get TABLE NAME LINES in descending order?
                                                                                                              • Can we do Image swatches for color variants?

                                                                                                                We want to do something like the attached screenshot on our new zoho store. We need image swatches instead of normal text selection. We want to user to select an image as color option. Is this doable? I don't see any option on zoho backend. Please h
                                                                                                              • annualy customer report

                                                                                                                we need a report per customer that looks like this invoic number cleaning laundry repair management 01 january xxx euro xx euro xx euro xxx euro 02 february xxx euro xxx euro x euro xxxx euro and so on the years 12 months is that possible to make and
                                                                                                              • Export your notes from Notebook!

                                                                                                                Dear users, The long awaited feature is now live. Yes, you can now export your notes from Notebook app in bulk. But the feature has just started with web app alone for now. You can try the export feature as mentioned below: Go to our web app, https://notebook.zoho.com Go to 'Settings' > 'Export' Now, select the format: You can select either ZNote or HTML Once done, you can use the same to import or can have this a local backup of your notes. Note: Export for other platforms are in development and
                                                                                                              • our customers have difficult to understand the Statements

                                                                                                                our costumers have big problems to understand Zohos Statements. we need a text box after the payment number to explain what the payments are for. Is it possible to develop a version of the Statement with that kind of box and if so whu can do it
                                                                                                              • Canva Integration

                                                                                                                Hello! As many marketing departments are streamlining their teams, many have begun utilizing Canva for all design mockups and approvals prior to its integration into Marketing automation software. While Zoho Social has this integration already accomplished,
                                                                                                              • Transfer between two customers (Peters Rental account to Peters Private account)

                                                                                                                we are a Property Management company. Our customers have to accounts (registered as two customers - Peter Rental and Peter Private On the rental account all income and costs fron rental activities are noted. On the private account all private are noted
                                                                                                              • Prepopulating Fields

                                                                                                                Hello, I have a form (Assets) with 2 lookup fields: Client (from Clients) Site (from Client Sites) I modified the code (highlighted in red below), so the Site dropdown shows the list of sites related to the Client. must have Client_Site ( type = picklist
                                                                                                              • CRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive

                                                                                                                Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage files
                                                                                                              • REPORT THAT SHOWS SOH FOR ITEMS THAT HAVE ZERO SALES

                                                                                                                When we started Zoho we imported a lot of inventory lines directly off our suppliers price list and have never ordered / sold. I want to clean up our data base and remove all these lines. What reeport will show me what lines have zero sales as well as
                                                                                                              • Automation series : Close all tasks once the project is completed

                                                                                                                When a project is marked as Completed, it might still have open tasks such as recurring tasks that were not marked as complete, tasks that are no longer relevant, or tasks that no longer need attention after closure. To ensure the project reflects its
                                                                                                              • Direct Feed (Bank)

                                                                                                                Is Direct feed integration for AlRajhi and ADCB bank supported by Zoho Books in GCC/Saudi
                                                                                                              • Converting Sales Order to Purchase Order

                                                                                                                Hi All, Firstly, this code works to convert a sales order(SO) to a purchase order (PO) via a button, however I am running into an issue when I convert the SO where the values from the line items are not pulled across from the SO to the PO. The ones in
                                                                                                              • All new Address Field in Zoho CRM: maintain structured and accurate address inputs

                                                                                                                The address field will be available exclusively for IN DC users. We'll keep you updated on the DC-specific rollout soon. It's currently available for all new sign-ups and for existing Zoho CRM orgs which are in the Professional edition. Latest update
                                                                                                              • Horrible Connectivity!

                                                                                                                I have used Meetings several times, but most of the time the connection is horrible. The video freezes, the audio freezes, and we end up cancelling the meeting. I am on a high speed internet connection, and Zoom works fine, so I know it's not a problem
                                                                                                              • Is it really true that I can't set the default 'deposit to' account in 2025?

                                                                                                                I've been using Books for 7 years and the default account has never been a problem. I usually manually reconcile invoices and have never had a thought about which account. It has always been my account. However, I recently noticed that for the past 4
                                                                                                              • Zoho DataPrep switching Date Format

                                                                                                                When using a pipeline that is importing Zoho Analytics data into Zoho DataPrep, the month and day of date fields are switched for some columns. For example, a Zoho Analytics record of "Nov. 8, 2025" will appear in Zoho DataPrep as "2025/08/11" in "yyyy/MM/dd"
                                                                                                              • Using Another Field Value for Workflow Field Update

                                                                                                                I'm trying to setup a Workflow with a "Field Update" action on the Lead module, but I would like the new value to actually be taken from a DIFFERENT Field's on the Lead record (vs just defining some static value..) Is this possible? Could I simply use
                                                                                                              • Tax information

                                                                                                                Hello, I need help/guidance on how to add my organization's Tax/VAT information. Thank you Pavly
                                                                                                              • Build smarter Guided Conversations with Fork Blocks

                                                                                                                When your customers arrive on your support channel, they're not there to explore. They are usually confused and stuck while trying to fix something important. We understand how stressful that moment can feel and we want your bot to make things easier,
                                                                                                              • Next Page