Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Microsoft Phone Link

                                                                                                                  Does anyone know if you can use Microsoft Phone Link to make calls through Zoho?
                                                                                                                • Voip Phone system that integrates with Zoho

                                                                                                                  Just checking to see if anyone could tell me what phone system they are using with Zoho that is on the list of systems that integrate with Zoho.  I use Vonage and have been with them for quite a few years but their service has really gone down hill and
                                                                                                                • Removing Related Modules Lookup Fields Assignment / Relationship

                                                                                                                  Issue: When creating a related list, I accidently selected module itself creating a circle reference. See attached. Situation: I wish to relating a custom module called "Phone Calls" to Leads and Contacts. Outcome: 1) I either want to remove the this
                                                                                                                • Multiple email accounts, one phone

                                                                                                                  Hi, I have my own businesses that use Zoho mail, now I would like to use zoho mail also for a local charity. But I noticed adding a phone number is compulsory for registering an account, and I only have one phone. This means I can not open an account because my phone number is already in use for another account. Can this be fixed? Thanks Vincent
                                                                                                                • Seriously - Create multiple contacts for leads, (With Company as lead) Zoho CRM

                                                                                                                  In Zoho CRM, considering a comapny as a lead, you need us to allow addition of more than one contact. Currently the Lead Section is missing "Add contact" feature which is available in "Accounts". When you know that a particular lead can have multiple
                                                                                                                • can I link a contacts to multiple accounts

                                                                                                                  can I link a contacts to multiple accounts
                                                                                                                • Free webinar! Digitize recruitment and onboarding with Zoho Sign and Zoho Recruit

                                                                                                                  Hello, Tired of being buried in onboarding paperwork? With the integration between Zoho Sign and Zoho Recruit, a powerful applicant tracking system, you can digitize and streamline the entire recruitment and onboarding process, all from one platform.
                                                                                                                • Open Activities view.

                                                                                                                  I really like the new views for the open and closed activities inside the deals. But when you are in the tab view instead of the column view you can only complete and edit the open activity there isn't the 3 dot option to be able to delete the activ
                                                                                                                • Potentially Outdated and Vulnerable Chromium Engine Installed by Ulaa Browser Installer

                                                                                                                  I just installed Ulaa Browser a few minutes ago. Whats My Browser page shows I am using an outdated Chromium engine meaning I might be vulnerable for security exploits that might have got fixed in the new version.
                                                                                                                • Potentially hardcoded list of Browsers to import from (after Ulaa Setup)

                                                                                                                  I have just installed Ulaa Browser and found that the list of browser to import data is potentially hardcoded ones rather than looking at the system. I do not have FF, IE and Edge is not my default itself. I would appreciated if Ulaa detected my browsers
                                                                                                                • From Layout to Code: Finding Custom Field IDs in Zoho Projects.

                                                                                                                  Hello everyone! Ever found yourself wondering how to get the API names and IDs of custom fields in Zoho Projects while working on custom functions? Here’s a simple and effective way to do it! This method makes it super easy to locate the right field details
                                                                                                                • Can I create multiple Zoho mail accounts?

                                                                                                                  Hey there I've searched the FAQ for this but didn't find the answer I was looking for. I like to have a different e-mail address for personal stuff, for creating account on websites, and for online banking / e-commerce. Is it allowed to create three accounts for one person on Zoho? I already have one personal e-mail address, can I sign up two more times from the homepage? Thanks
                                                                                                                • Employee type and source translation

                                                                                                                  In Zoho People, when I fill in the employee’s information, there is the option to determine the type of employment (employee type) and the hiring source. Both options ALWAYS appear in English. It is extremely inconvenient to deal with poorly translated
                                                                                                                • Sync Issue Between Zoho Notebook Web App on Firefox (PC) and Android App

                                                                                                                  Hi Zoho Notebook Community, I'm facing a sync problem with Zoho Notebook. When I use the web version on Mozilla Firefox browser on my PC, I create and save new notes, and I've synced them successfully. However, these new notes aren't showing up in my
                                                                                                                • Request for Clarity on Timeline for True GPT/Zia Auto-Response Capabilities

                                                                                                                  I appreciate Zoho’s steady innovation, but I’m concerned that Desk and Zia remain well behind modern AI capabilities. For years, GPT-based tools have been able to generate and send contextual responses, yet Zoho Desk only supports summarization or suggested
                                                                                                                • Notebook audio recordings disappearing

                                                                                                                  I have recently been experiencing issues where some of my attached audio recordings are disappearing. I am referring specifically to ones made within a Note card in Notebook on mobile, made by pressing the "+" button and choosing "Record audio" (or similar),
                                                                                                                • Has anyone built a ticket export that allows Help Center users to export the tickets shown in the My Area list they are looking at?

                                                                                                                  Hi, We are moving to Zoho Desk soon. Our current support system displays an option in our help center allowing customers to export their Open, Closed, or all tickets based on which list they are looking at. We need to offer the same in Zoho Desk help
                                                                                                                • Zoho Desk: Q2 2025 | What's New

                                                                                                                  Hello everyone, We are excited to announce Zoho Desk's 2025 Autumn updates. This release brings new features and enhancements that improve work management and enable businesses to provide a better overall support experience. Spanning from Zia Agents to
                                                                                                                • Checkboxes not adhering to any policy in mail merge - data from CRM

                                                                                                                  I want checkboxes to appear depending on whether the checkbox in the CRM module is ticked or not. However, the tickboxes that appear are either ticked or not, but don't correlate to the actual selections in the CRM module. This is is despite updating
                                                                                                                • Items Landed Cost and Profit?

                                                                                                                  Hello, we recently went live with Zoho Inventory, and I have a question about the Landed Cost feature. The FAQ reads: "Tracking the landed cost helps determine the overall cost incurred in procuring the product. This, in turn, helps you to decide the
                                                                                                                • CC and/or BCC users in email templates

                                                                                                                  I would like the ability to automatically assign a CC and BCC "User (company employee)" into email templates. Specifically, I would like to be able to add the "User who owns the client" as a CC automatically on any interview scheduled or candidate submitted
                                                                                                                • Create Contract API Endpoint Unclear "inputfields" Requirements

                                                                                                                  Hello, I'm trying to create a Deluge function that accepts inputs from a form in Zoho Creator and creates a barebones contract of a given type. See below for the current code, cleaned of authentication information. // Fetch form data // Hidden field client_name
                                                                                                                • Kaizen #46 - Handling Notes through Zoho CRM API (Part 1/2)

                                                                                                                  Hello everyone! Welcome back to another week of Kaizen! This week, we will discuss Handling Notes through Zoho CRM API. What will you learn from this post? Notes in Zoho CRM Working with Notes through Notes APIs 1. Notes in Zoho CRM 1a. Why add Notes to records? Notes are a great way to summarize your observations on customer and prospect interactions and outcomes. By saving notes as CRM data, a sales rep will always be able to keep track of how a sale is progressing. To know more about notes in
                                                                                                                • Marketer's Space - Why email marketing matters in ecommerce (and how to get started with Zoho Campaigns)

                                                                                                                  Hello Marketers, Welcome to this week's Marketer's space post. Today, we'll discus why email marketing matters in ecommerce businesses. Running an online store is exciting but challenging. If you're running an online store, you've probably experienced
                                                                                                                • Zoho Campaigns Event timestamps do not propagate to Zoho CRM

                                                                                                                  We have integrated Zoho CRM and Zoho Campaigns. But when looking at Contact records, the Campaign event data is missing the actual timestamps: especially when a particular email was sent. They're not in the Campaigns related list, and the cannot be found
                                                                                                                • Kaizen #121 : Customize List Views using Client Script

                                                                                                                  Hello everyone! Welcome back to another interesting Kaizen post. In this post, we can discuss how to customize List Views using Client Script. This post will answer the questions Ability to remove public views by the super admin in the Zoho CRM and Is
                                                                                                                • Setting default From address when replying to request

                                                                                                                  At the moment, if I want to reply to a request, the From field has three options, company@zohosupport.com, support@company.zohosupport.com, and support@company.com.  The first two are really internal address that should never be seen by the customer and
                                                                                                                • Tip #45 - Explore Your Support Reach with Zoho Assist’s Geo Insights - 'Insider Insights'

                                                                                                                  Understanding where your remote support sessions are happening can help you make smarter decisions, allocate resources effectively, and improve overall customer satisfaction. In this week's Zoho Assist's community post we will be exploring Geo Insights
                                                                                                                • Formatting of text pasted into Zoho documents

                                                                                                                  Howdy, I'm a newbie and finding Zoho an improvement to MS Word. Consider yourself hugged. High on my wish list would be plain text cut-and-paste. When pasting text from the web to Zoho, presently Zoho imports the formatting along with the text. This means that every cut-and-paste operation brings in text in a different font, size, or style. Can we have at least the option of importing plain text without formatting (or better yet, is this option already out there?) ... Thanks Helen
                                                                                                                • Add additional features to Zoho Tables

                                                                                                                  Zoho Tables is a really great tool, why not add features like diagramming capability into the tool from applications like Draw.io which I believe is open source, you should be able to do wireframes, process flow diagrams, network design, etc. Please note
                                                                                                                • The Social Wall: August 2025

                                                                                                                  Hello everyone, As summer ends, Zoho Social is gearing up for some exciting, bigger updates lined up for the months ahead. While those are in the works, we rolled out a few handy feature updates in August to keep your social media management running smoothly.
                                                                                                                • The Social Wall: July 2025

                                                                                                                  Hello everyone! July has brought some exciting new updates to Zoho Social. From powerful enhancements in the Social Toolkit to new capabilities in the mobile app, we’ve packed this month with features designed to help you level up your social media presence.
                                                                                                                • Use Zoho Creator as a source for merge templates in Zoho Writer

                                                                                                                  Hello all! We're excited to share that we've enhanced Zoho Creator's integration with Zoho Writer to make this combination even more powerful. You can now use Zoho Creator as a data source for mail merge templates in Zoho Writer. Making more data from
                                                                                                                • Tagged problem !!!

                                                                                                                  Damn it, we're one of dozens of construction companies in Africa, but we can't link purchasing invoices to projects. Why isn't this feature available?
                                                                                                                • Syntax for URLs in HTML Snippets

                                                                                                                  What are some best practices for inserting a URL in an HTML snippet? I've looked at Zoho Help articles on navigation-based and functional-based URLs, but I'm still unclear on how to incorporate them in an HTML snippet. For example, 1. How do I link to
                                                                                                                • The Social Wall: June 2025

                                                                                                                  Hello everyone, We’re back with June Zoho Social highlights. This month brought some exciting feature updates—especially within the Social Toolkit—to enhance your social media presence. We engaged with several MSME companies through community meet-ups
                                                                                                                • Tip 7: How to fetch data from another application?

                                                                                                                  Hi everyone, Following our Zoho Creator - Tips and Tricks series every fortnight, we are back today with a tip based on one of the most popular questions asked in our forum. This tip would help you fetch data from another application(App B) and use it
                                                                                                                • The Social Wall: May 2025

                                                                                                                  Hey everyone, We're excited to share some powerful updates for the month of May! Let's take a look! Reply to your Instagram and Facebook comments privately through direct messages Are you tired of cluttered comment threads or exposing customer queries
                                                                                                                • Sub-Form Fields as Filters for Reports

                                                                                                                  Hi, I would like to use the Sub-Form Fields as Filters in Reports just like we do for Main Page Fields. Thanks Dan
                                                                                                                • Zoho CRM Formula - Current Time minus Date/Time field

                                                                                                                  Hello, I am trying to prevent duplicate emails going to clients when more than 1 deal is being updated. To do this, I would like to create a formula to identify if a date/time field is >= 2 hours ago. Can someone please help me write this formula? Example:
                                                                                                                • Next Page