Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Accounting on the Go Series-55: Seamlessly Add New Vendors While Creating Transactions from Scanned Documents

                                                                                                                  Hi there! We’ve made handling documents and vendors in Zoho Books even simpler. Now, when you upload a document and scan it, if the app detects a vendor that isn’t already in your organization, you don’t have to leave the page to add them manually. With
                                                                                                                • Accounting on the Go Series-57: Effortlessly Add and Manage Bank Accounts from the Zoho Books Mobile App

                                                                                                                  Hi all, Great news for our users in the US and Canada! Managing your finances just got a whole lot easier. Zoho Books has always made it seamless to integrate bank accounts and fetch feeds automatically. Now, we’re taking it up a notch—directly from the
                                                                                                                • Accounting on the Go Series:58-Effortless Compliance: Download XML for Invoices & Credit Notes on Mobile

                                                                                                                  Hi there! In Mexico, XML files are crucial for electronic invoicing and fulfilling SAT (Tax Administration Service) requirements. These files ensure your transactions are accurately recorded and tax-compliant. Now, you don’t need to rely on the web app
                                                                                                                • Auto Generated Invoice number YEAR

                                                                                                                  Auto Generated Invoice number shows transaction year as 25 even though it's 24 still.
                                                                                                                • Books generiert keine valide XRechnung

                                                                                                                  Hallo zusammen, ich möchte hier ein Problem ansprechen, das mir aktuell bei der Nutzung von Books erhebliche Schwierigkeiten bereitet, und hoffe auf Austausch oder Lösungsansätze von anderen Nutzern. Bei der Erstellung von XRechnungen mit Books treten
                                                                                                                • WHEN UPDATE ORGANIZATION INFO OLD INVOICES ALSO CHANGE INVOICING DATA

                                                                                                                  Hi We have updated our tax information because we have become a company, up until now we were an individual. The problem is that when updating the data in the zoho books profile all the old invoices change their tax information as well. Is there a way
                                                                                                                • Invoice import error - duplicate customer name column - there are no duplicates

                                                                                                                  It is not allowing the importing of any rows because of a duplicate customer name problem, but there are no duplicates in the custoemr name row. Has anyone dealt with this issue before?
                                                                                                                • [WEBINAR][Feb 2025] Automate your entire financial operations from receipts to reconciliation with Zoho Books & Zoho Expense integration

                                                                                                                  Hello there, We are hosting a free, live webinar on the importance of travel and expense management solutions for businesses, and how Zoho Expense automatically syncs with Zoho Books to simplify your accounting even further. The webinar is on February
                                                                                                                • Customize Layout

                                                                                                                  I am using "Customize Layot" for customize Quotation template but I try to add logo into the header by "%ScaledLogoImage%" code but dont add logo into header. what is problem?
                                                                                                                • Problem - cant add Users (i.e. Zoho one / CRM Users) to BCC or CC in email, i.e. Sales orders or Retainers

                                                                                                                  I can go to zoho books email templates, and select any email template, and automatically include any Zoho One user, i.e. member of staff. However in the context of sending an email, it will not let us add a member of staff from the user list, instead
                                                                                                                • UK MTD ITSA

                                                                                                                  UK Making Tax Digital for Income Tax I have had notice this is to apply from April 2026. What is Zoho doing about this? I will need to start planning to implement this in the next months so need an update as to what I will and will not be able to do in
                                                                                                                • Zoho Books Roadshows are back in the UAE!

                                                                                                                  Hello there, Business owners and accounting professionals of the UAE, we’re coming to your cities! FTA accredited Zoho Books is now officially one of the Digital Tax Integrators in the UAE. With the newly launched direct VAT filing capabilities, we're
                                                                                                                • New user After moving over from QBO

                                                                                                                  New user observations/suggestions. QBO took away a lot of features I was used to with the desktop version. Chaos ensued. Zoho Books has a lot of what I was used to and a bit more. Good deal Some things I have run into and suggest some upgrades. 1: The
                                                                                                                • Sole Trader - Financial Advisor (Appointed Representative) - Paid via Capital Account but no Invoicing...

                                                                                                                  Hi. I'm about to venture into a new business after 12 months of intensive learning/exams. A little chuffed if I may say so especially at 52! I really like the look of ZoHo Books for my modest enterprise but I'm in need of some guidance, please. My services
                                                                                                                • Multi-Unit Inventory with Flexible Unit Selection (Purchase in One Unit, Sell in Another)

                                                                                                                  We need multi-unit inventory management in Zoho Books with the flexibility to choose units (e.g., Box or Piece) at the time of purchase or sale. For example, if 1 Box = 10 Pieces, we should be able to record purchases in Boxes but sell either in Boxes
                                                                                                                • Disputed Purchase Invoices

                                                                                                                  We have recently moved to Zoho Books from Sage. In Sage we were able to post a purchase/vendor invoice but mark it as on dispute. This would usually be a pricing query or if something was damaged. It would show in their ledger, so we could agree their
                                                                                                                • No TDS Deduction

                                                                                                                  In some of our case, where we are reselling items at the same rate we purchased. In this scenario, Indian IT Law has a provision to request customer not to deduct TDS if the transaction value is same. TDS is paid by us (intermediary reseller) before we
                                                                                                                • CBSA - GST CHARGES on imports

                                                                                                                  Hi there, We have a questions about landed cost categorization. We received a shipment from overseas. CBSA invoiced us for the GST on the items. Now we entered the CBSA-GST as a separate bill and attached it as landed cost to the main invoice based on
                                                                                                                • Zoho Books

                                                                                                                  How do I manually insert opening balance?
                                                                                                                • Sales order & purchase order item links for item details

                                                                                                                  This is fantastic for checking lots of things, I use it a lot. It would be great to see it extended to invoices & bills On another note, may as well throw in my favourite whinge ..... Wish you guys would get the PO receive differences sorted urgently,
                                                                                                                • Bank charges are applied. Please select a bank account.

                                                                                                                  Hello, I'm trying to add bank charges to a customer payment, but I get the error message "Bank charges are applied. Please select a bank account." I found this old thread, where it says that I need to "select a Bank account for the 'Deposit To' dropdown
                                                                                                                • How to add receipts

                                                                                                                  How to add receipts
                                                                                                                • Support for auto-upgrade in TrueSync (for Windows)

                                                                                                                  WorkDrive TrueSync app now supports auto-upgrading to the latest version for Windows OS. You must manually download and install the TrueSync app version 3.4.0 to avail this feature. Download the latest TrueSync app for Windows (version 3.4.0) Supported
                                                                                                                • WorkDrive API Documentation

                                                                                                                  WorkDrive provides users and developers an extensive set of APIs to help integrate functionalities of Zoho WorkDrive with other Zoho applications and third-party tools. We have published the official WorkDrive API Documentation page for all external users.
                                                                                                                • March 15, 2023: Zoho Docs is discontinued

                                                                                                                  As of today (March 15, 2023) Zoho Docs is discontinued for all users. We would like to thank our customers for trusting us for so many years! Going forward, we're confident you'll enjoy using Zoho WorkDrive for all your advanced file management and collaboration
                                                                                                                • Introducing WorkDrive 4.0: Enhanced productivity. Advanced data administration. (Phase 1)

                                                                                                                  Hello All, We're excited to share the release of WorkDrive 4.0, which includes important new features and enhancements focused primarily on productivity, secure collaboration, data administration, integrations, and user experience. Read the official announcement
                                                                                                                • External download link limit

                                                                                                                  Can You please help us to understand this For Zoho WorkDrive external users, the download limit is a maximum of 5 GB total download size and a maximum of 50 first-level files and folders What is the meaning of first level? We are using these files in
                                                                                                                • Dynamically catching new file creations

                                                                                                                  I have a team folder with many subfolders, and in those folders we add new documents all the time. I'd like to have a workflow or script to notify me (and then take other actions) when a file is added anywhere in that structure that ends in "summary.txt".
                                                                                                                • Rotate an Image in Workdrive Image Editor

                                                                                                                  I don't know if I'm just missing something, but my team needs a way to rotate images in Workdrive and save them at that new orientation. For example one of our ground crew members will take photos of job sites vertically (9:16) on his phone and upload
                                                                                                                • Workflow workdrive rollout

                                                                                                                  Hi! When will workflow be rolled out to all users? Thanks.
                                                                                                                • Creating and managing a Team Folder using WorkDrive TrueSync

                                                                                                                  Hello everyone, Are you tired of constantly switching between your Desktop TrueSync app and the WorkDrive web app to create and manage Team Folders? We’ve made things easier for you. You can now create and manage Team Folders directly within the TrueSync
                                                                                                                • Edit images seamlessly with WorkDrive's built-in Image Editor

                                                                                                                  Are you tired of switching between multiple tools just to make simple edits to your images? We understand the hassle, which is why Zoho WorkDrive now comes with a built-in image editing tool, powered by Zoho Annotator. This tool allows you to edit images
                                                                                                                • Supercharge your email workflow with WorkDrive's add-in for Microsoft Outlook

                                                                                                                  Consider this: You’re handling a critical project, and your inbox is packed with important attachments, email threads, and client communications. The back-and-forth routine of downloading files to your computer, uploading them to WorkDrive, and manually
                                                                                                                • Secure and promote your content with Custom Watermarking

                                                                                                                  Imagine this: You’re a professional photographer who regularly shares your work online with potential clients and collaborators. Recently, you notice that some of your images have been reposted without any credit or permission. This not only impacts your
                                                                                                                • Join us in Singapore for the Zoho WorkDrive User Group meetup!

                                                                                                                  Hello, everyone! Exciting news! We'll be hosting an upcoming Zoho WorkDrive user group meetup in the beautiful city of Singapore this November. At this Zoho User Group meetup, we'll guide you through ways to use WorkDrive as a platform and build custom
                                                                                                                • WorkDrive TrueSync now supports ARM64-based Windows devices!

                                                                                                                  We’re excited to announce that the Zoho WorkDrive TrueSync app now fully supports Windows devices with ARM64 architecture! Whether you're working on an ARM-based device or an x64 processor, you can now enjoy the same seamless file synchronization experience
                                                                                                                • Option to Disable Download for Documents Shared via Permalink

                                                                                                                  Dear Zoho Writer Team, Currently, when sharing a Writer document using the regular permalink (Collaborators with external users), there is no option to restrict the ability to download the document. While the external share link allows such restrictions,
                                                                                                                • How to Download a File from Zoho WorkDrive Using a Public Link

                                                                                                                  How to Download a File from Zoho WorkDrive Using a Public Link If you're working with Zoho WorkDrive and want to download a file using a public link, here's a simple method to do so using API or a basic script. This approach helps developers or teams
                                                                                                                • domain not verified error

                                                                                                                  Hi when i try to upload a video from zoho creator widget to zoho work drive iam getting domain not verified error.I don't know what to do .In zoho api console this is my home page url https://creatorapp.zoho.com/ and this is my redirect url:www.google.com.Iam
                                                                                                                • Live Webinar: Getting Started with Zoho WorkDrive - A Complete Overview

                                                                                                                  Hello everyone, We’re excited to invite you to our upcoming live webinar! Discover how to set up your team, bring in your data, and make the most of WorkDrive’s collaboration, organization, AI, and security capabilities. This session is perfect for anyone
                                                                                                                • Next Page