Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Using a third party service provider want to move directly with Zoho

                                                                                                                  Hi good day I’m currently using Zoho but I’m using a third party service provider I want to move directly with you guys I’m using Zoho email and invoices and my domain please let me know if it’s possible to move away from the third party provider my email
                                                                                                                • Request for Assistance Regarding Email Sending Issue (554 5.1.8 - Email Outgoing Blocked)

                                                                                                                  Dear Zoho Support Team, I hope this message finds you well. I am writing to request assistance with an issue we are currently facing regarding our Zoho Mail account. Our email account, admin@tuyensinhcanuoc.com, is encountering the following error when
                                                                                                                • Zoho Mail API returns empty inbox (0 messages) but webmail shows 37 unread emails

                                                                                                                  Hello, I'm experiencing a discrepancy between Zoho Webmail and the Mail API (EU region). **Setup:** - Account: EU datacenter (mail.zoho.eu) - API: Self Client OAuth2 via api-console.zoho.eu - Scopes: ZohoMail.messages.READ, ZohoMail.messages.UPDATE, ZohoMail.folders.READ,
                                                                                                                • ShipStation and Zoho Inventory

                                                                                                                  Hello, I am looking to sync zoho inventory with shipstation ZOHO INVENTORY           SHIP STATION Sales Order  ==>  create ORDERS INVOICE  <==    Shipments What exactly does BETA mean on the Shipstation connector?  This is required for me to sign-on in the next month. Thanks in advance for your efforts
                                                                                                                • E

                                                                                                                  We are trying to add our Zoho Form embed in our Elementor Page Builder. After adding Zoho Forms widget in elementor page builder it’s displaying in backend page builder but it’s giving 403 error while trying to save, as it’s not reflecting in front end.
                                                                                                                • Connecting Zoho Inventory to ShipStation

                                                                                                                  we are looking for someone to help connect via API shipStation with Zoho inventory. Any ideas? Thanks. Uri
                                                                                                                • custom module import.

                                                                                                                  Is there a way to import data into a custom module? Thanks Rudy
                                                                                                                • HEIC File Type Viewer

                                                                                                                  Hi, It would be nice to be able to click on the images in the All Entries/Reports Tables which are HEIC the same as JPG, PNG, etc. so they open in a viewer from Zoho or the Attachment Service, today HEIC requires you to download each image and open it
                                                                                                                • How to dynamically pass IDs from one API response to a second API call in Zoho DataPrep?

                                                                                                                  Hi Team, I am setting up a global consolidated reporting pipeline in Zoho Analytics (India) using Zoho DataPrep as the ETL engine. I am pulling data from multiple Zoho Books Data Centers (US, Singapore, India) thats why i used the URL Source (OAuth 2.0)
                                                                                                                • Building Toppings #4 - Setting up and using connections in Bigin toppings

                                                                                                                  When building a topping to extend Bigin's functionality and connect it with third-party applications, creating and handling connections is an important step. Connections provide a secure way for your topping to authenticate and communicate with other
                                                                                                                • Need code format to specify default values

                                                                                                                  Can someone please direct me to the code syntax or the proper translation per the instructions circled below. These instructions don't seem correct.
                                                                                                                • AI Interview Insights: Turn Recorded Interviews into Quick Transcripts & Summaries

                                                                                                                  Evaluating interviews shouldn’t require replaying long recordings or taking manual notes. With AI Interview Insights, you can now review complete transcripts and AI-generated summaries of your One-way (Recorded) interviews right inside Zoho Recruit. This
                                                                                                                • Record payment: Payment Mode vs. Deposit To and how to "connect" them!?

                                                                                                                  How do we set up that when we choose:  "Payment Mode"= Cash, then "Deposit to" is automatically set to Petty Cash, and if we choose  "Payment Mode"= Check, then "Deposit to" is automatically set to Undeposited Checks, and if we choose  "Payment Mode"=
                                                                                                                • Grouping Undeposited Funds to Move to other accounts

                                                                                                                  In the bank option it would be nice to check what transactions in undeposited funds I want to move to other accounts. Then while checking this it can accumulate totals and created whats essentially a deposit slip. Once the transaction is moved it should
                                                                                                                • Depositing funds to account

                                                                                                                  Hello, I have been using Quickbooks for many years but am considering moving to Zoho Books so I am currently running through various workflows and am working on the Invoicing aspect. In QB, the process is to create an invoice, receive payment and then
                                                                                                                • Facing email delivery issues? Verify your domain's DNS records

                                                                                                                  Have you ever wondered why your legitimate emails are landing in the recipient’s spam folder? Or been surprised to see emails sent from your registered domain getting rejected by recipient email servers? Why does this happen? In most cases, this happens
                                                                                                                • Order of Departments in Help Desk

                                                                                                                  In the end user portal, , the departments are sorted by the date of creation of the department (or perhaps their id). Is there a way to choose the display order of the departments or at least to be able to sort them alphabetically?
                                                                                                                • COGS - Account showing negetive

                                                                                                                  I have multiple COGS account and in these all there is one account is negetive so suggest why it is showing negetive value.?
                                                                                                                • Create CRM Deal from Books Quote and Auto Update Deal Stage

                                                                                                                  I want to set up an automation where, whenever a Quote is created in Zoho Books, a Deal is automatically created in Zoho CRM with the Quote amount, customer details, and some custom fields from Zoho Books. Additionally, when the Sales Order is converted
                                                                                                                • %PaymentLink%

                                                                                                                  Does not work. Software creates a BAD link. ....and yes payment options are turned on. Link on the invoice pdf once opened will work but this template is a joke.
                                                                                                                • Google Photos

                                                                                                                  I am hoping that my question already has a fix. I current have Google synced accounts that I want to get away from. One in particular on is Google photos. Is there any software, or 3rd parties that I can join to back my photos up straight to specifically designated file in the ZOHO cloud that's tied to Docs? Please advise... Mike 
                                                                                                                • Zoho Books Items Categorisation/Grouping/Folder

                                                                                                                  Is there a way to do items categorisation? a folder structure? Product Type A - Option 1/2/3 Product Type B - Option 1/2/3 Current problem : I have more than 50 items on the list, its hard for team to navigate.
                                                                                                                • Cash payments before invoice date

                                                                                                                  We have been using zoho books for our hospitality business for some time and have been very happy with the system. However in 2025 an update was pushed through and we are now not able to record payments for invoices before the invoice date. the case scenario
                                                                                                                • Copy / Duplicate Workflow

                                                                                                                  I have workflows setup that are very similar to each other. We have a monitoring system watching servers, and all notifications - no matter what client it is about - will come from a  noreply@ address which is not very helpful in having it auto assigned to the right account. I have setup a workflow that will change the contact name of the ticket (currently it would say noreply@) to the correct customer which is based on the subject line, as that mentions which server the alert it is about. I need
                                                                                                                • Transfer between two customers (Peters Rental account to Peters Private account)

                                                                                                                  we are a Property Management company. Our customers have to accounts (registered as two customers - Peter Rental and Peter Private On the rental account all income and costs fron rental activities are noted. On the private account all private are noted
                                                                                                                • Automation#18: Automatically Fetch Values from Contacts to the Tickets Module

                                                                                                                  Hello Everyone, Welcome to this week's edition, where you can seamlessly sync fields from the Contacts to the Tickets module. For efficient business operations, it's crucial to have details mapped across different modules. Zylker Secure offers antivirus
                                                                                                                • Charge multiple invoices

                                                                                                                  We use auto-charge/recurring invoicing for 100's of clients. If we create a single manual invoice we can charge it to the credit card for these clients with the CC added to their recurring invoices. If a client has multiple outstanding invoices, is there a way to make one charge for all of the invoices instead of a separate charge to the cc for each invoice?
                                                                                                                • Handling/tracking escalation management in Zoho

                                                                                                                  Hi all, I am working on finding a tool to register and track our escalation management process. Specifically, this is about client escalations, typically related to project delivery issues. The idea is that we could have some sort of form with core questions
                                                                                                                • Global Choice List share ownership

                                                                                                                  I have created several forms that use one or more Global Choice Lists. These lists have been published to Org. I would like to allow one or more admins to edit the choices in these lists. Any help appreciated. Geoff
                                                                                                                • Setting up property management in Zoho Books

                                                                                                                  Hi, I run a property management business that manages property complexes. There are multiple owners, some owning more than one property on the same complex. My role is to manage the fees they pay for maintenance of common areas, such as the swimming pool
                                                                                                                • How to prevent users from switching price lists in an order?

                                                                                                                  Hi, I have Zoho Finance integrated with Zoho CRM. My team will be placing orders through the CRM using the Finance module. When creating a new customer I will assign it a price list, I don't want the sales rep to switch to a different Price List, other
                                                                                                                • Unable to Change Display Name for my outlook mail ID

                                                                                                                  Hello, I recently added my personal outlook mail to Zohomail (in the web interface) using POP3 options. I tried to change the display name of the outlook account, but it changes for a brief moment and reverts back to the old name. Kindly help me with
                                                                                                                • Mollie Connect in Zoho Books

                                                                                                                  Have noticed that Zoho Books is missing the most popular payment provider in Europe: Mollie. Knowing that Mollie has a solid Java SDK, why is this not yet implemented in Zoho Books for European customers?  More info: https://docs.mollie.com/oauth/ov
                                                                                                                • Unable to change Lookup field from Multi Select to Single Select

                                                                                                                  I am trying to change a Lookup field in my Zoho Creator form from Multi Select to Single Select, but I am unable to find any option to do this.
                                                                                                                • Simple Callback Notifications Needed

                                                                                                                  My team are terrible at remembering their CRM callbacks, often due to how long in the future they are set for. Is there a way i can set an e-mail notification for when a callback is due? For example we set it for 9am one day and five minutes before they
                                                                                                                • Personal Data (RODO), Cookies / Trackers - ePrivacy

                                                                                                                  I have noticed several issues that should be addressed on the customer support page. Zoho Desk provides the support portal, but it currently lacks the following options: A GDPR and personal data processing consent checkbox before logging in, located in
                                                                                                                • How to set a multi-lookup field as mandatory?

                                                                                                                • Allow Multiple Scheduled Appointments with Zoho Support

                                                                                                                  Dear Zoho Team, I hope you're doing well. First, thank you for introducing the option to schedule support calls via the Zoho CRM booking link. This has been a fantastic enhancement, eliminating the need for back-and-forth coordination when scheduling
                                                                                                                • Audit Log for Zoho One Admin Panel

                                                                                                                  Dear Zoho One Team, We would like to request the addition of an Audit Log feature in the Zoho One Admin Panel. This log should provide visibility into any changes made within the Zoho One admin panel and directory, including but not limited to: Adding,
                                                                                                                • Bug: OAuth 2.0 State Parameter fails with Pipe Delimiters (RFC 6749 Non-Compliance)

                                                                                                                  I've discovered a bug in how Zoho's API Console handles the OAuth 2.0 authorization flow when the state parameter contains pipe characters (|), and I'm hoping the Zoho team can address this in a future update. The Issue Zoho's OAuth 2.0 implementation
                                                                                                                • Next Page