Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Computer Showing Offline in Unattended Access

                                                                                                                  I have a computer that was connected to the internet but showing offline in Assist. I tried uninstalling the program and deleting it from Zoho Assist the reinstalling and it still does not show up. I have been a user for several months and am not pleased with the lack of connectivity with Assist. If this continues I will have to find another product. The computer I reinstalled it on is not even showing up in Assist now. The name is NYRVLI-PC. Thanks
                                                                                                                • Zoho CRM Portal Error

                                                                                                                  Hi, We’re experiencing an issue with the Zoho CRM Portal. When we try to access it, we receive an HTTPS connection error: net::ERR_CERT_COMMON_NAME_INVALID. If we proceed past that, we then get a 400 Bad Request error. Could you please help us resolve
                                                                                                                • Integrating Zoho CRM EmbeddedApp SDK with Next.js — Initialization and Data Fetching Issues

                                                                                                                  You can get an idea from my code I have given in end: First, I worked on a React project and tried the same thing — it worked. My goal was to import the Zoho script and then load contacts using the Zoho Widget SDK, which was successful in React. Now,
                                                                                                                • monetizing the courses

                                                                                                                  Can I add a price for course enrollment ?
                                                                                                                • Can we add zoho books features like invoices estemates etc on our zohocommerce website. When our customer login with their account they can able to see all books features in one place on zohocommerce?

                                                                                                                  Can we add zoho books features like invoices estemates etc on our zohocommerce website. When our customer login with their account they can able to see all books features in one place on zohocommerce?
                                                                                                                • Taxes for EU B2B Transactions

                                                                                                                  Currently, ZC doesn't seem to have a procedure for validating VAT numbers of businesses purchasing in another EU state, and removing local VAT is valid. This is essential for all inter EU B2B trade.
                                                                                                                • Unable to Receive Emails on Zoho Mail After Office 365 Coexistence Setup – Error: 553 Relaying Disallowed

                                                                                                                  Hello, My domain name is bigniter.com, and I’ve been using Zoho Mail as my email service provider without any issues. Recently, I followed the steps outlined in the Zoho documentation to enable Coexistence with Office 365: 🔗 https://www.zoho.com/mail/help/adminconsole/coexistence-with-office365.html#multi-server
                                                                                                                • Candidate Registration/Invitation

                                                                                                                  It would be great to include the 'invite' candidate functionality into some of the automation functions - ether through a custom function trigger or webhook or accessible through an email template.  Currently there is no way to add this functionality into any workflows or blueprint steps which, I find limits the ability to invite candidates to engage with us directly through our candidate portal. 
                                                                                                                • [Free Webinar] Learning Table Series - Creator for Infrastructure Management | A Partner-driven collaborative session

                                                                                                                  Hello Everyone! We’re excited to invite you to another edition of Learning Table Series, where we showcase how Zoho Creator empowers industries with innovative and automated solutions. About the Learning Table Series The Learning Table Series is a free,
                                                                                                                • I have error AS101 when I try to add paypal@mydomain.com to Zoho

                                                                                                                  Please help me with this. I tried to call the help line 4 times but don't get any response.
                                                                                                                • VAT on Multiple Expenses

                                                                                                                  Firstly, might I say I am based in Ireland, and I'm not an accountant, and therefore my understanding of procedures could be somewhat limited. Scenario. An employee pays for items throughout a month, and therefore their claim for expenses includes multiple items - some of the items are VAT deductible (eg, an IT product) and some are not (eg, a client lunch). Discovery The odd thing in ZBooks, is that whilst it seems possible to add a VAT rate to a one-off item of expense, it is not possible to add
                                                                                                                • Agent assignment filter?

                                                                                                                  Godo day, We are starting to play with FSM to see if it's going to work for our needs. Now so far we have found that it's very restrcitve in the field department you you have layout rules or can't even hide fields depending on the users roles. We can't
                                                                                                                • Kiosk Page Refresh

                                                                                                                  We have a Kiosk running from a button in contacts to update values and also add related lists, which works great, but when the kiosk is finished the page does not refresh to show the changes. Is there a way to force the contact to refresh/update when
                                                                                                                • Tips & tricks: Make SalesIQ automations work for you

                                                                                                                  Every day, thousands of visitors land on your website. Some browse, some buy, and some leave without a word. But, wouldn’t it be great if you could automatically know who’s interested, engage them at the right moment, and never miss a lead, and all this
                                                                                                                • Digest Octobre - Un résumé de ce qui s'est passé le mois dernier sur Community

                                                                                                                  Chers utilisateurs, Faisons le point sur les temps forts du mois d'octobre au sein de notre Community Zoho France. Partager des informations sensibles, comme des mots de passe, peut vite devenir compliqué et risqué. Les données circulent par email ou
                                                                                                                • How do I fully delete a user account

                                                                                                                  Hi, I have two old accounts on my CRM which are inactive and show as 'Closed' in the list of users. BUT they are basically still there. How do I fully delete these accounts? They appear in some parts of CRM still such as the on the contact record emails
                                                                                                                • Automatically Add Recurring Zoho Meeting Events to Zoho Calendar / Zoho Meeting Calendar

                                                                                                                  Hello Zoho Meeting Team, Hope you are doing well. We would like to request an enhancement regarding recurring meetings created inside Zoho Meeting. At the moment, when we schedule a recurring meeting in Zoho Meeting, it does not appear in Zoho Calendar
                                                                                                                • Credit Management: #6 Tackling Common Mistakes in Credit Based Billing

                                                                                                                  Businesses extend credit to build relationships, make buying easy, and drive more sales. But somewhere between extending and collecting, things begin to slip. A few late payments here and there, an overlooked invoice, and a backlog make cash flow feel
                                                                                                                • Updating Blueprint stage from my Deluge function

                                                                                                                  Hi all, I need to update the blueprint stage. Its almost working, but ONLY if the current Deal stage is 'Closed Lost'. On any of my custom stages that Ive specified in the pipeline it doesnt work! My code: dealmap = Map(); deal_stage = deal.get("Stage");
                                                                                                                • Canvas: is it possible to have a fixed header?

                                                                                                                  Hello. Does Canvas provide the option to have a fixed header, similar to the standard view? It would also be interesting for other parts of the interface, like the header or sidebar of the tab section. Thanks!
                                                                                                                • Limited layout rules in a module

                                                                                                                  There is a limit of 10 layout rules per module. Is there a way to get that functionality through different customization or workflow + custom function (easily accessible), etc. Having just 10 is limiting especially if module contains a lot of data. Are
                                                                                                                • Using Contains as a filter

                                                                                                                  We are using Zoho Reports, ServiceDesk Plus analytics. I do not see how to create a report filter using Contains comparison of a string values, is one string contained in another. For example, Task Title contains the word Monitor. Is this possible in Zoho Reports?  This reporting feature is available in SDP reporting. Thanks in advance, Craig Rice
                                                                                                                • Set off Unearned Revenue

                                                                                                                  Hi, I would like to get a clarification on the below. I have an opening balance of BD.725/- in my Unearned Revenue A/c which includes the overpayments of 6 clients. No One of my clients settled his invoices by paying the value of the invoice less the
                                                                                                                • How Zoho Desk contributes to the art of savings

                                                                                                                  Remember the first time your grandmother gave you cash for a birthday or New Year's gift, Christmas gift, or any special day? You probably tucked that money safely into a piggy bank, waiting for the day you could buy something precious or something you
                                                                                                                • Add Webhook Response Module to Zoho Flow

                                                                                                                  Hi Zoho Flow Team, We’d like to request a Webhook Response capability for Zoho Flow that can return a dynamic, computed reply to the original webhook caller after / during the flow runs. What exists today Zoho Flow’s webhook trigger can send custom acknowledgements
                                                                                                                • When moments in customer support get "spooky"

                                                                                                                  It’s Halloween again! Halloween is celebrated with spooky symbols and meanings based on history and traditions, with each region adding its own special touch. While we were kids, we would dress up in costumes along with friends, attend parties, and enjoy
                                                                                                                • How to backdate record payment for the invoice

                                                                                                                  I would like to record a payment which happened 2 days ago, but I am not able to select any date later than today. I backdated the invoice, too, but it doesn't change anything
                                                                                                                • Questions about Zoho mail

                                                                                                                  Hello, I had up until recently, the free version. I got a warning message that said, "The user account (Redacted email address) 6.02 GB in your organization is almost nearing the allocated storage of 5.00 GB. If the storage exceeds the allowed limit,
                                                                                                                • How can I delete duplicate transactions?

                                                                                                                  I want to delete the duplicates not simply exclude them. I have duplicates, because I had automatic bank feeds turned on (had to make sure this critical functionality was working before migrating to Zoho). Now when I import my csv's exported from Wave,
                                                                                                                • Multiple email addresses

                                                                                                                  Multiple email addresses: I understand there is the ability to add an additional email field to a contact, but the functionality here needs to be drastically revised. Currently, this second email address does not log under the contact email history, nor is it available as a send address when using the Send Mail button within CRM. We recently migrated from GoldMine. I hate GoldMine, but there is one thing they did well: Email. I suggest copying them. Change the email field to an editable picklist,
                                                                                                                • ViewID and Zoho Desk API

                                                                                                                  I'm looking at the documentation for Listing contacts and it looks like you can filter by ViewID. I assume this is views created in a department or all departments. Is this correct? And if so, how do I find the ViewID for that view? EDIT: I see the view
                                                                                                                • Print Tickets

                                                                                                                  We have field engineers who visit customers. We would like the option to print a job sheet with full details of the job and account/contact details.
                                                                                                                • Zoho Desk integration with Power BI

                                                                                                                  Hi, I want to be able to create a Power BI report which has live updates of ticket data from zoho desk, is this possile at all? Thanks Jack
                                                                                                                • Ability to Attach Images When Reporting Issues to Zoho Projects from Zoho Desk

                                                                                                                  Hi Zoho Desk Team, Hope you’re doing well. We’re using the Zoho Desk–Zoho Projects integration to report bugs directly from support tickets into the Zoho Projects issue tracker. This integration is extremely useful and helps us maintain smooth coordination
                                                                                                                • Ability to Choose Task List and Add Subtasks When Creating Tasks from Zoho Desk

                                                                                                                  Hi Zoho Desk Team, Hope you’re doing well. We’re using the Zoho Desk–Zoho Projects integration to seamlessly connect customer tickets with project tasks. While the integration works great overall, we noticed two important limitations that affect our workflow
                                                                                                                • Sync Task Status from Zoho Projects to Zoho Desk

                                                                                                                  Hi Zoho Desk Team, Hope you’re doing well. We’re actively using the Zoho Desk–Zoho Projects integration, which helps our support and project teams stay aligned. However, we noticed that when we change a task’s status in Zoho Projects, the change is not
                                                                                                                • Default/Private Departments in Zoho Desk

                                                                                                                  1) How does one configure a department to be private? 2) Also, how does one change the default department? 1) On the list of my company's Zoho Departments, I see that we have a default department, but I am unable to choose which department should be default. 2) From the Zoho documentation I see that in order to create a private department, one should uncheck "Display in customer portal" on the Add Department screen. However, is there a way to change this setting after the department has been created?
                                                                                                                • Zoho CRM IP Addresses to Whitelist

                                                                                                                  We were told to whitelist IP addresses from Zoho CRM.  (CRM, not Zoho Mail.) What is the current list of IP Addresses to whitelist for outbound mail? Is there a website where these IP addresses are published and updated?  Everything I could find is over
                                                                                                                • Color of Text Box Changes

                                                                                                                  Sometimes I find the color of text boxes changed to a different color. This seems to happen when I reopen the same slide deck later. In the image that I am attaching, you see that the colors of the whole "virus," the "irology" part of "virology," and
                                                                                                                • The difference between Zoho Marketing Automation and Zoho Campaigns

                                                                                                                  Greetings Marketers! This post aims to differentiate between Zoho Marketing Automation and Zoho Campaigns. By the time you get to the end of the post, you will be able to choose a product that objectively suits you. What is Zoho Marketing Automation?
                                                                                                                • Next Page