Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Digest Janvier - Un résumé de ce qui s'est passé le mois dernier sur Community

                                                                                                                  Bonjour chers utilisateurs, Le premier mois de l’année est déjà derrière nous ! Découvrons ensemble comment s'est passé janvier pour Zoho Community France. Nous avons démarré le mois avec une nouvelle intégration entre Zoho Desk et Zoho Contracts. Cette
                                                                                                                • contact support

                                                                                                                  I have 2 request for a phone call and not received one call as promise!
                                                                                                                • Can we fetch Deal owner and Deal owner email with COQL?

                                                                                                                  While fetching deal deatils with coql api and while fetching deal owner name and email it is just giving owner id not ginving name and email https://www.zohoapis.in/crm/v2/coql
                                                                                                                • Zoho Social - Cliq Integration / Bot

                                                                                                                  Dear community / zoho, I am looking for a way to create a bot within Zoho Cliq to update my colleagues about our Zoho Social activities. For example, if a new post is published, it would be great if this post automatically would be shared in our social
                                                                                                                • Comment to DM Automation

                                                                                                                  Comment to DM automation feature in Zoho Marketing Automation, similar to what tools like ManyChat offer. Use case: When a user comments on a social media post (Instagram / Facebook), the system should automatically: Send a private DM to the user Capture
                                                                                                                • Send / Send & Close keyboard shortcuts

                                                                                                                  Hello! My team is so close to using Zoho Desk with just the keyboard. Keyboard shortcuts really help us to be more efficient -- saving a second or two over thousands of tickets adds up quickly. It seems like the keyboard shortcuts in Desk are only for
                                                                                                                • Handling Agent Transfer from Marketing Automation Journey to SalesIQ WhatsApp

                                                                                                                  We are currently using Marketing Automation for WhatsApp marketing, and the features are great so far We have a scenario where, during a campaign or journey, we give customers an option to chat with our sales team. For this, we are using SalesIQ WhatsApp
                                                                                                                • Webinar - Getting Started with Zoho LandingPage

                                                                                                                  Want to launch landing pages that support real marketing goals? This webinar is designed to help you understand the role landing pages play in your overall campaigns and how they contribute to lead generation and conversions. During this session, you’ll
                                                                                                                • Increase Round Robin Scheduler Frequency in Zoho Desk

                                                                                                                  Dear Zoho Desk Team, We hope this message finds you well. We would like to request an enhancement to the Round Robin Scheduler in Zoho Desk to better address ticket assignment efficiency. Current Behavior At present, the Round Robin Scheduler operates
                                                                                                                • Zoho mail to contacts and leads, but not to accounts?

                                                                                                                  We use the accounts tab a lot for our business because they can be linked to the sales orders. Now we want to use the mail add on to link communication of our emails to our accounts. However this is only possible for contacts and leads? It would be convenient
                                                                                                                • Marketing Tip #19: Upsell on product page to increase order value

                                                                                                                  Upselling doesn’t have to feel pushy. It can actually feel helpful when done right. Think "people also bought", upgrades, add-ons, or complementary items. A smart upsell increases your average order value and improves the customer experience by making
                                                                                                                • CRM limit reached: only 2 subforms can be created

                                                                                                                  we recently stumbled upon a limit of 2 subforms per module. while we found a workaround on this occasion, only 2 subforms can be quite limiting in an enterprise setting. @Anonymous User I've read about imminent increase of other components (e.
                                                                                                                • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                                                                  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                                                                                • オンライン勉強会のお知らせ Zoho ワークアウト (2/19 参加無料)

                                                                                                                  ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 2月開催のZoho ワークアウトについてお知らせします。 今回はZoomにて、オンライン開催します。 ▶︎参加登録はこちら(無料) https://us02web.zoom.us/meeting/register/6AyVUxp6QDmMQiDGXGkxPA ━━━━━━━━━━━━━━━━━━━━━━━━ Zoho ワークアウトとは? Zoho ユーザー同士で交流しながら、サービスに関する疑問や不明点の解消を目指すイベント「Zoho
                                                                                                                • Removing Tables from HTML Inventory Templates - headers, footers and page number tags only?

                                                                                                                  I'm a bit confused by the update that is coming to HTML Inventory Templates https://help.zoho.com/portal/en/kb/crm-nextgen/customize-crm-account/customizing-templates/articles/nextgen-update-your-html-inventory-templates-for-pdf-generator-upgrade It says
                                                                                                                • Ability to Link Reported Issues from Zoho Desk to Specific Tasks or Subtasks in Zoho Projects

                                                                                                                  Hi Zoho Desk Team, Hope you're doing well. When reporting a bug from Zoho Desk to Zoho Projects, we’ve noticed that it’s currently not possible to select an existing task or subtask to associate the issue with. However, when working directly inside Zoho
                                                                                                                • How do i move multiple tickets to a different department?

                                                                                                                  Hello, i have several tickets that have been assigned to the wrong department.  I am talking about hundreds of automatically generated ones that come from a separate system. How can i select them all at once to move them to another department in one go? I can select them in "unsassigned open tickets view" but i can't find a "move to another department" option. I also can't seem to assign multiple tickets to the same agent in that same view. Could somebody advice?
                                                                                                                • Let us view and export the full price books data from CRM

                                                                                                                  I quote out of CRM, some of my clients have specialised pricing for specific products - therefore we use Price Books to manage these special prices. I can only see the breakdown of the products listed in the price book and the specialised pricing for
                                                                                                                • Set Default Status of Assembly to "Assembled" When Entered in UI

                                                                                                                  I've just discovered the new "confirmed" status of Assemblies within Inventory. While I understand the intent of this (allowing for manufacturing planning and raw material stock allocation), it was initially confusing to me when manually entering some
                                                                                                                • How to apply customized Zoho Crm Home Page to all users?

                                                                                                                  I have tried to study manuals and play with Zoho CRM but haven't found a way how to apply customized Zoho CRM Home Page as a (default) home page for other CRM users.. How that can be done, if possible? - kipi Moderation Update: Currently, each user has
                                                                                                                • Plug Sample #14: Automate Invoice Queries with SalesIQ Chatbot

                                                                                                                  Hi everyone! We're back with a powerful plug to make your Zobot smarter and your support faster. This time, we're solving a common friction point for finance teams by giving customers quick access to their invoices. We are going to be automating invoice
                                                                                                                • Enrich your contact and company details automatically using the Data Enrichment topping

                                                                                                                  Greetings, I hope you're all doing well. We're happy to announce the latest topping we've added to Bigin: The Data Enrichment topping, powered by WebAmigo. This topping helps you automatically enhance your contact and company records in Bigin. By leveraging
                                                                                                                • Zoho Billing Partial Proration...

                                                                                                                  Hi, In Zoho billing, we offer upgrades, downgrades, suspends, and resumes. We use online payment for most subscriptions, but have only a few that are offline, but we treat them the same either way.. We prorate only parts of these changes.. Right now zoho
                                                                                                                • Any update on adding New Customer Payment Providers who support in store terminal devices?

                                                                                                                  Currently there is only one Customer payment provider listed for terminal devices in USA- Everyware. They charge a monthly fee of almost $149 minimum. Will you add other providers - like Zoho Payments or Stripe or Worldpay that would allow integrated
                                                                                                                • Getting Subform Fields to Display Top to Bottom

                                                                                                                  I have a form where the fields are all in one column. I want to insert a subform where the fields are stacked in one column as well. I have built both the form and subform but the subform displays the fields from left to right instead of a stacked column. This will cause a problem displaying the subform correctly on mobile apps. How can I do this please?' Here is my form with subform now. As you can see the subform "Follow Up Activity" is displaying the fields left to right. I need them to go top
                                                                                                                • [Feature Deprecated] Gamescope in Zoho Projects

                                                                                                                  This feature is no longer available. All work and no play makes Jack a dull boy. Well, yes. That's true! Tiring work, buzzing coffee makers, busy turnstiles , occupied meeting rooms, back-to-back status updates and what not. Your everyday routine need not have to be this hectic. It can be more rewarding. It can be more fun. Zoho Projects believes in this fact and hence has launched Gamescope - A module that turns every project into an exciting game and boosts the spirit of your team by leaps and
                                                                                                                • Creating a task, i can not work out how to get ID for What_Id

                                                                                                                  hi From Module A function I map Module B record reference membershipid (ours). I need Module B Zoho ID to create the related to on task. All examples i've seen start with the Zoho ID. void automation.LTM_Assign_Dispute_Task(String membershipid) { try
                                                                                                                • Can't connect to POP and SMTP over VPN

                                                                                                                  I use Thunderbird to access Zoho Mail via POP and SMTP. I have configured an app-specific password for Thunderbird to use. Everything is properly configured and works correctly — until I enable my VPN. I'm using a paid commercial VPN service (Mullvad).
                                                                                                                • my zoho mail is hacked

                                                                                                                  my email is sending my username and password to people i dont know
                                                                                                                • Suministrar accesos

                                                                                                                  Asunto: Enable UPLOAD_RULE for API Attachments Mensaje: Hola, Necesito habilitar la capacidad de subir attachments vía API en Zoho Mail. Actualmente recibo el error: "UPLOAD_RULE_NOT_CONFIGURED" al usar el endpoint: POST /api/accounts/{accountId}/messages/attachments
                                                                                                                • Introducing real-time document commenting and collaboration in Zoho Sign

                                                                                                                  Hi, there! We are delighted to introduce Document commenting, a feature that helps you communicate with your recipients more efficiently for a streamlined document signing process. Some key benefits include: Collaborate with your recipients easily without
                                                                                                                • Ticket Loop from Auto Responce

                                                                                                                  We’re seeing an issue where a client’s help desk triggers a ticket loop. When they email us, our system creates a ticket and sends the acknowledgement. Their system then creates a ticket from that acknowledgement and sends one back to us, which creates
                                                                                                                • How to make rule to filter out some emails from receiving the notification "Receiving a new ticket"?

                                                                                                                  Good Morning, We are currently using the "Receiving a new ticket" notification rule to provide that auto reply template to a customer that emails us for support. However, there are some emails we don't want to receive this auto-reply. Where can I set
                                                                                                                • Zoho Signatures Missing

                                                                                                                  In the past after collecting signatures from two different PDFs I would merge them by calling an api and the signatures would appear in the combined PDF. Recently the signatures have disappeared whenever I combine the PDFs together. Why did this randomly
                                                                                                                • Filter button in the Zoho Sheet Android App doesn't toggle on

                                                                                                                  I am a new Zoho Sheets user and experiencing a specific issue with the filter functionality in the Android mobile application. Detailed Issue Description: The filter icon appears correctly in the toolbar. Upon tapping the filter icon/button, the toggle
                                                                                                                • Automate pushing Zoho CRM backups into Zoho WorkDrive

                                                                                                                  Through our Zoho One subscription we have both Zoho CRM and Zoho WorkDrive. We have regular backups setup in Zoho CRM. Once the backup is created, we are notified. Since we want to keep these backups for more than 7 days, we manually download them. They
                                                                                                                • Exciting Updates to the Kiosk Studio Feature in Zoho CRM!

                                                                                                                  Hello Everyone, We are here again with a series of new enhancements to Kiosk Studio, designed to elevate your experience and bring even greater efficiency to your business processes. These updates build upon our ongoing commitment to making Kiosk a powerful
                                                                                                                • New activity options for workflows

                                                                                                                  Greetings, We are excited to announce the addition of two new dynamic actions to our workflow functionality: Create Event and Schedule Call. These actions have been thoughtfully designed to enhance your workflow processes and bring more efficiency to
                                                                                                                • Remove the “One Migration Per User” Limitation in Zoho WorkDrive

                                                                                                                  Hi Zoho WorkDrive Team, Hope you are doing well. We would like to raise a critical feature request regarding the Google Drive → Zoho WorkDrive migration process. Current Limitation: Zoho WorkDrive currently enforces a hard limitation: A Zoho WorkDrive
                                                                                                                • Kaizen #228 - Process Large-Scale Migrated Data Using Catalyst Solutions

                                                                                                                  Howdy, tech wizards! This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process. Why Catalyst Solutions? Catalyst
                                                                                                                • Next Page