Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.







                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Introducing Assemblies and Kits in Zoho Inventory

                                                                                                                  Hello customers, We’re excited to share a major revamp to Zoho Inventory that brings both clarity and flexibility to your inventory management experience! Presenting Assemblies and Kits We’re thrilled to introduce Assemblies and Kits, which replaces the
                                                                                                                • Does the ability exist to make tax on the customer profile mandatory?

                                                                                                                  I am reaching out to inquire about the possibility of making the "Customer Tax" field mandatory when creating a new customer in Zoho. We want to ensure that all customers have their tax information recorded to maintain compliance with our internal processes.
                                                                                                                • email association with CRM

                                                                                                                  Why is it 2024 (almost 2025) and Zoho has not figured out how to integrate email with CRM? It is so inconsistent at associating emails within CRM. I am an attorney. I have clients and work with other attorneys. Attorney John Doe is associated with multiple
                                                                                                                • Credit Note for Shipped and Fatoora pushed invoices

                                                                                                                  We have shipped a Sales Order and created an Invoice. The Invoice is also pushed to Fatoora Now we need to create a credit note for the invoice When we try it, it says we need to create a Sales Return in the Zoho Books, we have already created a Sales
                                                                                                                • FSM - Timesheet entires for Internal Work

                                                                                                                  Hi FSM Team, Several of my clients have asked how they can manage internal timesheets within Zoho FSM. Since their technicians already spend most of their day working in FSM, it would be ideal if they could log all working hours directly in the FSM app.
                                                                                                                • Add a way of clearing fields values in Flow actions

                                                                                                                  It would be great if there was an option to set a field as Null when creating flows. I had an instance today where I just wanted to clear a long integer field in the CRM based on an action in Projects but I had to write a custom function. It would be
                                                                                                                • Role Management

                                                                                                                  I am creating an analytics dashboard for a company that will be utilized by its various departments such as Finance, Marketing, and HR. My goal is to design the dashboard with separate tabs for each department. Additionally, I plan to implement role-based
                                                                                                                • Announcing new features in Trident for Windows (v.1.37.5.0)

                                                                                                                  Hello Community! Trident for Windows just received a major update, with a range of capabilities that strengthen email security and enhance communication. This update focuses on making your mailbox safer and your overall email experience more reliable.
                                                                                                                • Early Payment Discount customize Text

                                                                                                                  Hi, I’m currently using Zoho Books and am trying to customize the standard “Early Payment Discount” message that appears in the PDF invoice template. I’ve reviewed the documentation here: https://www.zoho.com/books/help/invoice/early-payment-discount.html
                                                                                                                • Deprecation of SMS-based multi-factor authentication (MFA) mode

                                                                                                                  Overview of SMS-based OTP MFA mode The SMS-based OTP MFA method involves the delivery of a one-time password to a user's mobile phone via SMS. The user receives the OTP on their mobile phone and enters it to sign into their account. SMS-based OTPs offer
                                                                                                                • DKIM Now Mandatory - Changes to Zoho Forms Email Policies

                                                                                                                  Hello Zoho Forms Users, This post is to inform you about an important update regarding the authentication of all email domains in your Zoho Forms account. This year, we are doubling down on our commitment to deliver a secure, seamless, and empowering
                                                                                                                • Call description in notes

                                                                                                                  When completing a call, we type in the result of the call in the description. However, that does not show up under the notes history on the contact. We want to be able to see all the calls that have taken place for a contact wihtout having to go into
                                                                                                                • Email Address for Contact not Populating

                                                                                                                  When I click "Send Mail" from a Contact's page, their email address does not auto populate the "To" field. How do I make this happen?
                                                                                                                • New in CRM: Dynamic filters for lookup fields

                                                                                                                  Last modified on Oct 28, 2024: This feature was initially available only through Early Access upon request. It is now available to all users across all data centers, except for the IN DC. Users in the IN DC can temporarily request access using this form
                                                                                                                • Why hybrid project management might be the best fit for you?

                                                                                                                  Project management techniques are designed to equip teams with proven methods for easy and efficient project execution. While management teams may have apprehensions about adopting the hybrid method of project management, we’ve compiled the top reasons
                                                                                                                • Allow all Company Users to view all projects, but only owner/admins can change projects

                                                                                                                  I was wondering if there was a permission setting I could adjust to allow all our company users to see all projects created. Then, only the project owners and admins with the change permission. Thanks
                                                                                                                • Seeking help to be able to search on all custom functions that are defined

                                                                                                                  Hello I have a lot of custom functions defined (around 200) and i would like to search some specific strings in the content of those. Is there a way to accomplish that? If not, is there a way to download all existing custom functions in some files locally
                                                                                                                • Add Bulk Section / Grid Layout Duplicate Feature in Zoho Forms Builder

                                                                                                                  Currently in Zoho Forms, users can only duplicate individual fields. There is no option to duplicate an entire section or two-column/grid layout with all internal fields. This becomes inefficient when building structured forms such as Family Details,
                                                                                                                • Leistungsdatum in Rechnungen (Zoho Books)

                                                                                                                  Hallo, ist es irgendwie möglich den Leistungszeitraum in der Rechnung aufzuführen? Beste Grüße Aleks
                                                                                                                • Zoho Trident Windows - Streams Not Visible

                                                                                                                  Namaste We’re having an issue with Streams not being visible in Trident (Windows), which is important for us as we share many emails internally. It appears that the feature to show Streams above the Inbox folder, as seen in the default mailbox view, is
                                                                                                                • Sales IQ Chat Widget is Only Displaying Last Name

                                                                                                                  Can anyone suggest why the widget is only displaying "last name"?! We have the latest version of the wordpress plugin installed. Thanks Thanks!
                                                                                                                • Shopify - Item sync from Zoho Inventory

                                                                                                                  Hi team, We’ve connected Shopify with Zoho Inventory. We want that when an item is created in Zoho Inventory, it must create a product in Shopify. But currently, new items created in Zoho Inventory are not getting created in Shopify even after clicking
                                                                                                                • Kill zoho meeting

                                                                                                                  Saying the quiet part out loud. Can zoho please just give up on the idea that they can make a meeting platform and just make our workplace licenses cheaper when you remove it so people can switch to zoom or teams. Tired of the excuses, you guys cant make
                                                                                                                • Utilisation de Zoho en conformité avec l’article 286 du Code général des impôts (CGI)

                                                                                                                  Cher(e) client(e), Conformément à l’article 286 du Code général des impôts (CGI) impose aux entreprises assujetties à la TVA d’utiliser des systèmes de caisse ou de gestion commerciale certifiés lorsqu’elles enregistrent des ventes à des particuliers.
                                                                                                                • Unable to Create Task as a Support Administrator

                                                                                                                  Hello! I want to ask for help regarding creating tasks within the tickets. I am by default the Support Admin. I should be able to create tasks or activities right? But there's a prompt that I need to contact the Administrator. See photos for reference.
                                                                                                                • Introducing Forms in Zoho Sheet

                                                                                                                  We hereby bring you the power of ​forms in Zoho Sheet. ​Now, build and create your own customized forms using Zoho Sheet. Be it compiling a questionnaire or rolling out a survey, Zoho Sheet can do it all for you. Forms is an excellent feature that helps you collect information in the simplest of ways and having it in Zoho Sheet takes it a notch higher. Build Simple yet Powerful forms Building forms using Zoho Sheet is fairly simple. The exclusive 'Form' tab lets you create one quickly. Whether you
                                                                                                                • Layout one survey question in a time & redirect next Page based on previous response

                                                                                                                  I have doubt while, I am scripting survey on the Zoho where I redirecting to next page based on my previous response but didn’t get success on this. Please help me on this and tell me how I layout one survey questions in a time when I submit response
                                                                                                                • Zoho Bookings form pre-filled with Zoho Forms in

                                                                                                                  Hi, I've got a contact page on my website and I'd like to have the option to book an appointment (redirected to zoho bookings page) after an option is submitted on the contact form. how would I go about doing this? thanks
                                                                                                                • Support “Other” Option with Free Text in Dropdown Fields

                                                                                                                  Hello Zoho Bookings Team, Greetings, We would like to request an enhancement to the registration form fields in Zoho Bookings, specifically for dropdown fields. Current Limitation: At the moment, dropdown fields do not support an “Other” option that allows
                                                                                                                • Detailed Balance Sheet for tax preparer

                                                                                                                  I'm using the free edition of Zoho Books. My tax preparer is asking for "detailed" Profit & Loss and Balance Sheet reports which include all the activity and transactions within the various categories. The default reports do not include these details.
                                                                                                                • Sending automated messages that appear in the ticket's conversation thread

                                                                                                                  Good morning, esteemed Zoho Desk community, warm greetings Today I am here to raise the following problem, seeking a solution that I can implement: I need to implement an automation that allows me to send reminder messages to customers when I am waiting
                                                                                                                • Payment Card or Identity form-fill from Vault?

                                                                                                                  Hello! I'm working on replacing Bitwarden with Vault and one issue I've run into is that I can't find any option to fill address and payment forms from Payment Card or Identity info that has been saved in Vault. Is there a way to do this? Is it a planned
                                                                                                                • Ability to add VAT to Retainer Invoices

                                                                                                                  Hello, I've had a telephone conversation a month ago with Dinesh on this topic and my request to allow for the addition of VAT on Retainer Invoices.  It's currently not possible to add VAT to Retainer Invoices and it was mutually agreed that there is absolutely no reason why there shouldn't be, especially as TAX LAW makes VAT mandatory on each invoice in Europe!   So basically, what i'm saying is that if you don't allow us to add VAT to Retainer Invoices, than the whole Retainer Invoices becomes
                                                                                                                • Time Log Reminder

                                                                                                                  Tracking the time spent on tasks and issues is one of the most important functions of a timesheet. However, users may forget to update the time logs because they have their own goals to achieve. But, time logs must be updated at regular intervals to keep
                                                                                                                • [Early-access] Introducing Zoho's CommandCenter - Cross-Zoho business process automation

                                                                                                                    Resources to help Webinar recording | Documentation  Feature Restrictions Currently available on early-access only for US data center accounts Features Role CommandCenter as a Service uses signals across Zoho services to propel the movement of records
                                                                                                                • Tip #58- Accessibility Controls in Zoho Assist: Learning- 'Insider Insights'

                                                                                                                  Learning should be clear and interruption-free for everyone. Timely feedback plays an important role in helping users understand actions as they happen, without breaking their focus. In this post, we’ll explore the final section of Accessibility: Learning.
                                                                                                                • ZIA "Generate Content" action doesn't have contexual data from the ticket

                                                                                                                  "Generate Content" action doesn't have contexual data from the ticket. I try to get AI to help me with this ticket but it doesn't seem to have any ticket information as context. Although the ticket has a lot of information in it.
                                                                                                                • Zoho Desk - Zoho FSM Integration issue on Mobile and iPad

                                                                                                                  Hello Team, I am trying to create a Work Order (WO) using the Zoho FSM integration (Add-on Service) that is integrated with Zoho Desk. The issue is that the integration is not working on mobile devices and iPads. While I am able to create the WO, Request,
                                                                                                                • E-File Form 1099 Directly With the IRS From Zoho Books

                                                                                                                  The Form 1099 filing season has begun, and businesses are required to e-file certain forms with the IRS to report payments made to vendors and contractors. If your business made qualifying payments during the year, you must e-file the appropriate Form
                                                                                                                • Suggestions for showing subscribed Topics in CRM (contact record)

                                                                                                                  We have several Topics set up in ZMA. We also have a sync set up between ZMA and CRM. I'd like to display the subscribed topics on the CRM Contact record. This will allow the Sales team (who uses CRM) to see at a glance what topics a Contact is subscribed
                                                                                                                • Next Page