Kaizen #123 Data Synchronization from a third party application

Kaizen #123 Data Synchronization from a third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

        All-in-one knowledge management and training platform for your employees and customers.






                              Zoho Developer Community




                                                    • Desk Community Learning Series


                                                    • Digest


                                                    • Functions


                                                    • Meetups


                                                    • Kbase


                                                    • Resources


                                                    • Glossary


                                                    • Desk Marketplace


                                                    • MVP Corner


                                                    • Word of the Day


                                                    • Ask the Experts



                                                              • Sticky Posts

                                                              • Kaizen #198: Using Client Script for Custom Validation in Blueprint

                                                                Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
                                                              • Kaizen #226: Using ZRC in Client Script

                                                                Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this
                                                              • Kaizen #222 - Client Script Support for Notes Related List

                                                                Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how
                                                              • Kaizen #217 - Actions APIs : Tasks

                                                                Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.
                                                              • Kaizen #216 - Actions APIs : Email Notifications

                                                                Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are


                                                              Manage your brands on social media



                                                                    Zoho TeamInbox Resources



                                                                        Zoho CRM Plus Resources

                                                                          Zoho Books Resources


                                                                            Zoho Subscriptions Resources

                                                                              Zoho Projects Resources


                                                                                Zoho Sprints Resources


                                                                                  Qntrl Resources


                                                                                    Zoho Creator Resources



                                                                                        Zoho CRM Resources

                                                                                        • CRM Community Learning Series

                                                                                          CRM Community Learning Series


                                                                                        • Kaizen

                                                                                          Kaizen

                                                                                        • Functions

                                                                                          Functions

                                                                                        • Meetups

                                                                                          Meetups

                                                                                        • Kbase

                                                                                          Kbase

                                                                                        • Resources

                                                                                          Resources

                                                                                        • Digest

                                                                                          Digest

                                                                                        • CRM Marketplace

                                                                                          CRM Marketplace

                                                                                        • MVP Corner

                                                                                          MVP Corner









                                                                                            Design. Discuss. Deliver.

                                                                                            Create visually engaging stories with Zoho Show.

                                                                                            Get Started Now


                                                                                              Zoho Show Resources

                                                                                                Zoho Writer

                                                                                                Get Started. Write Away!

                                                                                                Writer is a powerful online word processor, designed for collaborative work.

                                                                                                  Zoho CRM コンテンツ




                                                                                                    Nederlandse Hulpbronnen


                                                                                                        ご検討中の方




                                                                                                                • Recent Topics

                                                                                                                • Monthly timesheet, consolidation of time by project

                                                                                                                  I have time logs for various jobs for project. Is it possible to consolidate the time spent for each job, when I am generating a timesheet for a month? I am getting the entries of jobs done on each day when I generate a timesheet for a month For example
                                                                                                                • Sending emails from an outlook account

                                                                                                                  Hi, I need to know if it's possible to send automatic emails from an Outlook account configured in Zoho CRM and, if so, how I can accomplish that. To give you some context, I set up a domain and created a function that generates PDF files to be sent later
                                                                                                                • Struggling with stock management in Zoho CRM – is Zoho Inventory the solution?

                                                                                                                  My biggest pain point today with Zoho is inventory management. I run a retail business and reliable stock management is absolutely critical. Obviously, I need this inventory to be visible inside the CRM. At first, I tried handling it through custom modules
                                                                                                                • can't login Kiosk URGENT

                                                                                                                  already try, can't login pls help to support. thanks.
                                                                                                                • 【Zoho CRM】CRM for Everyoneに関するアップデート:関連データ機能

                                                                                                                  ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 今回は「Zoho CRM アップデート情報」の中から、CRM for Everyoneの新機能「関連データ機能」をご紹介します。 関連データ機能は、あるタブのデータを別のタブに柔軟に関連付け、異なるタブで管理されている情報を1か所にまとめて表示できます。 たとえば、組織タブとチームタブのデータを関連付けることで、必要な情報に効率よくアクセスでき、顧客理解を深めながら他チームとの連携もスムーズに行えます。 目次 1. 関連データの設定方法
                                                                                                                • Profit / margins on Sales orders / Invoices / Estimates

                                                                                                                  When we select an SKU or item name in any of these documents, much info such as invoice.line_items.rate is pulled from the item & filled into the document being worked on. If we had another lineItem DB field (hidden) auto filled at the same time: invoice.line_items.purchase_rate
                                                                                                                • Inventory to Xero Invocie Sync Issues

                                                                                                                  Has anyone had an issue with Invoices not syncing to Xero. It seems to be an issue when there is VAT on a shipping cost, but I cannot be 100% as the error is vague: "Unable to export Invoice 'INV-000053' as the account mapped with some items does not
                                                                                                                • How to activate RFQ? What if a price list has ladder price for items?

                                                                                                                  Where can I find the option to activate request for quotation? How does it work? If the item has ladder price, does it gets calculated depending on how many items are in the cart?
                                                                                                                • Can't join canal Developers Zoho User

                                                                                                                  Hello, I received an invitation to join this channel, but I get an error when I try to join it, and I get the same error when I go to the Zoho Cliq interface > Search for a channel. Is this because I don't have a license linked to this email address?
                                                                                                                • Desk Email reply - set default font / use custom font

                                                                                                                  Hello, in our e-mails, which we send to our customers, a certain font must be used (Corporate Design): Segoe UI https://en.wikipedia.org/wiki/Segoe#Segoe_UI How can this be included? How can this be set as the default font to ensure that this font is
                                                                                                                • PDF Templates - Checkbox Borders

                                                                                                                  Is there a way to remove the border of a radio/checkbox on a PDF? I'd like to use the function of checkbox but if there's no easy way to remove the border (the PDF form already has a rectangle so it gets cluttered), then I'm forced to create a single
                                                                                                                • Settings Icon No Longer in ZOHO Desk?

                                                                                                                  In ZOHO desk, there has been a gear icon for settings. as of yesterday, it is no longer there. I showed up briefly this morning but is gone again. Anybody else experiecing this?
                                                                                                                • Tip #43 - Track, Review, and Analyze Your Assist Sessions with Reports-'Insider Insights'

                                                                                                                  Did you know you can generate detailed reports for both remote support sessions and unattended access sessions in Zoho Assist? This makes it easy to monitor technician activity, measure efficiency, and review customer interactions. Let us now take a closer
                                                                                                                • Simplified Call Logging

                                                                                                                  Our organization would like to start logging calls in our CRM; however, with 13 fields that can't be removed, our team is finding it extremely cumbersome. For our use case, we only need to record that a call happened theirfor would only need the following
                                                                                                                • Sub form doesn't as formula field

                                                                                                                  Is it possible to get formula field in sub form in futures?
                                                                                                                • Week date range in pivot table

                                                                                                                  Hello, I need to create a report that breakouts the data by week.  I am using the pivot table report, and breaking out the date by week, however the date is displayed as 'Week 1 2014' format.  Is there anyway to get the actual dates in there? ex. 1/6/2014-1/12/2014 Thanks,
                                                                                                                • How do I get Status History data of my Projects?

                                                                                                                  I want to build a table in Zoho Analytics that Groups by Date, when Projects entered a certain status. I cannot find Status History or any such useful data available in the Setup of my Data Source sync. Please advise how I can achieve this?
                                                                                                                • Is it possible to hide fields in a Subform?

                                                                                                                  Since layout rules cannot be used with Subforms, is there another way, or is it even possible, to hide fields in a subform based on a picklist fields within said subform? For example, if the Service Provided is Internet, then I do not want to see the
                                                                                                                • Weekly Tips :Instantly find what you need with Attachment Viewer

                                                                                                                  Your inbox must be packed with project emails, shared notes, and scattered attachments. You are looking for one specific file—a presentation slide or maybe a media clip from a team update—but don’t want to dig through endless email threads or switch between
                                                                                                                • Missing Zoho Desk integration option for form workflows

                                                                                                                  According to the help page "Configure Zoho Desk integration in form workflows" we should be able to select Zoho Desk as an integration target but when I open the integrations list then Zoho Desk is not being listed in it. We are on the Premium plan which should already support Zoho Desk integrations.
                                                                                                                • Gantt for 2 or more projects

                                                                                                                  Hello, I'm trying the free version of your produtc. It is veryyy good!!!! I don't know if in the Standard plan, I can overview a Gantt Graph for 2 or more Projects Milestone. This would be very helpfull for managing teams and taking decisions about who I will assign a task to. In the paid plan Do I have this possibility? Thank you.
                                                                                                                • Integrating a Zoho Project Gantt Chart into Reports

                                                                                                                  Is is possible to integrate a Zoho Project Gantt Chart into a Zoho Report Dashboard. I am in the process of creating Project Status Dashboards for the projects that we track in Zoho Projects and I would like to incorporate the gantt chart within Reports.  Please let me know! Thanks
                                                                                                                • ZOHO BOOKS - EXCESSIVELY SLOW TODAY

                                                                                                                  Dear Zoho Books This is not the first time but it seems to be 3 times per week now that the system is extremely slow. I work on Zoho Books 95% of my day so this is very frustrating. Zoho you need to do something about this. I have had my IT guy check
                                                                                                                • Displaying related quotes in sales order and back

                                                                                                                  Hi, My colleague liked to see to which sales orders, the quote has been converted. Quote shows Invoices, but not SO. Same, they would like to see the quotes in the sales order, as they can see invoices, packages, shipment, How can we achieve this ? Thank
                                                                                                                • Tip of the Week #71–Auto-move incoming messages to the right inboxes with keywords

                                                                                                                  We all know that customer-facing teams, especially your sales and support teams, can’t afford to miss even a single customer conversation. But sometimes, sales queries or support requests can easily get lost in a crowded inbox or even end up in the wrong
                                                                                                                • Clearing Fields using MACROS?

                                                                                                                  How would I go about clearing a follow-up field date from my deals? Currently I cannot set the new value as an empty box.
                                                                                                                • Migrating a Zoho Forms form into Zoho Creator

                                                                                                                  Hi, How can I migrate my Zoho Forms form into Zoho Creator? Thanks. Truly, Emad
                                                                                                                • Is there any way to recall an email sent using Zoho CRM?

                                                                                                                  If an email is sent using Zoho Mail, there is a recall option/functionality that is available to the sender. Is there any way to recall an email if it was sent using Zoho CRM? I can't seem to find that option. Any help would be appreciated.
                                                                                                                • Problem with reports due to "Connected" items change - Yes this IS a problem

                                                                                                                  Now that the change has been made to use "connected" items I can no longer run the reporting I need in CRM. I should be able to start with Deals as the parent, connect down to the Account (Account_Name) on the deal as the child, then to any child items
                                                                                                                • Introducing notifications in the vendor portal

                                                                                                                  Imagine this: You're a recruiter working with multiple vendors on a high-volume hiring project. You’ve just updated a job description after a last-minute change from the hiring manager. One of your vendors, however, is still working off the older version
                                                                                                                • Analytics for notes created

                                                                                                                  Is there a way I can see how many notes were created per day? Via reporting or analytics?
                                                                                                                • Cannot update Recurring_Activity on Tasks – RRULE not accepted

                                                                                                                  Hello, I am trying to update Tasks in Zoho CRM to make them recurring yearly, but I cannot find the correct recurrence pattern or way to update the Recurring_Activity field via API or Deluge. I have tried: Sending a string like "RRULE:FREQ=YEARLY;INTERVAL=1"
                                                                                                                • Add image to report...

                                                                                                                  Greetings, I send a weekly color coded report via Creator email. I would like to add the legend somewhere in the report. Header, footer where ever. I have the legend saved on Google Drive and can access it via shared link. Sure someone has wanted to add
                                                                                                                • More controls for User Fields in CRM

                                                                                                                  Dear All, We are here with a minor but crucial enhancement to the user fields—now set accessibility permissions to the records for user field. User field allows you to extend co-ownership of records to your peers. You can collaborate with them for certain
                                                                                                                • Calls to accounts rather than leads or contacts?

                                                                                                                  So..... We have a dilemma and I'm hoping someone has encountered this before and figured out a fix. We have just migrated to Zoho. It's great.....expect for how "Calls" are handled.... We are B2B. We do not use the leads module. A "Lead/Prospect" for
                                                                                                                • Image Upload Field | Zoho Canvas

                                                                                                                  I'm working on making a custom view for one of our team's modules. It's an image upload field (Placement Photo) that would allow our sales reps to upload a picture of the house their working on. However, I don't see that field as a opinion when building
                                                                                                                • Power of Automation :: Automated 'Delayed & Closed' Status Update Based on Due Date

                                                                                                                  Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate
                                                                                                                • Range names in Zoho Sheet are BROKEN!

                                                                                                                  Hi - you've pushed an update that has broken range names. A previously working spreadsheet now returns errors because the range names are not updating the values correctly. I've shared a video with the support desk to illustrate the problem. This spreadsheet
                                                                                                                • populate email address and name in zoho desk?

                                                                                                                  Is it possible to populate the email address and name in the zoho desk widget? We only use it in the context of an authenticated user, so we already know the user's name and email. Thanks,
                                                                                                                • Are there default/pre-built dashboards in Zoho Desk?

                                                                                                                  Hi, I am looking for some pre-built dashboard templates in Zoho Desk, similar to what we can find in CRM/Projects, etc Thank you
                                                                                                                • Next Page