Kaizen 123- Data Synchronization from third party application

Kaizen 123- Data Synchronization from third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere







                        Zoho Developer Community





                                              Use cases

                                              Make the most of Zoho Desk with the use cases.

                                               
                                                

                                              eBooks

                                              Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho Desk.

                                               
                                                

                                              Videos

                                              Watch comprehensive videos on features and other important topics that will help you master Zoho Desk.

                                               
                                                

                                              Webinar

                                              Sign up for our webinars and learn the Zoho Desk basics, from customization to automation and more

                                               
                                                
                                              • Desk Community Learning Series


                                              • Meetups


                                              • Ask the Experts


                                              • Kbase


                                              • Resources


                                              • Glossary


                                              • Desk Marketplace


                                              • MVP Corner




                                                        Manage your brands on social media



                                                              Zoho TeamInbox Resources



                                                                  Zoho CRM Plus Resources

                                                                    Zoho Books Resources


                                                                      Zoho Subscriptions Resources

                                                                        Zoho Projects Resources


                                                                          Zoho Sprints Resources


                                                                            Qntrl Resources


                                                                              Zoho Creator Resources



                                                                                  Zoho CRM Resources

                                                                                  • CRM Community Learning Series

                                                                                    CRM Community Learning Series


                                                                                  • Kaizen

                                                                                    Kaizen

                                                                                  • Functions

                                                                                    Functions

                                                                                  • Meetups

                                                                                    Meetups

                                                                                  • Kbase

                                                                                    Kbase

                                                                                  • Resources

                                                                                    Resources

                                                                                  • Digest

                                                                                    Digest

                                                                                  • CRM Marketplace

                                                                                    CRM Marketplace

                                                                                  • MVP Corner

                                                                                    MVP Corner







                                                                                      Design. Discuss. Deliver.

                                                                                      Create visually engaging stories with Zoho Show.

                                                                                      Get Started Now


                                                                                        Zoho Show Resources


                                                                                          Zoho Writer Writer

                                                                                          Get Started. Write Away!

                                                                                          Writer is a powerful online word processor, designed for collaborative work.

                                                                                            Zoho CRM コンテンツ






                                                                                              Nederlandse Hulpbronnen


                                                                                                  ご検討中の方




                                                                                                        • Recent Topics

                                                                                                        • Better integration between Zoho CRM and Zoho Bookings

                                                                                                          I've noticed that when a meeting which was created in Zoho Bookings is updated by a sales person in Zoho CRM, the change is not reflected back into Zoho Bookings. I have raised this with support who advised that meetings created in Bookings need to be
                                                                                                        • What is the difference between "Reply" and "Public Comment" in a ticket?

                                                                                                          Hey, my company is thinking about switching to Zoho Desk but we do not understand the difference between the "Reply" and the "Public Comment" feature. When should we use which? And why? What is the difference between them? Thank you for your help! Kindest
                                                                                                        • Tasks View for Opportunity/Jobs does not indicate "Related To" account

                                                                                                          How can ZoHo be setup so that when a task is created for an Opportunity, the subject automatically lists the underlying account? Right now, it's impossible to link the Account to the task when the task is created from the Job.  Therefore, the open task view is inadequate.  And it's a lot of maintenance to have to manually add the account to the subject...defeating the purpose of "Automated" workflow. 
                                                                                                        • Writing SQL Queries - After Comma Auto Suggesting Column

                                                                                                          When writing SQL Queries, does anyone else get super annoyed that after you type a comma and try to return to a new line it is automatically suggest a new column, so hitting return just inputs this suggested column instead of going to a new line? Anyone
                                                                                                        • can i show alert when i finish running the function?

                                                                                                          how i can show alert inside the custom function or popup notification. when i use alert inside the custom function it show this error Error at line number:  2 'ALERT' task can be used only in on load, on validate and on change actions ​
                                                                                                        • CRM calendar not syncing with Zoho Calendar

                                                                                                          The sync is not happening: This is my Zoho Calendar CalDAV synced with Outlook This is my Zoho CRM Calendar (sadly empty ...) I have enabled CalDAV Access. In Zoho Calendar, I have set up APP Calendar sync like this: What else can I check? Thank
                                                                                                        • Pipeline in Custom Modules

                                                                                                          I love the way the Sales Pipeline looks and functions with reports. I would like to add the save pipeline features and visualization to a custom module, however, I only see that these pipelines are only available for the Deals module. Is there a way to add pipelines to custom modules?
                                                                                                        • How to apply a tag to a ticket based on the to email address?

                                                                                                          I need to assign a tag to a ticket if the ticket was sent to a specific email address. For example, we have the email accounting[at]company.com forwarding into Zoho Desk. We would like all emails that were sent to this address to be tagged with an Accounting
                                                                                                        • How to work with getFieldNames formdata functions ,Any Examples

                                                                                                          I don't find any example showing usage of getFieldNames. Where do i find .is there any Help documents available
                                                                                                        • Ecommerce integration with prestashop

                                                                                                          Development of campaigns integration with prestahop. When???
                                                                                                        • Cannot associate event with other objects when creating?

                                                                                                          I am attempting to associate an Event with one or more other objects when creating it via the API. According to the API docs, the "Related_To" property is a jsonobject even though the description says "Provide the details about the entities the event
                                                                                                        • Zoho Cliq not working on airplanes

                                                                                                          Hi, My team and I have been having this constant issue of cliq not working when connected to an airplane's wifi. Is there a reason for this? We have tried on different Airlines and it doesn't work on any of them. We need assistance here since we are constantly
                                                                                                        • Validation Rules Trigger on Untouched Fields

                                                                                                          In Zoho Desk, validation rules trigger for ALL fields during an update—even fields that weren't modified in the current edit. This behavior is fundamentally different from Zoho CRM and other Zoho products, where validation rules only apply to fields actually
                                                                                                        • Departments in Zoho

                                                                                                          I'm trying to set up a department. It looks like Departments have been removed from Zoho One. Is there another option? We have our main account setup, but want to set up a separate small department that can't see our tabs.
                                                                                                        • How to disable subform records dynamically

                                                                                                          Hi Everyone I have a requirement to dynamically disable or hide subform fields based on certain criteria. However, the code I tried didn't work as expected. It either disabled or hid all the fields in all subform records or none at all. What I actually
                                                                                                        • Client Script Operate Timeout

                                                                                                          Hi Zoho, I have set a client script that use for ( i =1; length < i , i++) to fetch all of the product in order I have an order have 30+ products, seems this script will be stopped when it is checking rough 10 + products because of timeout (may be). May
                                                                                                        • Why and When to Manually Regenerate Reports?

                                                                                                          What is the protocol for needing to regenerate reports?  It is very confusing when data in reports pulls correctly sometimes, but are inaccurate other times.   Why is report regenerating necessary at all - can the report function not read the live data as inputted into the forms?  We have built lots of reports and it is very tedious and time consuming to have to regenerate the report before each use of that data to make sure it is accurate.  Is there a way to know when a report should or shouldn't
                                                                                                        • Import template from Zoho Writer

                                                                                                          I am trying to import a mail merge template - tried to import direct from my .docx file on my hard drive and the formatting went all over the place. I then imported the .docx file in my Zoho Docs and then fixed up the formatting within Zoho Writer. Can
                                                                                                        • CRM for Everyone - More Actions Option to Create Record

                                                                                                          Please consider the option create a new record for the module from the More Actions menu. I know there is an "Add New" icon further down the menu to create a record for any module, but this just seems more intuitive and could reduce the need for the "Add
                                                                                                        • Configuración

                                                                                                          Hola acabo de instalar Zoho CRM y quiero configurarlo correctamente. Al respecto me surgen algunas dudas tales como la diferencia entre: Cuentas, posibles Clientes y Contactos. ¿Conceptualmente que son cada uno? ¿Como se se relacionan entre ellos? Si
                                                                                                        • Is there a way to request a password?

                                                                                                          We add customers info into the vaults and I wanted to see if we could do some sort of "file request" like how dropbox offers with files. It would be awesome if a customer could go to a link and input a "title, username, password, url" all securely and it then shows up in our team vault or something. Not sure if that is safe, but it's the best I can think of to be semi scalable and obviously better than sending emails. I am open to another idea, just thought this would be a great feature.  Thanks,
                                                                                                        • Enhancements to Custom Connectors in Zoho Creator

                                                                                                          Hello everyone, At Zoho Creator, we believe in providing you with the necessary tools to achieve a well connected ecosystem of apps. Our Custom Connectors feature is a testament to this, enabling you to integrate with a wide range of external services
                                                                                                        • AutoScan Not Working Since April -Support says it with engineering

                                                                                                          Hi there, Autoscan has not been working on my account since April. Without this feature, completing expenses reports is laborious and error-prone. I keep asking for updates seeing as this is a critical feature, but told that it's being looked into and
                                                                                                        • Clone a Module??

                                                                                                          I am giong to repurpose the Vendors module but would like to have a separate but very similar module for another group of contacts called Buyers. I have already repurposed Contacts to Sellers. Is it possible to clone (make a duplicate) module of Vendors
                                                                                                        • Zoho books bulk update for invoices

                                                                                                          Hi Zoho team, Why only 25 invoices can be bulk update at a time. I am booking 800 to 900 invoices in a month. Please increase this limit? Please dont tell that you dont have enough request for this task. 25 limit for bulk update is too much low
                                                                                                        • Querying CloudSQL using NodeJS?

                                                                                                          How can I query CloudSQL over nodejs? Are there any rest apis from which I perform Select Queries in the data of a Workspace? In the v1 we had C#, Python, Java for CloudSQL Now I only see Java? I am confused about the overall API of Analytics, there any
                                                                                                        • How can I move a ticket from Department A to Department B?

                                                                                                          Hi there, how do I move ticket from Department A to Department B? I went in the ticket and tried to scroll down the menu under the "department" but it won't let me, I am a support admin. Did I do anything wrong? or am I doing it in the wrong place?
                                                                                                        • Sending Email with Attachment (PDF, Excel file)

                                                                                                          Hi, I'm new to Zoho CRM and I'm setting up a flow to send an Email with Attachment when user reaching a certain stage of a Deal. In detail, I've created a Blueprint which requires user to attach a PDF file when reaching a certain point of the stage and
                                                                                                        • How to Integrate Zoho CRM with Traffic Rider Mod APK Unlimited Money?

                                                                                                          Hello, I am exploring the possibilities of integrating Zoho CRM with a mobile game app. I am working on a Traffic Rider Mod APK Unlimited Money version where we want to store user engagement data like session time, high scores, and in-app interactions.
                                                                                                        • Rename Record Templates in Zoho Creator

                                                                                                          Hi Team, I need to rename the Record Templates based on the name of the employee which i have configured in Zoho Creator Deluge Script. How to do this one? // Attachments :template:PDF_ECF:Exit_Clearance_Form input.ID as PDF
                                                                                                        • Les Zoholics 2025 sont de retour

                                                                                                          Bonjour à tous, Préparez-vous pour deux jours exceptionnels au cœur de l’écosystème Zoho ! Nous avons le plaisir de vous convier à Zoholics France 2025, un événement dédié à l’innovation, à l’expertise et à la communauté Zoho. Au programme : • Présentations
                                                                                                        • Engineering Change Order Process using Tickets?

                                                                                                          Hi Zoho Community, I'm working with my company to create an Engineering Change Order/Request (ECO/ECR) process where engineers can internally create tickets for changes in a product that must be reviewed, approved and implemented by people or teams in the company.  I'm using Blueprint to create statuses and transitions for the various ticket states as the ECO is being processed, but it continually seems like Tickets aren't a good way to build an ECO process. My question is:  Does anyone have experience
                                                                                                        • Sesión "Ask me anything" Zoho en Español - 26 de junio de 2025, de 14:00 a 17:00 (en español)

                                                                                                          ¡Hola Comunidad! ¿Quieres entender mejor Zoho CRM o Zoho Desk? Nuestros expertos estarán disponibles para responder a todas tus preguntas durante nuestra sesión "Ask me anything". Puedes comentar este artículo y durante tres horas, nuestro equipo estará
                                                                                                        • Not receiving instagram verfication emails

                                                                                                          Hi- I'm not receiving Not receiving instagram verfication emails.  I have seen other users having the same issue and your answer was to contact instagram.  But instagram no longer has a support email.  
                                                                                                        • CRM Feature Request

                                                                                                          When enabling Translation in Zoho CRM I would like to see the ability to remove the language "English (US)". I have encountered a user experience issue with a client recently. The client uses English (UK) but they are based in Spain, so they have activated
                                                                                                        • Change work hours per day for employees

                                                                                                          Hello, Is there a way to modify the work hours per day for employees in Zoho projects? This would be helpful for resource allocation to more accurately see when an employee who works 35 hours a week vs 40 hours has a full schedule. Thanks.
                                                                                                        • Is there a way to set Document Owner/Sender via the API

                                                                                                          When sending requests for zoho sign, it would seem zoho uses the id of the person that created the zoho api cred to determine the owner_id, is there a way to set a default for this?
                                                                                                        • Zoho Commerce

                                                                                                          Hi, I have zoho one and use Zoho Books. I am very interested in Zoho Commerce , especially with how all is integrated but have a question. I do not want my store to show prices for customers that are not log in. Is there a way to hide the prices if not
                                                                                                        • Name Change and Delete Email ID and Alias

                                                                                                          Hello please i require urgent assistance, 1.) I would also like to change the name that appears when people receive my emails. I have an info@spacetraiders.com but when people receive my email its say Ronma Adedeji instead of either Info or Space Traiders..
                                                                                                        • PROBLEMA AL CONECTAR EL DOMINIO

                                                                                                          hola buenas tardes hoy estoy tratando de crear mi registro y no me deja conectar el dominio  me dice que el dominio ya está vinculado a una cuenta y la verdad no tiene conectado el dominio a ninguna cuenta  tengo acceso al hosting y cpanel que se compro pero al revisar los dns estan apuntando al mismo hosting del panel  del proveedor del domini ... como hago para poder crear la cuenta con ese dominio, incluso cree un registro pero no  me deja vincularlo el dominio es integrasoft.co agradezco me ayuden  
                                                                                                        • Next Page