Kaizen 123- Data Synchronization from third party application

Kaizen 123- Data Synchronization from third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

          Zoho Developer Community




                                    Zoho Desk Resources

                                    • Desk Community Learning Series


                                    • Digest


                                    • Functions


                                    • Meetups


                                    • Kbase


                                    • Resources


                                    • Glossary


                                    • Desk Marketplace


                                    • MVP Corner


                                    • Word of the Day



                                        Zoho Marketing Automation


                                                Manage your brands on social media



                                                      Zoho TeamInbox Resources

                                                        Zoho DataPrep Resources



                                                          Zoho CRM Plus Resources

                                                            Zoho Books Resources


                                                              Zoho Subscriptions Resources

                                                                Zoho Projects Resources


                                                                  Zoho Sprints Resources


                                                                    Qntrl Resources


                                                                      Zoho Creator Resources



                                                                          Zoho Campaigns Resources


                                                                            Zoho CRM Resources

                                                                            • CRM Community Learning Series

                                                                              CRM Community Learning Series


                                                                            • Kaizen

                                                                              Kaizen

                                                                            • Functions

                                                                              Functions

                                                                            • Meetups

                                                                              Meetups

                                                                            • Kbase

                                                                              Kbase

                                                                            • Resources

                                                                              Resources

                                                                            • Digest

                                                                              Digest

                                                                            • CRM Marketplace

                                                                              CRM Marketplace

                                                                            • MVP Corner

                                                                              MVP Corner





                                                                                Design. Discuss. Deliver.

                                                                                Create visually engaging stories with Zoho Show.

                                                                                Get Started Now


                                                                                  Zoho Show Resources


                                                                                    Zoho Writer Writer

                                                                                    Get Started. Write Away!

                                                                                    Writer is a powerful online word processor, designed for collaborative work.

                                                                                      Zoho CRM コンテンツ






                                                                                        Nederlandse Hulpbronnen


                                                                                            ご検討中の方





                                                                                                  • Recent Topics

                                                                                                  • Subform Time field to string.

                                                                                                    Good afternoon All. I have a Subform 'Delivery_Receiving_Hours' that captures Day (Dropdown), Time_Open (Time), and Time_Close (Time). I need to capture this data and send it to a multiline field in the CRM. The code, posted below, below will capture
                                                                                                  • workflow for bounced email gets triggered, but email is status = opened

                                                                                                    Hello, I have a workflow that sends me an email if outgoing email are bounced. Now I got some kind of this emails, but the corrosponding contacts have status = open at the email. Why this bounce-workflow is triggered? Reports > Email Reports > Bounce
                                                                                                  • Data export

                                                                                                    I need to export our customer's data and projects' data for our purpose but am unable to export full data i only get around 3160 projects and around 2k customer can you please help me to get full data, please
                                                                                                  • Adjusting Physical Inventory

                                                                                                    Not getting very far with support on this one, they say they are going to fix it but nothings happened since November. Please give this a thumbs up if you would like to see this feature or comment if you have some insight. Use Case: Inventory set to be
                                                                                                  • Zoho Marketing Plus : Un outil tout-en-un pour la création de pages, la collaboration et la gestion du calendrier marketing

                                                                                                    Nous sommes ravis de vous présenter trois nouvelles fonctionnalités puissantes de Zoho Marketing Plus s’enrichit désormais d’un page web (l'éditeur de pages), qui vous permet de créer des pages attrayantes et à fort taux de conversion pour vos campagnes
                                                                                                  • Grouping payments to match deposits

                                                                                                    Is there a way to group multiple invoice payments together so they match credit card batches and grouped deposits in the bank account? Basically, we are creating invoices for each of our transactions, and applying a payment to each of the invoices. Our payments are either credit cards or checks. We want to be able to group payments together so when our bank account reflects a credit card batch made up of many transactions, or the deposit we took to the bank that has multiple checks from different
                                                                                                  • Employees can not add some expenses suddenly

                                                                                                    Zoho expense was working fine and whenever there was a new merchant, it would automatically add and also the same auto added in Zoho Books (due to merchant-vendor sync) untill now. From today, it is having problems in searching the existing vendors and
                                                                                                  • Zoho email setup in office365

                                                                                                    When i am trying to setup zoho mail setup using my domain in office365 and it is not working and it says that we couldn't log on to the incoming (IMAP) server and please check your email address and password and try again. I was able to login using my
                                                                                                  • iOS 10: Caller ID new feature?

                                                                                                    Hi, in the update history of the iOS App (for iOS10) - v.3.2 - i found the point "caller identification" has this feature been deactivated again? i cannot find anything on my iphone on how to activate this feature. or does it just work from the beginning?
                                                                                                  • Recommendations to store meeting notes for easy access from Contacts, Accounts & Deals module records?

                                                                                                    I would like your advice on how to achieve this use case for my organization. It’s related to where/how best to store meeting notes from a conversation with Contact(s) working at an Account (Company) in the context of a Deal. The ideal solution (from
                                                                                                  • Bank reconciliation. Match Transaction -filter

                                                                                                    When matching an imported bank statement file we only get a match if it is an excact match on both amount and date. Then a suggestions comes up with a very broad selection regarding amount, and no default "between" dates. I can then go an manually adjust the filter, and have to put in from-to amounts and dates. How do I set a default from-to date?  As an example, I would like the date to be +- 3 days, Thanks.
                                                                                                  • Added new staff but does not appear in other organization list

                                                                                                    Hi, I added the new staff under Sales Manager in the contacts, but it does not appear in the other organization list where I need to create a contact, and I can't select the newly added Sales Manager
                                                                                                  • Integrating Calendly with Zoho Calendar in Zoho Mail

                                                                                                    I moved my office into a business incubator space that uses Calendly for meeting management and events. Calendly doesn't have a integration with Zoho Calendar and vice versa. I was directed to Zapier for integration but it doesn't have an integration
                                                                                                  • Map fields from module X to a lookup field in subform in module Y

                                                                                                    Hi there In the 1st screenshot attached, you can see a subform in myLeads module. You can see that there is a number already filled there - that is the 'Property ID' and it is a single line field. It is the 'Property ID' of an entry I have in another
                                                                                                  • 🎄 Jingle, Mingle, and Automate: Spread Christmas Cheer with Zoho Desk Auto-Replies! 🎄

                                                                                                    Hello Everyone! Welcome to this week's episode of the Community Learning Series. Christmas is in the air, and I’m sure we can all feel the jingle and the mingle of the season! The folks at Zylker Techfix are no exception—they’re busy with holiday plans
                                                                                                  • how to create a new line in string in Client Script?

                                                                                                    I want to show an alert using client script, I need to add a new line in String, I assume I can use \n\n inside a string, but unfortunately it doesnt work ZDK.Client.showAlert("First Line \n\nI expect this is in second line");
                                                                                                  • Surely it's time Inline editing from views

                                                                                                    I think the first request I found for in-line editing from grids was approximately 12 years ago - that post was locked because it was suggested Zoho sheetview solved the problem. However, it's now 2024, and in-line editing from grids is just a basic expectation.
                                                                                                  • Multi branding issue with sender addresses

                                                                                                    Hello, I'm currently working on a project involving two (seperate) brands. Named 'Windeck' and 'Prolance'. They've chosen CRM Plus and I'm currently working on CRM, SalesIQ, Social and Marketing Automation. So far, I'm able to make enough separations
                                                                                                  • How to Replace an Assessment in a Job Opening on Zoho Recruit

                                                                                                    Hi everyone, I’m currently using Zoho Recruit and would like to replace the assessment linked to a specific job opening. I want to remove the existing assessment and add a new one. What is the best way to do this without losing any important data or affecting
                                                                                                  • Is there API Doc for Zoho Survey?

                                                                                                    Hi everyone, Is there API doc for Zoho Survey? Currently evaluating a solution - use case to automate survey administration especially for internal use. But after a brief search, I couldn't find API doc for this. So I thought I should ask here. Than
                                                                                                  • Email Campaigns overview page is missing SENT DATE and # people sent to!

                                                                                                    I would like to see the date the email campaign was sent, so I can understand and track when each email campaign was sent. Right now, unless you go to a contact who received a campaign, you cannot see when the campaign was sent (!!!!!!). So, if my boss
                                                                                                  • SEO recommendation of H1 tag for website tittle

                                                                                                    The exact words are “ It is good practice to place the page title inside the H1tag.” Now I already have one H1 tag on my website but it is not website tittle. In the SEO recommendation that is clear too that I have h1 tag on my page. Now I don’t know
                                                                                                  • How to choose other payment methodes than creditcards

                                                                                                    We have connected stripe as a payment provider in zoho books, booking, commerce and checkout. In stripe we selected al major payment methodes for Belgium (mainly bancontact). However, at checkout customers seems to have only the possibility to pay with
                                                                                                  • Introducing Zia LLM: Zoho’s in-house Generative AI solution for CRM's AI capabilities

                                                                                                    Hello everyone, We're excited to announce the launch of our in-house Large Language Model (LLM) by Zia to power our AI offerings. What is LLM? LLM stands for Large Language Model, a powerful AI technology that processes and generates human-like text based
                                                                                                  • How to call a Creator function which is in a different Creator application?

                                                                                                    How to call a Creator function which is in a different Creator application?
                                                                                                  • Can the code in my "Successful form submission" WF be invoked from a function?

                                                                                                    Can "Successful form submission" be invoked from a function? Data gets into a form manually and programatically. My code in "successful form submission" is good and I want to reuse it/call it, from another function which does Insert Into How to achieve
                                                                                                  • Kaizen #169 - Serialization and Schema Management in Queries

                                                                                                    Hello everyone! Welcome back to another post in the Kaizen series! In Kaizen #166, we discussed handling Variables in Queries and associating the query in Kiosk. This week, we will discuss Serialization and Schema management in Queries. Business Scenario
                                                                                                  • Introducing Keyboard Shortcuts for Zoho CRM

                                                                                                    Dear Customers, We're happy to introduce keyboard shortcuts for Zoho CRM features! Until now, you might have been navigating to modules manually using the mouse, and at times, it could be tedious, especially when you had to search for specific modules
                                                                                                  • Feature Request: Notebooks within notebooks (Tree-like structure)

                                                                                                    Dear Zoho! I already migrated all my stuff from Google Keep, Im really fond of Zoho Notebook so far. One thing that could make the service much more powerful is multi-level notebooks (or tree like structure). For example, entering into Notebook named
                                                                                                  • Can't get authorization for Sandbox environment using the self client

                                                                                                    Hello, After creating a self client, and following the client-credentials method (as it's not optimat to manually generate a code for every 10 minutes), after inputting the sandbox org id for SOID parameter, im getting the error: "error": "no_org". For
                                                                                                  • Create landing pages from Zoho Marketing Plus

                                                                                                    Hey everyone, Over the last few months, we've introduced various features and enhancements to bolster the marketing capabilities of Zoho Marketing Plus and make it simpler for everyone. To that end, we're excited to announce that Zoho LandingPage is now
                                                                                                  • Custom service report or Zoho forms integration

                                                                                                    Hello, So far the experience with Zoho FSM and the integration with Books has been good, however there are limitations with service reports. As with my business, many organisations send technicians to different types of jobs that call for a different
                                                                                                  • Email tracking subdomain

                                                                                                    The Email Tracking configuration screen of the ZeptoMail asks for a subdomain. I have gone through the documentation but could not find more information about how that subdomain is used by ZeptoMail to track the emails. Can someone throw some light about
                                                                                                  • Chart View group X-axis values above a value

                                                                                                    I have a data set with X values ranging from 0 up to 300-400, the Y values are an AVG of the values for the given X. I am interested in the values at the low end of the scale, say 0-10 and want the X values 10 and greater to be grouped into a single category
                                                                                                  • How do I get the Text Account name instead of the Reference number?

                                                                                                    Good Morning everyone! I am very new to zoho analytics. I'm trying to create some pivot tables and when I add the Account or Division name it comes up as a reference number. From my research, I need to use a lookup. I'm having a really hard time understanding
                                                                                                  • Custom API - Need to create a string return value, not only MAP

                                                                                                    @Support: When creating a Custom API it only allows a return from a function of MAP type. The service I'm using requires a string return, how can this be achieved?
                                                                                                  • missing video-urgent

                                                                                                    hi..I have a problem regarding zoho meeting. I already record almost 2hrs for my interview session. After end my session, i'm stop the recording. Its happened when i didn't received any meeting recording at my email.But i received recording 7minutes after
                                                                                                  • Duplicate New Ticket Emails

                                                                                                    I am getting two Ticket emails when a ticket is created. The first one goes to me as the Admin helpdesk@ And because I am an Agent as well it also send to helpdesk@, support1@ and support2@ Is there any way to stop this?
                                                                                                  • Ticket Response Email Template

                                                                                                    Is there a way to have an email template automatically selected when I click on Reply for a ticket?
                                                                                                  • Unpublishing Job Post for LinkedIn

                                                                                                    Hi, We have experienced issue with the above. We did unpublish a job in Recruit but it still appear under Job section in our LinkedIn page. It did not remove from LinkedIn as it should be.  Please check and advise. Regards, Snyder 
                                                                                                  • Next Page