Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Merge invoices
This has come up three times in a month of me using the system. a client whom i do work for on a regular bases asked me to invoice separate jobs on a single invoice. I would then go and cancel the old invoices and make a new one. It would be great if i could just select the invoices and click merge to create a single invoice.Pipeline stages for product-led growth (PLG) subscription services
(Estagios de pipeline com foco em serviço digital por assinatura (PLG)) Dear colleagues, I am writing to request assistance with a significant challenge I am currently facing. I manage a digital service that operates on a subscription-based model. I amMultiselect lookup in subform
It would be SO SO useful if subforms could support a multiselect look up field! Is this in the works??Zoho CRM for Email ( Microsoft add on app) Feb 25, 2025
We are currently experiencing issues with Zoho CRM for Outlook despite updating the Microsoft add on app and Zoho software to the latest versions. When working in Outlook on our Macs and PC's and 365 on the web, when we try to link an email in OutlookSheet View for Reports
We often need to filter our contacts in a way that is too complex for Custom Filters (which now allows filtering by Related Modules - amazing! - but doesn't allow for an *OR* relationship between the multiple filters) or Custom View (which does not yetFeature enhancement: Highlight rows based on a cell value
Hello Sheet users, We're excited to announce a new feature enhacement, shaped directly by your valuable feedback! As you might know, conditional formatting is a great tool for anyone dealing with large data sets. Previously, if you’ve ever wanted to drawUrgent: Unable to Receive OTP Email for Portal User Registration in Zoho Creator
I paid to enable the Portal User feature on 2/25, and followed the official instructions (Youtube video: Customer Portals | Zoho Creator) to set up the Portal User using my Gmail account. However, I am not receiving the OTP email and am unable to successfullyZoho not receiving verification email.
I developed a website registration page and need to send verification email. I am using sendinblue for sending out the verification emails. Somehow, the mail is not being received by the Zoho users. Gmail, outlook, yahoo domains works fine but when I try to send the mail to my Zoho account I am not able to receive the mail. Can you please let me know what might be the issue?Restore deleted spreadsheet
Hi all, I can not find my spreadsheet. It seems that my spreadsheet is deleted. Please, restore the spreadsheet. Thanks in advanceLookup fields can't be used for anything important
Hi It seems the lookup fields are mostly.... informative, you can at most link stuff between modules... You can't use lookup fields in blueprints, you can't use them in layout rules or anything... It that correct?Function to check frequency of customer ordering patterns
Hi, I'm no coder, so I thought I could make use of Chat GPT to generate some Deluge script to create a function that would check order frequency of our customers so that I could send an email if they go beyond their usual order pattern. Chat GPT and ClaudeBetter Control over Team Ordering and Display
Two feature requests for Zoho Desk team drop downs: The ability to change the order of teams, currently it's not in alphabetical order, which is confusing. I'm thinking a manual ordering via number or drag and drop, alphabetical order, and creation orderClear specific tag or tags from all leads
I'm looking for a way to clear a specific tag from all Leads. I have a function when a tag is added to a lead they are added to a campaign and their status is set to "invited". I need to find a method to remove the tag. If I try to delete the tag to clearCisco Webex Calling Intergration
Hi Guys, Our organisation is looking at a move from Salesforce to Zoho. We have found there is no support for Cisco Webex Calling however? Is there a way to enable this or are there any apps which can provide this? Thanks!Has Anyone Else Built a State Machine into a Function?
I'm in the process of building a state machine for lead lifecycle management within Zoho CRM, and I'm curious if anyone else has tried integrating a state machine into their Deluge functions, especially considering Zoho’s 12-function cap. I’ve managedIs there a way for a custom function to trigger a custom made email alert?
Hi Zoho Team, We have currently maxed out our 30 workflows in one of our department and we are trying to find ways to make it more efficient. We usually use our workflows to send automated email notifications per team leader for approval of tickets. IsUnified WhatsApp Number Management in Zoho Desk and SalesIQ
Dear Zoho Desk Support Team, We are currently utilizing both Zoho Desk and Zoho SalesIQ for our customer support operations. While both platforms offer WhatsApp integration, we are facing challenges due to the requirement of separate WhatsApp numbersMulti User Lookup field on the Zoho CRM iOS app.
Hello all! Hope you are all safe and sound. We have now brought in support for the Multi User Lookup field on the Zoho CRM mobile app. Below are the enhancements that are supported in this update: * Multi User Lookup field can be added through theHelp with Deluge Script in Zoho SalesIQ (Input Variables Always Null)
Hi everyone, I’m working on a Deluge script for Zoho SalesIQ, specifically for a block in Zobot. The script is supposed to: Check the visitor’s name when they start a chat. If the name is empty, return false; if a name is present, return true. However,Item/service subtotal
Just discovered & really pleased that we can drag to re-order the line items in Sales orders & Invoices, a very nice feature which doesn't seem to be documented? It would be nice to be able to insert a subtotal as a line item to complete this great featureHide/Show Certain Fields When Creating/Editing Invoices, Estimates, Etc.
Hello there, I would like the ability to hide or show certain fields for different employees/roles in my company when they are creating or editing invoices, estimates, etc.. Most of the time, we don't use half of the fields shown on the create/edit invoiceIs there a way to make an account inactive in ZoHo Desk
We have a few Clients "Accounts" that we no longer do business with. It would be beneficial for them to not show up in lists. However, we want to go back and view tickets, time, etc.Any way to Associate a Recurring Invoice with Deal/Potential in Zoho CRM
When we create an Invoice in Zoho Books, we have the option to associate that invoice with a Potential/Deal that is tied to that same customer. Can we do the same thing with Recurring Invoices? When creating a Recurring Invoice, I want to be able to AssociateSSO/SAML
Question - how are you guys handling SSO for Desk? It seems the only options lead back to ADFS which is a dying technology, stupidly complicated to set up and configure (I mean, you can put that on your resume) and seems like overkill for SSO when AAD does it so much better and effortlessly. We want to have our clients have SSO to log into our support portal and we want our Agents to have SSO as well. I shouldn't have to expose our server, turn on ADFS, configure it, install multiple clients (SyncAre downloadable product available in Zoho Commerce
Hi all. We're considering switching to Zoho Commerce for our shop, but we sell software and remote services. Is there a features for downloadable products? I can't find any information about this. Thank you very much AliceAsk the Experts 17: Elevate Customer Communication With Multichannel & Instant Messaging
Hello Everyone, Get ready for the opportunity to interact with Zoho Desk specialists! This month, we’re diving into Multichannel Operations, helping you manage customer interactions across various platforms seamlessly. Why Join? Customers reach out throughZOHO DESK
I use Outlook Exchange to receive emails and want to forward them to Zoho Desk for better management. Additionally, I want to reply to emails in Zoho Desk and have them sent to customers via Outlook Exchange, but I don’t know how to configure it.Automating Custom Web Link Messages to Customers via Instant Messaging in Zoho Desk
Hello, I am looking for assistance with Zoho Desk's Instant Messaging feature. Is there a way to automate the sending of a custom web link to customers every day at 6:30 PM? Any guidance or suggestions on how to achieve this would be greatly appreciated.Exciting Update: Multi WhatsApp Business Account (WABA) Support Now Available in SalesIQ!
We’re pleased to share an important update that will enhance the way you manage your WhatsApp Business accounts (WABAs) within SalesIQ. With the launch of Multi WABA support, you can now connect and manage multiple brands more effectively, each underAn error occurred while uploading the attachment.
From the image below, attaching files is not possible via the website but is supported through the Mobile App.One time offer in Zoho Campaigns
Hello! I have an idea to improve my sales. I would like to make a "one time offer" in Zoho Campaigns with a clock that will count down the bid validity period. Is it possible? Regards.How to Initiate WhatsApp Message on SalesIQ?
I've just activated a Business WhatsApp phone number through SalesIQ because of its touted omnichannel chat approach. Sounds exciting. I understand that when a customer sends me a WA message, I can reply to it on SalesIQ and keep the chat going, perfect.Multiple self client instances
there a couple of details that deserve clarification / improvement in Zoho's OAuth 2: 1. for building true server apps - cloud to cloud REST - one has to choose Self Client when creating a new instance in API Console; Server-based Appplications is slightlyWorkdrive Oauth2 Token Isn't Refreshing
I have set up oauth for a bunch of zoho apis and have never had a problem with oauth. With workdrive i am using the exact same template i usually use for the other zoho apps and it is not working. All requests will work for the first hour then stops soZoho Desk - Delete an email conversation
When using Zoho support desk I see an option to delete and edit conversations that are comments, BUT if the conversation was automatically added as a conversation from an email, then the option to delete it seems not to be available. Is there a way toOrganizational Emails Send through Email Relay Server
Hello, It would be great if the the Organizational Emails could send through the Email Relay Server as well. I often need to send email as someone else and since we use the GMail API option I have to use the organizational emails to do this. We use Postmark for our Email Relay Server to increase deliver-ability which is something that we have had issues with when sending everything through Zohos servers. Also with postmark there is no need for each organizational email to have its own credentialsEmoji support in workdrive
Hello, We are using emojis in our filenames and folder to help visually find relevant data. Currently Workdrive is stripping out emojis and the rest of the filename (along with the suffix), making the file unreadable when they are synchronized with desktop.What is a 'user' in Zoho Surveys, and can I have multiple Offline Kiosks running?
Is a 'user' the amount of people that can make and edit surveys on the website? I basically just want to know if I am able to have multiple Offline Kiosks running on multiple devices at the same time. Either on the Basic or Plus plan which both have 1Fundation / Non profit : Could you help ?
Hi, We're about to handle a Medical research fundation, non profitable structure, what kind of prices could we hope from you ? Could you help us ? We would need Survey, automated campaign and ZIA Thanks in advanceCan´t copy fields
Hi Im trying to copy the information from a registry in a module to a new one in another module, the following code works, the thing is, it does'nt copy the rest of information in the new registry, i'm learning Deluge in an autonomy way so, i don't nowNext Page