Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Ask the Experts 18: Supercharge Self-Service: Simplify Support, Empower Customers!
Welcome to the Ask the Experts Session 18 focusing on Zoho Desk’s Self-Service features! With the AI buzzing around like busy bees, ever thought of how you can incorporate AI into Self service? Explore how Zoho Desk's Help Center, Knowledge Base (KB),Zoho Sign up forms on Wordpress website not working smoothly
We have set up the forms on the website and linked them to respective email journeys. But when wesubmit the form the email generated/sent to user is not consistent. Sometimes the journey email is sent, sometimes a email verification message is sent ,Recurring Events Not Appearing in "My Events" and therefore not syncing with Google Apps
We use the Google Sync functionality for our events, and it appears to have been working fine except: I've created a set of recurring events that I noticed were missing from my Google Apps calendar. Upon further research, it appears this is occurringGlobal Search Custom Field
I need to be able to search a custom field within a lookup field (see attached screenshot). The custom field will not glean any resultsUnparsable Date when hour side is 02
So this is a crazy error I am getting. I got this one 1-2 years ago as well, and specifically added conditional clauses to workaround that. But now I would like to create a topic and see if anyone else is getting this error. Here is how it goes: We havecomment dedicated to specific cell?
We have built a sheet for a customer, in which they can add comments to specific cells. However, when we filter the sheet it seems that the comment does not stick to the content of a cell, but the cell itself. Resulting in us not knowing to what the clientAn update to improve email delivery | Email Authentication & Relay
Dear Zoho Recruit Community, We hope this message finds you well. This post is to inform you about an important update regarding the authentication of all email domains in your Zoho Recruit account. Effective 31st March 2025, emails sent using email addressesCan't add company logo and favicon as a superadmin
I am logged in as the superadmin but can't add company logo and favicon - the fields are greyed out. Any idea as to what the problem could be?Public links for WorkDrive files
Our company is considering building an app with Creator that allows one to upload files into the WorkDrive through a Creator form, simultaneously collecting information for a content database in Creator. Ideally, we'd like to centralise the entire workflowCreator to WorkDrive workflow | Missing "Upload file" WorkDrive action in Flow or not?
I am trying to build a Creator app with a form through which files can be uploaded to WorkDrive and simultaneously added to a Creator database. As far as I understood, automating this Creator-WorkDrive workflow can best be done with Flow, offering greaterHow can I edit the Picklist History?
I was working with some custom functions in CRM that inadvertently triggered unwanted changes in a picklist history for the Accounts module. It is absolutely essential for the business that I roll these changes in the subform back because they throw offZoho Desk Partners with Microsoft's M365 Copilot for seamless customer service experiences
Hello Zoho Desk users, We are happy to announce that Zoho Desk has partnered with Microsoft's M365 to empower customer service teams with enhanced capabilities and seamless experiences for agents. Microsoft announced their partnership during their keynotehow to send recurring estimates instead of recurring invoices
Hello, Is it possible to use the recurring invoice functionality to send recurring estimate to our customers ? We first need to send them an invoice in order to receive their purchase order and then send them the invoice with their purchase order numberPetty cash discrepancy
How do I record a petty cash discrepancy? We had money go missing and need to document that in the books, but I'm not sure how to put that in. It's not an expense, just a loss of cash.Alternative / optional Position
How do you create an alternative position or an optional position (article) in offers?Option to copy/duplicate Custom Ticket views
Hi Team, Hope you're all well. I was wondering if you would consider a feature on Custom Ticket views: The option to copy or duplicate an existing custom ticket view It would help tremendously for views with a lot of criteria that could be reused multipleHundreds of folders or tags (for each client): best practices?
I'm preparing to switch from Gmail Workspace where each of my clients has their own label, assigned automatically after I manually set it up when someone becomes a customer. I know that those labels can be converted into folders in Zoho Mail, resultingAnnouncing new features in Trident for macOS (v.1.6.0)
Hello everyone! Trident for macOS (v.1.6.0) is here with new features to elevate your workplace communication and productivity. Let's take a quick look at them. Create filters from emails . In addition to creating email filters from Mail Settings, youChanging Color Theme of Guided Conversations
Hello, We have recently added Guided Conversations to one of our websites, but I am wondering if there is a way to customize the color scheme so it matches the appearance of the website? Thank you in advance!How do I add new line and format text in zoho desk ticket?
I am generating a ticket via API and looking to update "description" field of ticket whenever I need. Is there a document where I can find supported formatting for description field on a ticket. I need to find: Adding a new line and adding bold text,Using Snippets in Comments
One of our Agents asked about using snippets in comments. We don't see this as an option at this time, only in reply mode. Is there a way to enable this?zet pack not working
We are using the zet pack command to package our Zoho extension. However, after running the command, the extension gets packed, but the resulting package is empty. We've attached a screenshot for reference. Could you please assist us with resolving thisHow to configure Choice-based Field Rules??
I have multiple choice, drop down and subform in what i created. However, I can't configure the choice based rules. There's no button link for that. Please help https://forms.zohopublic.com/specialevents1/form/IslandKidsConference2025/formperma/ZCXsCustom module system name
Is there a way to define custom module system name? Currently it's something like `CustomModuleX` This is different to API name or display name.Line formatting in data coming from Zoho Flow
I have a flow that creates a zoho desk ticket. The text is not formatted with line breaks once it arrives at Zoho Desk. How do I force line breaks?There is no opton to download the Active User list in Zoho CRM Of users
There is no opton to download the Active User list in Zoho CRM of UsersSession recording PageSense Blank Layers using Canvas
Hello, We have implemented page sense in our SaaS platform and there are key elements on the pages that are not being show in the recordings. For example, the lower section is a map canvas using webgl and the upper section is a timeline style canvas.Help with Tickets Endpoint
Hello Team, I am using the ticket lookup endpoint to retrieve the list of tickets associated with a particular account. GET https://desk.zoho.eu/api/v1/accounts/{accountID}/tickets I want to know if there is a way to filter the tickets to retrieve onlyWhy is my deluge code not executing properly? (New and learning deluge)
I'm trying to update a sales order number through deluge for an automation I'm trying to build. What is wrong with it? /* replacing SO with SOR for sales order pre-fix */ SO = salesorder.get("salesorder_number"); replace = SO.replaceFirst("SO","SOR");merge the Multiple POs to single PO if Vendor of PO"s --in Zoho Inventory
HI Merge the Multiple POs to single PO if Vendor of PO"s are Same ----in Zoho inventory Please provide any work around to achive this .Flow based on New Deal - layout field problems
Hi. I've built a Flow to create a Deal-specific folder within WorkDrive. It works except for referencing the Deal's layout. In my Zoho CRM, there are two types of Deals (separated by layout). For my Flow: Trigger on New Deal Then, I call 'Fetch module entry' as I am trying to grab the Deal's layout. As I process through my logic, I have a Decision based upon the Deal's layout. However, within the 'Fetch module entry', I had to specify the Deal's layout so all Deals added (no matter their actualCRM
I have a portal set up where a contact can see other contacts within an account automatically. When a contact in the portal enters a deal, how do I make sure that deal is assigned to the account so other contacts in the account can see the deal was generated?A fresh interface and functionality improvements for Zoho CRM's data sharing settings
Data sharing is essential for maintaining proper access controls in your CRM. Whether you're setting default permissions or defining custom sharing rules, these settings determine how records are shared among users, roles, and groups. Previously, navigatingfield is not saving on lead
Hi, Not sure what i changed but on edit in a lead, i am inputting details and its not saving. just highlights it in yellow with no error message.Zoho AI Translate – No Support for Hebrew?
I want to use Zoho AI Translate, but I see that it does not support Hebrew. Since Zoho supports multiple languages, I was hoping this function would also work with Hebrew. Is there any plan to add Hebrew support in the near future? Or is there an alternativeBased onthe multipick list value want to Show in the pick list field
Based onthe multipick list value want to Show in the Single pick list field Database Region is multipick list if it contain Saudi and UAE then region pick list want to show Saudi and UAE Database Region is multipick list if it contain Saudi then regionHow can I view Help Center articles in 'Sandbox' mode?
I have published some help center articles but when I try to preview the knowledge base page, it says 'We have no content to display'.How Do I Refund a Customer Directly to Their Credit Card?
Hi, I use books to auto-charge my customers credit card. But when I create a credit note there doesn't seem to be a way to directly refund the amount back to their credit card. Is the only way to refund a credit note by doing it "offline" - or manually-Mass Update Doesn't Trigger Workflow
Hello there, I made a workflow in Zoho Desk, and its related to Ticket module, when I update the field workflow triggers, but the problem is when I update the field in Tickets module List View (Mass Update) the workflow isn't triggering. If I can't triggerRejected Accounts still listed for Deal
Hello guys I have a approval process thats validate the Accounts in CRM The issue is even when a account is rejected you can open a deal with it It's not supposed to appear in the lookup field or the approval process become pointless Can you guys helpNext Page