Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Centralize Knowledge. Transform Learning.
All-in-one knowledge management and training platform for your employees and customers.
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Kaizen #198: Using Client Script for Custom Validation in Blueprint
Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.Kaizen #226: Using ZRC in Client Script
Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In thisKaizen #222 - Client Script Support for Notes Related List
Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore howKaizen #217 - Actions APIs : Tasks
Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.Kaizen #216 - Actions APIs : Email Notifications
Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Zoho Commerce and Third-party shipping (MachShip) API integration
We are implementing a third-party shipping (MachShip) API integration for our Zoho Commerce store and have made significant progress. However, we need guidance on a specific technical challenge. Current Challenge: We need to get the customer input toNeed help to evaluate if Commerce is good for me
Hi, I just want to quickly check if Zoho Commerce can fulfill my needs. Here is what I am looking for: - Multi-vendor plateform : We will be 3-4 different farms that will offer similar products (ex. tomatoes) to few selected customers (retaurants). AllWhen will Zoho Trident come to Linux?
It already is implemented for MacOS, which is *nix based operating system. When will it be ported to Linux?Zoho Commerce + MCP + Claude
Hi everyone, I wanted to share a really impressive experience I had with Zoho Commerce + MCP + Claude. I created an MCP server using Zoho MCP Server and connected my Claude instance to my store built with Zoho Commerce. For safety, I enabled only productUK payroll entries
Hey guys, Nett payroll payments are imported direct into the bank, using an external payroll system (will be glad for Zoho to have a UK payroll app) At present I have monthly recurring bills for HMRC which are auto entered & paid when due. This seemsauto add as member the contact owner
is there a way that i can make a zoho flow that will add the owner of the contact as a member of the chat after the round robin assignment?Get Files Associated to Data Template via API
I have a data template with multiple files associated to it, and trying to write a Deluge script that will fetch files associated with this data template. I created the script below based on the WorkDrive API documentation, one request uses the data templatesZoho Book - Banking Module - Cash Credit account
I have a CC account with a bank. I initially added the account as a 'Bank Account' under the banking module in Zoho Books. However, this CC Bank Account is showing as an asset instead of a liability. I have added the account as a credit card account butAdding a custom formula field from sub-forms
I have used my 5 Aggregated Sum fields in the subform. I have another column I want to add a summary of. This can be in a separate section/field. The formula field won't pull in information from the sub-form. How do I pull information from the sub-formRemove Zoho Header from Portals
I have a portal page with custom domain. But when I print directly from a webpage, the Zoho CRM header shows. It kind of kills the branding aspect. Is there a way to get rid of this?Creator List not showing
I've edited a list in creator, but when I open it in the live version the list is completely different. Anyone know why this is happening? Edit Version: Live Version:Please design a better spam filter
Hi Team, While I love Zoho Mail for all the awesome features you have designed - this is the one you need to do a lot of work on - Spam Filter. Your spam categorization seems to be very inefficient - any email containing brand images or a tracking pixelHost not reachable when sending
For few partner that I send regular emails now got error and they don't receive emails Temporary failure fam-xxx.hr/185.xx.xx.xxx Host not reachable. Have more than 20 mails in https://mailadmin.zoho.eu/cpanel/home.do#mailSettings/retryEmail pleaseIP flagged as abusive
I'm getting the error that 136.143.188.15 is listed as abusive. I've checked with mxtoolbox.com and it is indeed in the listApp passwords stop working
Some of my app passwords appear to stop working after awhile. Seems to happen every 6 months or so. Just had to change one again for the second time in a year. Is this a bug or a feature? Or perhaps changes are being made on the back end on occasion thatDuplicate entries for contacts birthdays
Good morning I have recently started to use my Zoho calendar and noticed that there are multiple birthday events showing for some of my contacts. I have checked my contacts and there were duplicates for some contacts which I have now rectified but theProblem: New Email Consistently Failing to Sync/Display in Inbox
Recently, I have noticed that my Zoho Email inbox does not indicate I've received a new email, until I go to *another* folder and then return to the inbox. Only then will new emails be displayed. This has happened 100% of the time, during the past fewZoho IMAP Access to mail doesn't sync messages
As stated in the topic - I have company account email address setup, and it has ( as in the picutre ) ActiveSync and Imap access enabled. However the messages does not show up while logged in in Zoho Mail however when I log in in thunderbird everythingCliq iOS can't see shared screen
Hello, I had this morning a video call with a colleague. She is using Cliq Desktop MacOS and wanted to share her screen with me. I'm on iPad. I noticed, while she shared her screen, I could only see her video, but not the shared screen... Does Cliq iOS is able to display shared screen, or is it somewhere else to be found ? RegardsTemps working custom view in Zoho Recruit
How do you create a custom view of temps working for this week. I have tried multiple combinations of job status, start date, end date etc but I am unable to find the correct combinations to show the temps I have working this week?? Any suggestions on how I can do this.Client Scripts similar to CRM?
Hi, Is there any plans for Client Scripts to become a feature of Recruit? Feel this will be a great benefit to a number of users. Would be great for example in cases where a Client has banned / blacklisted a Candidate - so that if that candidate is associatedAnnouncing Zoho Sheet desktop app for macOS and Windows (Beta)
Hello Sheet users, We know you’ve been waiting for this one. It has always been the top priority on our roadmap to provide a single native desktop app for macOS and Windows that works both online and offline. Today, we are excited to announce that theAutomated entries past the current month in a calendar report
Hi all, I have an automation problem. I have a form which on successfull entry adds either 5 or 10 more of these entries with a slight change so our customers can see it throug a calendar report on the webiste. The entry put in manually shows up perfectlyBilling Status Update
Hello Latha, I’m working on a new automation (deluge) to fulfill one of our requirements. In this automation, there is a step to update the Work Order billing status from “Not Yet Invoiced” to “Non-Billable.” I tried to find the API information relevantWhat's New in Zoho POS - April 2026
Hello everyone, Welcome to Zoho POS’s monthly update, where we share our latest feature updates, enhancements, events, and more. Let’s take a look at how April went. Access and manage other web applications in Zoho POS with Web Tabs You can now accessMany Zoho POS Issues
Can not apply credits from a customers account as a form of payment. It shows that you can but there is a bug that does not execute the action. Reported many times. Can not view Sessions from Zoho POS WebView, throws a JQUERY error Workflows and actionsHOW I CAN GET BETA VERSION FOT TESTING
HI. ZOHO I AM INTERESTED IN YOUR BETA SOFTWARE TO EXPERIENCE NEW FEATURES.Multiple EAN code for product
option to add multiple EAN code for product???Issue adding/changing mobile number for OTP
Hi Zoho Community, I’m trying to add or change my mobile number, but I keep getting this error: “We’re unable to send OTP to this mobile number. Please contact support-as@eu.zohocorp.com” Because of this, I can’t verify my number or continue the setup.Zoho Payroll Canada does not work
We submitted payroll within the required timelines. The money was withdrawn from our account and our employees have not been paid. Impossible to get a hold of someone at Zoho. I can’t believe the company operates like this. What a bloody disgrace. ShameEnhance Appointment Buffers in Zoho Bookings
There was previously a long-standing feature request related to enhancing the way appointment buffers work in Zoho Bookings, but it looks like the original post has been deleted. I am therefore adding a new request that Zoho Bookings adjust how appointmentFunctionality based URL to find records due today or in the next week
I need to construct a URL to filter a view to records where the evaluation is due soon (i.e. in the next 7 days). To me, that includes today, but Zoho has a different opinion. Zoho defines 39 as NEXT_7_DAYS but this gives unexpected results. https://creatorapp.zoho.eu/...#Report:My_Evaluations?Evaluation_due_by_op=39Allow customers to choose meeting venue and meeting duration on booking page
My business primarily involves one-to-one meetings with my clients. Given the hybrid-work world we now find ourselves in, these meetings can take several forms (which I think of as the meeting "venue"): In-person Zoom Phone call I currently handle theseEvent Time Zone in meeting invites are confusing users
When sending calendar invites to internal and external users, the first section "Event Time Zone" is confusing people and they are automatically declining events. Can this section please be removed??? It already shows the correct time zone next to theForwarding without verification
Hi, I use Tripit to keep track of all my business travel. I've recently moved over to Zoho and wanted to set up a forwarding rule to send various travel confirmation emails automatically to plans@tripit.com Obviously this is an email address I don't control,Filter by user in Pivot Chart
I have a Pivot chart where the data should be filtered by user. The user enters the system and should see only the data that correspond to it. Can anyone help me?All new Address Field in Zoho CRM: maintain structured and accurate address inputs
Availability Update: 29 September 2025: It's currently available for all new sign-ups and for existing Zoho CRM orgs which are in the Professional edition exclusively for IN DC users. 2 March 2026: Available to users in all DCs except US and EU DC. 24Whatsapp Limitation Questions
Good day, I would like to find out about the functionality or possibility of all the below points within the Zoho/WhatsApp integration. Will WhatsApp buttons ever be possible in the future? Will WhatsApp Re-directs to different users be possible basedInternal mails on our company domain (managed by Zoho) do not get delivered
Hi last week Thursday and Friday a colleague had sent me two emails which did not show up at all in my inbox, spam or anywhere else. What this a problem with Zoho mail in general or did this affect just us? From the forum is reads like many problems had#2 The Tax Talk Nobody Wants to Have
"We are also able to manage our GST invoices without worrying about compliance issues. With Zoho Invoice, we are happy, our tax consultant is happy, and so are our customers," rejoices Shivprateek Habib, Partner, Flutterboots Services LLP. That's usuallyNext Page