Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Kaizen #216 - Actions APIs : Email Notifications
Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes areKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achievedKaizen #142: How to Navigate to Another Page in Zoho CRM using Client Script
Hello everyone! Welcome back to another exciting Kaizen post. In this post, let us see how you can you navigate to different Pages using Client Script. In this Kaizen post, Need to Navigate to different Pages Client Script ZDKs related to navigation A.Kaizen #210 - Answering your Questions | Event Management System using ZDK CLI
Hello Everyone, Welcome back to yet another post in the Kaizen Series! As you already may know, for the Kaizen #200 milestone, we asked for your feedback and many of you suggested topics for us to discuss. We have been writing on these topics over theKaizen #197: Frequently Asked Questions on GraphQL APIs
🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Standalone custom function not generating logs
Why dont't standalone custom functions generate logs when the're called from another function? I have some functions (workflow, buttons and blueprint) that have common parts, so I put that part in a standalone function which is called from the others.Automated log-out/session end
I'm concerned about security of our data. Is it possible to set an automatic time-out for user sessions on Zoho CRM, after a certain period of inactivity or when the session reaches a certain duration (12 hours perhaps)?Add "Reset MFA" Option for Zoho Creator Client Portal Users
Hello Zoho Creator Team, We hope you are doing well. We would like to request an important enhancement related to Multi-Factor Authentication (MFA) for client portal users in Zoho Creator. Currently, Creator allows us to enforce MFA for portal users,Support Bots and Automations in External Channels
Hello Zoho Cliq Team, How are you? We actively use Zoho Cliq for collaboration, including with our external developers. For this purpose, external channels are a key tool since they work seamlessly within the same interface as all of our other channelsUrgent Security Feature Request – Add MFA to Zoho Projects Client Portal Hello Zoho Projects Team,
Hello Zoho Projects Team, We hope you are doing well. We would like to submit an urgent security enhancement request regarding the Zoho Projects Client Portal. At this time, as far as we are aware, there is no Multi-Factor Authentication (MFA) availableCreator HTML page refresh
Hi, I have added around 5 different html snippets in single creator page. I understand, I can refresh the entire page from page script using Navigational URLs https://help.zoho.com/portal/en/kb/creator/developer-guide/others/url-patterns/articles/navigational-urlsZoho Desk - Custom Module Related List Columns on Tickets
I have a custom module in Zoho Desk called Asana Tasks, each task has a lookup to a Ticket. On the Ticket I want to see the columns of the Asana Tasks in the related list . Is there a way to do this? Right now it just has the name of the record and IGoogle Analytics import data inaccurate (as of October 11, 2025)
We have Zoho Analytics connected to GA4 to import daily event data. This has been running without issue for a couple of years. However, a month ago we started noticing discrepancies. All data until October 10 lines up perfectly - October 11 onward isWhat is your opinion of the new UI?
Hi Everyone, I would like to see what everyone thinks of the new Zoho One Dashboard. I don't get it, but perhaps I'm missing something. What are your thoughts?What are the OAuth scopes needed to access the Zoho MCP server?
I think I'm authenticating with the MCP server endpoint successfully using a bearer token but I can't successfully initialize an MCP session. I'm using the official MCP Python sdk (https://github.com/modelcontextprotocol/python-sdk). I get as far as hereAccount blocked for IMAP use
Hello, My email client (Evolution) can't sync mail anymore. It gives the error: "Your account is temporarily blocked for IMAP use. This may happen if you exceed the maximum number of simultaneous IMAP connections allowed. Kindly try again after some time."Is SMTP included in the free plan?
My client has a Zoho Mail Free 14 day trial that ends tomorrow. She has set up a domain email address with SMTP on a third party app and is wondering if it will continue working once her plan downgrades to the free forever plan. ThanksZoho mail issue.
I have verified my domain ohhoexpress.online by adding necessary TXT and MX records. But when I am sending email to any external email id, it is showing as zohomail.com. Also while I am sending mail to internal mail id, it is saying user not found whereasMy email is blocked from sending
I get this message when I try to send any message .. > Unable to send message, Reason 554 5.1.8 Email Outgoing Blocked We have urgent emails to send and the issue continues to persist from yesterday. Please help us with this issue as soon as possibl[WEBINAR] Smooth year-end closure with Zoho Books (KENYA)
Hello there, This webinar is for all Kenyan businesses looking to wrap up their financial year smoothly! Join our free session to learn how Zoho Books can simplify your year-end process. What to expect from this webinar: - All the latest updates in Zohocannot recieve or send emails
we are not recieving or cannot send emails.Shows sysytem errorZoho Sites - contact us form
I have added a contact us form to my website. I wanted to change the layout so I deleted the section of Contact Us form. When I again try to add the section of Contact Us form, it shows error that "This form already exists on this page." I am unableswitching from POP3 to IMAPI do I need to change settings in Web email?
hi so I am using Outlook with My Zoho mail, also have Verizon email /AOL. they no longer support POP3 and said imapi is much better, so I want to switch my ZoHo to imapai. I googled and it said i need to change it in my account....i looked at my accountWhat functionality is new in the Z025 Spaces UI dashboard compared to the previous Zoho One dashboard?
My company is interested in creating a revamped dashboard this coming year, and are trying to weigh different platform options. The new Spaces UI is said to be a more "feature-rich" dashboard experience than the previous. Could a developer please explainCollections Management: #5 Convenience on Offer, using Self-Checkout
"Is this the right page? Oh, now, it wants my address again. Why am I being redirected?" These were the exact thoughts running through Karan's mind as he tried to subscribe to an application he genuinely liked. He clicked "Buy Now", expecting a quickUploading a signed template from Sign to Creator
Good day, Please help me on how to load a signed document back into Creator after the process has been completed in Sign. Below is the code that I am trying, pdfFile = response.toFile("SignedDocument_4901354000000372029.pdf"); info pdfFile; // AttachTip #50- A Closer Look at the Unattended Access Dashboard- 'Insider Insights'
Having complete visibility and quick access to everything you need certainly makes managing multiple remote devices a lot easier, and that is precisely what the Unattended Access Dashboard in Zoho Assist is designed to offer. Once you go to the UnattendedHow to update custom multi-user field in Zoho Projects?
I'm trying to update custom multi-user fields in Zoho Projects via a Deluge function in CRM. The code I have so far is below. It works for updating standard project fields and single-line custom fields, but it does not work to update multi-user fields.Tip of the Week #75– Manage your social media messages from a single shared inbox.
Are you tired of jumping between apps or browser tabs to reply to your business's Facebook and Instagram DMs? Handling customer messages on social media might seem simple, but switching between multiple platforms can easily lead to lost messages, duplicateZoho Map integration tasks have changed - you need to "Locate all instances of Zoho Map integration tasks in your Deluge scripts by searching for the v1 marker... before 16 January 2026"
The Zoho Map deluge integration task has been changed (as at 21 October 2025) to provide a more structured, JSON-like response. This change affects all three Zoho Map integration tasks (Geocode, Reverse Geocode, and Distance Between). More details canUsing files from Zoho CRM in Gemini/ChatGPT/Claude
Hi all, I’ve got subscriptions to Gemini and a few other AI tools which I use for tasks like data enrichment, email composition, etc. In our workflow, we often receive various documents from clients — such as process workflows, BRDs/requirement documentsZoho Analytics & Zoho Creator - Modified Time value
I'm trying to use the Zoho Creator system field 'modified time' in Zoho Analytics, but it's consistently showing 12 hours 'out' In Zoho Creator In Zoho Analytics Is this a constant difference that I just need to correct with a timezone change - or isZoho CRM - Option to create Follow-Up Task
When completing a Zoho CRM Task, it would be very helpful if there was an option to "Complete and Create Follow-Up Task" in the pop-up which appears. It could clone the task you are closing and then show it on the screen in edit mode, all the user wouldPortal For Different Apps
I found some older threads on this but didn't see anything very recent. I'm new to Zoho One so forgive me if my terminology is off a bit. I was hoping set up a single point of entry into Zoho One. So, many of the apps could be found in one single placeCalls undetected
Zoho Voice records indicate my last call ended at 6:00 PM. All incoming and outgoing calls occurred between 6:00 PM and 7:00 PM.Zoho Projects - Project Details on the Project Menu
Hi Project's team, I've helped may businesses setup and use Zoho Project and one thing I see time and time again is confusion on where to find the Project Details information. I would be much more intuitive if Project Details was on the menu before Dashboard.Introducing WhatsApp integration in Bigin
Greetings! In today's business landscape, messaging apps play a significant role in customer operations. Customers can engage with businesses, seek support, ask questions, receive personalized recommendations, read reviews, and even make purchases—allZia Conversation Summary: Context at a glance for every customer interaction
Hello everyone! Every customer conversation tells a story—but in CRM, that story is rarely in one place. A sales rep moving between multiple leads has to reopen long email threads, check call remarks, and revisit meeting notes just to remember what wasZoho Projects - Show Task List as dropdown field on Task records
Hi Project's Team, I noticed today that there is no field on a task record related to the task list it belongs to. A dropdown would be helpful for quickly moving tasks between lists while in a task. I know that you can go to "Other Actions" and chooseChanging the Default Search Criteria for Finding Duplicates
Hey everyone, is it possible to adjust the default search criteria for finding and merging duplicate records? Right now, CRM uses some (in my opinion nonsensical) fields as search criteria for duplicate records which do nothing except dilute the results.My followed tickets extension is not working under the All departments view
Hi. I've installed the My followed tickets extension. However, when I try to open the extension under the all departments view, I get the following message: 'Sorry, this extension is not supported in the All Departments view.' How can I solve this pTicket Time Entry to Timesheet
The title just about sums it up. I have searched here and not found anything relevant, but If I overlooked, then please set me straight. We have staff that do nothing but close tickets in desk all day long. These tickets represent their timesheet. Is there a way to have this information sync or for a tech to go into their timesheet themselves and sync it with their tickets of the same timeframe?? We waste a ton of time doing timesheets and the old "Clock in/Clock out" isnt detailed enough for us!!Calls undetected.
The call is not showing on the call log.Calls undetected
Zoho is not reading calls made.Multi-currency and Products
One of the main reasons I have gone down the Zoho route is because I need multi-currency support. However, I find that products can only be priced in the home currency, We sell to the US and UK. However, we maintain different price lists for each.Next Page