Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Centralize Knowledge. Transform Learning.
All-in-one knowledge management and training platform for your employees and customers.
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Kaizen #198: Using Client Script for Custom Validation in Blueprint
Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.Kaizen #226: Using ZRC in Client Script
Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In thisKaizen #222 - Client Script Support for Notes Related List
Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore howKaizen #217 - Actions APIs : Tasks
Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.Kaizen #216 - Actions APIs : Email Notifications
Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Client Script also planned for Zoho Desk?
Hello there, I modified something in Zoho CRM the other day and was amazed at the possibilities offered by the "Client Script" feature in conjunction with the ZDK. You can lock any fields on the screen, edit them, you can react to various events (fieldView Answer Bot conversations?
We are trialing Zia and are experimenting with Answer Bot on our knowledge base. So far so good! Management asks me if it is possible to view Answer Bot conversations, the purpose being to look over its shoulder and confirm that it is working as desAdd Comment workflow action delayed by 1–2 minutes in Zoho Desk
Hello Zoho Support team, We are experiencing an issue with a workflow in Zoho Desk. The workflow is triggered when the Quote ID field is updated and performs two actions: A standard Field Update An Add Comment custom action When the Quote ID field isZoho CRM Layout Rules: Nine New Actions, Profile-Based Execution, and Interactive Preview
Hello everyone, Availability: This feature is now available for customers in the JP and SA DCs. It is planned to be released for other customers in soon. We’re excited to announce powerful new enhancements to Layout Rules in Zoho CRM - a feature builtZoho Books and TRAINING SALES receipt label for eTims?
Hi, Can Zoho Books implement TRAINING SALES receipts and push them to eTims for test? In other words how can we send to Zoho or even create in Zoho training mode invoices and TEST the workflow POS>Zoho Books>eTims and back without actually registeringReporting Tags and COGS
Is there any way to get COGS recorded against reporting tags? The use cases seems very straightforward to me. If I'm running a P/L report against a specific reporting tag (I use mine for customer type, but it could be used for regions, etc.), the revenueHide Custom Fields for Customers or Vendors
For whatever reason, all custom fields added to the Customer module also get added to the Vendor module. This leads to a clunky experience for users, as many custom fields we add only apply to one or the other. There should be a "hide" option to hideOnboarding process
I urgently need onboarding support. My account still shows trial period.Detailed General Ledger - Display Transaction Description in Reports
Hi Zoho Team, I'm currently using Zoho Books and have a question regarding the Detailed General Ledger report. When we enter a description or narration in transactions (Bills, Journal Entries, Expenses, etc.), the report does not display this description.Which Zoho CRM tasks would you want an AI assistant to handle?
I’ve started delegating my Zoho CRM work to AI – and many tasks now take me half the time. It has been so useful that we’re now working to make it available to everyone. Instead of clicking through records and tabs, I simply describe what needs to beZoho Mail Does Not Integrate with Analytics
Hello everyone, Just wanted to make a quick post about the problem with zoho mail. Currently it does not connect to Zoho Analytics, which I think is crazy considering both of these applications have existed for a long period of time. Even zoho voice aNet ##th Terms (e.g. due on the following 15th)
I have a client who pays me twice monthly per our contract. It would be helpful to have terms on my invoices where the invoice is due on the 15th or the 30th of the same or the following month.Project status update visibility within a project
Currently the only place within Zoho projects to see status updates for a given project is actually not in the project it's self it's in the Collaboration section and then by searching for a given project and clicking on the status tab. It would be greatUsing custom CRM fields
I have created a series of fields in the Deals module that need to be linked to fields in a contract I have customized. I can only see how to map standard fields, not the custom ones. How do I do this?Como limitar o Zia Agent?
Geralmente quando estamos criando um agente de IA no ZIa Agent Studio, colocamos o que ele deve fazer, utilizando as instruções que colocamos no Knowledge Base. Mas como colocamos limites que não devem ser ultrapassados? Para isso utilizamos os Guardrails,An important update for Zoho Cliq desktop users
The latest version of the Zoho Cliq desktop app (v1.8.0) will no longer be supported on macOS 10.15 Catalina and earlier versions. This is because the framework we use (Electron) no longer supports some older macOS versions. If you’re using macOS 10.15Zoho Flow - Zoho Desk trigger "Ticket created or updated" overrides Department
Just in case anyone else has this issue. We had a Zoho Flow with the Zoho Desk trigger "Ticket created or updated". In the trigger I had selected a Department (Sales). What happens is that the Department ID (Sales) is submitted as part of the request2026 Product Roadmap and Upcoming Features
This is your guide to what is coming in Zoho Vertical Studio throughout 2026. We’ll update this post throughout the year as items move from development to release, and as and when new initiatives are added. Once a feature is released, it will be reflectedIntroducing Canvas templates: Design once, apply everywhere
Greetings all! We're happy to introduce Canvas templates, a Forms view enhancement that enables you to design form layouts just once and apply them seamlessly across multiple modules. Imagine you use five CRM modules (Contacts, Deals, Leads, and more),Zoho Projects - Will there ever be a send email feature in Zoho Project?
Hi team, Are there plans to or will there ever be a sendemail feature in Zoho Projects, brining it in line with other similar platforms like Asana, ClickUp and Monday? I know that you can add comments via email to a specific task, but I believe this onlyWhy cant i see the Campaigns module in my Zoho app
I am on the ZOHO CRM Standard edition. I can clearly see and use the Campaign module when viewing the ZOHO CRM desktop. When I switch to my phone, the Campaign Module does not show. I have gone through all the settings I can think of, but I still can't📣 Ask the team behind Zoho SalesIQ: Summer '26 Q&A
Hi everyone! Following our recent webinars, Summer '26 Release: What's New in Zoho SalesIQ—where we walked you through all the new features, what they do, and how they work—and Driving the AI Evolution: Building Smarter Customer Experiences with ZohoAlways display images from this sender – Is this feature available?
In Zoho mail, I had my "Load external images" setting set to "Ask me", and that's fine. That's the setting I prefer. What's not fine though is I always need to tick "Display now" for each email I get, regardless if I've done that multiple times from severalSet Custom Icon for Custom Modules in new Zoho CRM UI
Displayed sample tracking for multi branch setup.
We have multiple branches and one of the biggest challenges for us to be able to track if a particular item is in display or not. I am shooting in the dark but I am wondering if there is a solution for the following : Each product used for display isHow do I migrate from Office 365 to Zoho Mail?
Check out Advik Email Migration Wizard, this software is specially developed to move mailboxes from Office 365 to Zoho Webmail. In addition you can migrate from Gmail, Yahoo, Rediffmail and 80+ webmail servers to ZOHO MAIL. Isn't it amazing? This is an all in one email migration solution. Steps to export emails from Office 365 to Zoho Webmail are as follows; Run Advik Email Migration Tool in your system. Select Office 365 as source and enter its login credentials. Select mailbox folders and chooseConnection Issues
Morning. Seems like a few of my colleagues that are offsite are having issues with connection to cliq, even when their internet is working perfectly fine and only 1 person onsite has issues. Can you please explain? ThanksGoals API - Zoho People
Hi Team, I would like to get the API details for retrieving organisation-wise Goals from Zoho People. Currently, I am able to retrieve individual employee goals using the following API: https://people.zoho.com/api/v3/performance/goal/{emp_id} However,Backorder process review - Automating Removal of Sales Order from "On Hold" When PO is Received
Hello Zoho Inventory Team, Currently, sales orders in On Hold status are released only when the bill for the purchase order is created. In our workflow, it would be much more efficient if the sales order could automatically move out of On Hold as soonCliq iOS can't see shared screen
Hello, I had this morning a video call with a colleague. She is using Cliq Desktop MacOS and wanted to share her screen with me. I'm on iPad. I noticed, while she shared her screen, I could only see her video, but not the shared screen... Does Cliq iOS is able to display shared screen, or is it somewhere else to be found ? RegardsWhich Is the best online store to buy office furniture in Australia ?
The most important part of your office is the office furniture. Your office is not complete when you have not put the required furniture in place. You need a furniture in your office so that you and your employees have desks, tables, and chairs whichAdd "Product Code" as available column and search criteria for Assemblies
Hi Zoho Community. For product (item) assemblies, it would be helpful to be able to create a custom view search criteria that limits the view by product code or perhaps product category. Currently, Product Code is not an available option to display asWhat's New - July 2026 | Zoho Backstage
Hello everyone, We hope you’re having an expectation-exceeding event experience! We’ve been hard at work bringing you a bunch of enhancements designed to make event management even easier. Let’s take a look at what’s new in Zoho Backstage. Export deliveryDelete Field that is used in a Zoho Flow connection
I'm trying to delete a Field used in a Webhook created by Zoho Flow with CRM Connection and i get the following alert: When going to the alert i get to the following issue, can't edit it since its been deployed by a pluggin But yes i have here the promptedQ3 USA ZUG Meetups - Build AI Agents with Zoho
Hi everyone! What if you could build your own AI agent from scratch in just one session? That's exactly what we're going to do at the upcoming ZUG Meetups across the United States, where you'll get hands-on experience with Zia Agents and explore how AIData Export is forcing Creation/Modified date just like reports
The data export now forces you to select a created or modified date. There is not an All-Time option. This severely cripples the whole function of data export for audit purposes. If the data export (or reports for that matter), requires a date, thereDefault ticket template in helpcenter
Hello, I have a web form and a ticket template created. How can I make that my default ticket template? If an user clicks New ticket or create a ticket, I want that template to be the default one. Thank you for the time and info.Zia Agents looks promising, but I still cannot deploy my first agent or connect WhatsApp after weeks of support tickets
Hi Everyone, I am posting here because I am stuck and need practical help from someone who has successfully deployed a Zia Agent with WhatsApp. Zia Agents looks like a very promising product. I have watched the platform expand quickly, and I have noticedValid characters for use in email addresses using Zoho's apps/APIs
We have found an issue with the + sign character in zoho subs - we are allowed to create a customer with an email address that has a plus sign but the API doesn't allow the + so the billing portal is not created. We are going to prevent users from usingFeature Request – Per-Channel Pre-Chat Form Controls
We'd like the ability to enable/disable the pre-chat form on a per-channel basis, rather than having a single global setting. On the website widget, asking for name and email before chat starts makes sense. On WhatsApp, however, email isn't necessaryNext Page