Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Centralize Knowledge. Transform Learning.
All-in-one knowledge management and training platform for your employees and customers.
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Kaizen #198: Using Client Script for Custom Validation in Blueprint
Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.Kaizen #226: Using ZRC in Client Script
Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In thisKaizen #222 - Client Script Support for Notes Related List
Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore howKaizen #217 - Actions APIs : Tasks
Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.Kaizen #216 - Actions APIs : Email Notifications
Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
WebDAV support
I need WebDAV support so that I can upload/download (and modify) documents from my local file system. Is anything planned in his direction?Replying from same domain as a catch-all?
I have 2 domains setup on Zoho, both with associated email addresses. They look something like this: john@example.com (primary address) john@test.com (this domain also has a catch-all setup) I use the catch-all for test.com as a public-facing email addressIs there a way to show contact emails in the Account?
I know I can see the emails I have sent and received on a Contact detail view, but I want to be able to see all the emails that have been sent and received between all an Accounts Contacts on the Account Detail view. That way when I see the Account detailInternal Fillable Contract with Zoho Writer (Before Sending to Client)
Hi everyone, I’m trying to automate the following process in Zoho CRM and would appreciate some guidance. Process: When a Deal moves to a specific stage, CRM triggers an automation. CRM sends a contract template to an internal team member so they canImport Quote from CRM into document in Writer?
Hi, I created a quote for a proposal in the Zoho CRM and was wondering if there is a way to embed the quote into a document in Zoho Writer so I can make our sales proposals look a little nicer. Is that possible?Zoho Sheet for Desktop
Does Zoho plans to develop a Desktop version of Sheet that installs on the computer like was done with Writer?Can I convert MSG to HTML on Mac?
Yes, you can convert MSG files to HTML on a Mac using software Aryson MSG file Converter. This tool allows you to convert Outlook MSG emails into multiple formats, including HTML, PDF, EML, PST, and more. It preserves all email content, attachments, metadata,Duplicate Leads Notification Help!
Hello! I have several web forms that have a duplicate lead notification that are being sent to the creator of the web form. I understand how to change the form entry notification, but I am specifically looking to change the recipient of the "Duplicate Lead" notification. Any help you can offer is greatly appreciated! TIA, ~ JennAdding Calendar Display to Home Page
I would like to display my calendar in a module on the Zoho CRM Home Page. I can't figure out a way to do this. Is it possible to display the CRM Calendar on the Home Page?Schedule a Call by Date and Time when a specific lead status is selected
Hi Wanting to create a workflow where a call can be scheduled by date & time when a specific lead status is selected. Can only currently set the date by Due Date - Trigger Date - Plus 'x' day(s) ThanksIncoming email replies not automatically associating with Deals/Opportunities - Is this possible in Zoho CRM?
Hello Zoho Community, I'm running a travel agency (B2B and B2C) and we've been struggling with what seems like a basic functionality that we cannot get to work properly. Our use case: We send emails to suppliers (hotels, transportation companies) andDuplicate Leads Concerns with Round Robin and Lead Approval Process
It is great to have the Duplicate Lead Approval Process, there are a few issues with the process that I would greatly appreciate taken consideration in enhancing. It appears that A Lead comes in Lead owner assigned by the Round Robin Check for Duplicate,Private email threads
When sending a Private email or receiving a response once the email thread has been marked as 'Private'- is there a way to trigger the system to "restart the clock"? I am finding that when someone responds to a 'Private' thread that it is opening theSLA Notification
Team members have been receiving this notification and feel as though it is as random. Explanation on this notification is appreciated! Also, is there a way to disable this notification?Map: Output None
Hi, Checking if you help me inspect this block The code below triggers during Successful form Submission from another Form. // rec_a = formA[ID!=0]; rec_b = formB[ID!=0]; ListA = List(); //subformA for each recA in rec_a.ItemSubform { for each recB inUse arbitrary images in deluge
I am sending emails from my deluge code and I would like to be able to include a few images in the email template. While I know that I could put the on a website somewhere and link to them in the HTML, that is not what I want due to email client securityHow to add "All Open AND Overdue" back to the Home Page Task Component?
Hi everyone, I’m looking for a way to restore the Tasks component dropdown list on the Zoho CRM Home Page. Since the recent update to the Task area in my Home Page Classic View, the dropdown options (e.g., My Next 7 Days + Overdue) are too restrictiveSync Attachments in Comments from Zoho Desk to Zoho Projects
Hello Zoho Desk Team, We hope you're doing well. We are actively using the Zoho Desk–Zoho Projects integration, especially for reporting and managing bugs/issues between support and development teams. Current Limitation: After creating a bugs/issues viaAdd a way to connect Log360 Cloud logs with Zoho analytics
Hi, Several month ago Log360 Cloud was added to zoho one - and this is great. But as far as I see there is no prebuilt way to connect Zoho analytics to the logs we have in Log360 Cloud. Please add a prebuilt connection like we have for so many other zohoAuto sync Photo storage
Hello I am new to Zoho Workdrive and was wondering if the is a way of automatically syncing photos on my Android phone to my workdrive as want to move away from Google? ThanksFuncionalidades y configuration ZohoDesk
Creo que no estoy sacando el provecho adecuado a la application quisiera solicitar una capacitación al respectoHow to call Functions and perform Write Operations using Page Scripts?
Hi everyone, How to call a function from a Zoho Creator application within a Page Script, also how to perform write operations (adding or updating records) using page scripts?Connecting Airwallex in Zoho Books
I have a question. I’m trying to connect our Airwallex account to Zoho Books, but the integration does not seem to work. Could you please guide me on how to properly connect Airwallex? I need to see the expenses also, since it only shows the profits nowQuestions Regarding Helpdesk & SalesIQ Customization and Email Setup
Hello, I hope you’re doing well. I have a few questions regarding Helpdesk and SalesIQ: Can the emails sent to customers via helpdesk tickets be fully customized — including signature, subject line, and other elements? Also, is it possible to send theseButton Duplication when emails forwarded or replied
When I create a new email template in campaigns and send it out it looks fine to all the users, however if that email is then on forwarded or replied to then all buttons within the email duplicate. See below The one on the left is the original one.Workdrive backup and default storage selection
Hi community, I have been informed by Zoho that workdrive will be the default storage mechanism for crm and projects. These 2 Apps seem to have different design on how they interact with workdrive for storage. 1. Crm: seems to have files saves at theAsk the Experts 27: Onboarding and managing support reps
Hello everyone, We are back with our Ask the Experts (ATE) series for 2026. This year, we bring experts to help you address customer support challenges using Zoho Desk. For our first ATE, we are getting into the human side of customer support. "EveryCRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive
Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage filessync two zoho crm
Hello everyone. Is it possible to sync 2 zoho crm? what would be the easiest way? I am thinking of Flow. I have a Custom Module that I would like to share with my client. We both use zoho crm. Regards.Introducing Rollup summary in Zoho CRM
------------------------------------------Moderated on 5th July'23---------------------------------------------- Rollup summary is now available for all organizations in all the DCs. Hello All, We hope you're well! We're here with an exciting update thatEnhancements for Currencies in Zoho CRM: Automatic exchange rate updates, options to update record exchange rates, and more
The multi-currency feature helps you track currencies region-wise. This can apply to Sales, CTC, or any other currency-related data. You can record amounts in a customer’s local currency, while the CRM automatically converts them to your home currencyMarketer's Space: Why your open rates suddenly drop to zero?
Hey everyone—welcome back to another post in Marketer's Space! If you regularly track the performance of your campaigns, open rates are probably one of the first numbers you check after sending an email. So it can be quite alarming when a campaign suddenlyBulk upload images and specifications to products
Hi, Many users have asked this over the years and I am also asking the same. Is there any way in which we can bulk upload product (variant) images and product specifications. The current way to upload/select image for every variant is too cumbersome.I'd like to Import Contacts for an SMS Campaign, but they don't have email addresses.
I've got a contact list of about 4000 contacts. Some only have mobile phone and not email. Can I still import the contacts?How to sync overtime from zoho people to zoho payroll
Hi everyone, I’m currently setting up Zoho People with Zoho Payroll. While configuring the sync, I noticed I can only fetch Employee Profiles, LOP (Loss of Pay), and enable the Employee Portal. I can’t seem to find a way to automatically pull OvertimeBulk Associate Tasks/Task Lists to Milestones (Similar to Issues)
Hello Zoho Projects Team, We hope you are doing well. We would like to request an enhancement regarding Milestone management for Tasks in Zoho Projects. Current Behavior: In the Issues module, it is possible to select multiple issues and easily associateIs Zoho Sites still actively being developed?
Hello, Is Zoho Sites still actively being developed as part of the Zoho ecosystem? I noticed that the What's New page (https://www.zoho.com/sites/whats-new.html) does not show any updates since Q1 2025. We were considering migrating our website from SquarespaceAdding new data to the sheet with most recent data being added to top row.
I am using a form that I built from within sheets. When the form is submitted, is it possible to have the data entered in (for example) row 2 (since row 1 is the heading) instead of being added into the next available row which should be hundreds ofneed a packing list feature
In our business, goods listed on an invoice are packed in separate boxes and shipped off. for e.g. an invoice may have 10 items. each item could then be packed in different boxes depending on qty of each item. this packing list is as important as the invoice for purposes of shipping documents. Request you to add this feature asap.Editing Item Group to add Image
I did not have the image of the product when the Item Group was created. Now I have the product image, and would like to associate/add to the Item Group. However the Item Group Edit functionality does not show/allow adding/changing image. Please helNext Page