Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • Is the ChatGPT Assistant integration capable of recognizing WhatsApp voice messages?

      I was wondering: if a visitor sends me a voice message on WhatsApp, would the assistant be able to transcribe it and reply to them?

    • Zoho Creator to Zoho CRM Images

      Right now, I am trying to setup a Notes form within Zoho Creator. This Notes will note the Note section under Accounts > Selected Account. Right now, I use Zoho Flow to push the notes and it works just fine, with text only. Images do not get sent (there

    • 【Zoho CRM】レポート機能のアップデート

      ユーザーの皆さま、こんにちは。コミュニティチームの藤澤です。 今回は「Zoho CRM アップデート情報」の中から、レポート機能のアップデートをご紹介します。 目次 1. レポートのエクスポート時のレコードIDの表示について 2. 通貨項目の表示について 3. レポートの削除の監査ログへの反映について 1. レポートのエクスポート時のレコードIDの表示について これまで、レポートをエクスポートするとファイルにレコードIDが必ず含まれていました。レコードIDが識別子として役立つ場合もありますが、実際には多くの企業で参照されることはありません。

    • Translation in zoho bookings

      We cant translate zoho booking emails. The general text we can change. But what about text like: ""Here a link to join the meeting online:"" and "Add to Zoho Calendar" and "Add to Google Calendar"? No professional business have mixed languages. Its looking

    • Is there any way to bill one client in different currencies on different invoices?

      I have some customers who have their currency set as USD and most of their billing is done in USD.   However, from time to time I have a need to bill them in my base currency GBP for some specific invoices, but there seems to be no way of doing this that I can see. The only workaround that I can see is to create two client records for the same client, one for USD billing and one for GBP billing, but this is not an ideal situation. Is it likely that the (hopefully!) soon to arrive multi-currency support

    • API name for all fields in Zoho Project (Standard or custom)

      Hi! I struggle to find easily all API name of all field in Zoho Project to build my API rest with other services. We can find them very fast in CRM but not in PRoject.   Could you share a function to get the list of all API Name of any field of an App

    • Disappearing Mouse cursor in Zoho Mail / Windows 11 (Chrome + Edge)

      I'm seeing an issue when writing mails with the light theme with the mouse cursor being white and the document area also being white - making it nearly impossible to see the mouse cursor. I see the problem on Windows 11 under Chrome and Edge. (Yet to

    • Zoho Assist not rendering NinjaTrader chart properly

      Hi everyone. Just installed and testing Zoho Assist. I want to display my laptop' screen (Windows 11) on a monitor connected to my Mac mini. The laptop is running a stock trading program called NinjaTrader. Basically, when running, this program displays

    • Link(s) between Notes

      Hello Everyone, It would be great if links could be created between notes. Let's say we have 5 Notes A, B, C , D, E. I would like to be able to link Note A to Note B but not in other way, so no link appears in Note B linking to Note A. An so on, linking

    • Zoho Books | Product updates | August 2025

      Hello users, We’ve rolled out new features and enhancements in Zoho Books. From the right sidebar where you can manage all your widgets, to integrating Zoho Payments feeds in Zoho Books, explore the updates designed to enhance your bookkeeping experience.

    • Feature Request: Assign Documents to Already Entered Bills, Expenses, Invoices, etc.

      Hi Zoho Team, We are regular users of the Documents module in Zoho Books and appreciate its ability to keep financial records well-organized. However, we’ve noticed a limitation: There is no way to attach a document from the "Documents > Files" section

    • I don't see any WITHDRAWL transaction at all

      Hi I manually imported my bank statement to Zoho books today and I am a complete newbie. I have been reading the knowledgebase but unable to fix this. I only see "Uncategorized 91 DEPOSIT transactions". I don't see any WITHDRAWL transaction at all. Also,

    • Shared inbox unable to see replies

      Hi we are a small company me and someone else, we have a shared inbox for our sale@ and contact@ however we have this issue where by if i reply to an email or the other person reply to the email, it does not show it to them and therefore we end up replying

    • Kaizen #136 - Zoho CRM Widgets using ReactJS

      Hey there! Welcome back to yet another insightful post in our Kaizen series! In this post, let's explore how to use ReactJS for Zoho CRM widgets. We will utilize the sample widget from one of our previous posts - Geocoding Leads' Addresses in ZOHO CRM

    • 404 error at checkout

      Our customers are getting a 404 error at checkout. Anyone else with the same problem?

    • FONT Sizing in Notebook

      Hi Kishore - What is the status of adding font sizing to the application? I have several things that I have pasted directly into Notebook and the fonts are HUGE! I would like the ability to highlight them and reduce the font to a legible size. Nothing

    • Can managers Upload documents to their direct rapports?

      Admin employees have the ability to upload documents to employees' files; however, managers do not have add/manage button - is it possible for managers to upload their direct reports' documents, such as absence documents or 121 documents. Is there something

    • Leave balance display for next year

      Is there a way to not have a rollover or not limit the leave balance depending on the date. For example an employee has 10 days leave balance and wants to apply for January leave in December. They cant because the rollover doesnt show the leave balance

    • Please add an “Auto-Apply Unused Credits” toggle

      Hello — please add a simple org-level option to automatically apply unused credits (credit notes, excess payments, retainers) to new invoices and/or bills. An ON/OFF toggle with choices “invoices”, “bills”, or “both” would save lots of manual work for

    • Zoho Books not working/loading

      Hi! I haven't been able to access/load Zoho Books for the past hours. I get a time out (and it is not due to my internet connection). Could you please check this asap? Thank you!

    • Custom Fields with Data Types for Expense and Payments Received in Zoho Books

      Hi all, We are glad to present to you, the option to create Custom Fields for the Expense and Payments received modules in Zoho Books. This also comes with an icing on top of it - Yes, the custom fields can now be created with different data types. Types like Text, Number, Decimal, Amount, Auto Number and Check Box are supported as of now. Rush to the gear icon at the top right corner, select 'More Settings', choose 'Preferences' in the left pane. Click the Expense/Payment preferences where you can

    • Zoho Cliq - Incident alert (Server outage - IN DC) | August 28

      We've received server down alerts and are currently investigating the issue (IN DC) to find the root cause. Our team is actively working to restore normal operations at the earliest. Status: Under investigation Start time: 09:44:21 AM IST Affected location:

    • Claude + MCP Server + Zoho CRM Integration – AI-Powered Sales Automation

      Hello Zoho Community 👋 I’m excited to share a recent integration we’ve worked on at OfficehubTech: ✅ Claude + MCP Server + Zoho CRM This integration connects Zoho CRM with Claude AI through our custom MCP Server, enabling intelligent AI-driven responses

    • How can I see content of system generated mails from zBooks?

      System generated mails for offers or invices appear in the mail tab of the designated customer. How can I view the content? It also doesn't appear in zMail sent folder.

    • Limitations on editing a message in Cliq

      Hi I've checked the documentations and there's no mention of how many times a message can be edited. When trying with code, I get various numbers such as ~1000 edits or so. Please mention if there's a limit on how many times one can change a message via

    • Narrative 10: Assignment Rules - Streamlining Ticket Management

      Behind the scenes of a successful ticketing system: BTS Series Narrative 10: Assignment Rules - Streamlining Ticket Management In the complex world of customer support, a flood of incoming tickets can hit the help desk in seconds. Businesses must do more

    • Free webinar! Digitize recruitment and onboarding with Zoho Sign and Zoho Recruit

      Hello, Tired of being buried in onboarding paperwork? With the integration between Zoho Sign and Zoho Recruit, a powerful applicant tracking system, you can digitize and streamline the entire recruitment and onboarding process, all from one platform.

    • Custom form - Duplicate Value Present

      I am new to Zoho People. I create a new form on Zoho People quite simple. A date (by default the current date) and a radio button with 3 options (Yes, No, Not applicable) I defined the date as ‘No duplicate’ as only one entry by date is allowed. I added:

    • Zoho API connection issues

      Hi, Today at around 1930 GMT our application started experiencing intermittent timeouts from the Zoho API. These intermittent timeouts are not enabling our app to work properly. The API connection was working just fine before. HTTPSConnectionPool(host='www.zohoapis.com',

    • Query Regarding our Partnership between AIC-JKLU and Zoho

      Dear Zoho Team, I am writing to raise a concern on behalf of AIC-JKLU, one of Zoho’s incubator partners. Recently, our startups have been facing difficulties while trying to get themselves onboarded on Zoho through our dedicated partner link. Unfortunately,

    • Disputed Purchase Invoices

      We have recently moved to Zoho Books from Sage. In Sage we were able to post a purchase/vendor invoice but mark it as on dispute. This would usually be a pricing query or if something was damaged. It would show in their ledger, so we could agree their

    • Getting events in the future

      Hi I am trying to get events in the future by calling this API Endpoint https://www.zohoapis.eu/crm/v8/Events?fields=Event_Title,Created_By,Created_Time,Start_DateTime But that gives me all events in the database. How do I make a query that returns all

    • Created Date/Invalid Fields

      Since Saturday we have suddenly had issues with our webhooks and data retrieval from CRM. Specifically how Created Date is handled. It appears there was some sort of change within CRM that broke a lot of our code that has been in place for several years.

    • add employees and customiz

      I want a Zoho software to track employees record. I logged in a demo so now I tried adding employee, first thing when I click on add employee it takes me to a page where it says add user maximum 10 users, my first doubt why users I don't want employees

    • Revenue Management: #9 Revenue Recognition in Media & Publishing

      Media & Publishing industry has evolved in recent times. It offers subscriptions, bundles digital and print access, runs sponsored content, and sometimes even sells ad spaces. If you run a media or publishing business, you will always get into a situation

    • Zoho CRM Community Digest - July 2025 | Part 2:

      Hello, Everyone! We’re closing out July with a can’t-miss highlight: Zoholics Europe 2025! Happening from September to October, it’s your chance to level up your CRM skills, covering everything from automation and CPQ to dashboards and advanced workflows.

    • How can I trigger a flow action only once while updating contact?

      Hi, we have a trigger to merge&mail file when the field YYY is filled out. For this acion I used "Create or update module entry". But unfortunately we get tens of email on a day with this merged file, because the contact is being regularly updated. The

    • Copy a Record Template from one Form to another

      I have a Creator application with several forms.  I developed a record template for one of the reports/forms but want to use most of it for another of the form/report combinations in the application. Is there a way to copy the template (code or otherwise) to another form?

    • Tip of the Week #70 – Create common team signatures for your shared inboxes

      Did you know that a small detail, such as an email signature, can make a big difference in how your brand is perceived? One simple yet smart way to enhance your team’s communication is by creating common team signatures for your shared inboxes. Instead

    • Enhanced data export features: XLSX format, custom character encoding, and selective record export

      Greetings all, Here are a few enhancements related to exporting CRM data, including the ability to export data in XLSX file format now. The Export feature under Data Administration now offers new options that expand its flexibility and enable users to

    • Next Page