Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • Does Zoho Learn integrate with Zoho Connect,People,Workdrive,Project,Desk?

      Can we propose Zoho LEarn as a centralised Knowledge Portal tool that can get synched with the other Zoho products and serve as a central Knowledge repository?

    • Zoho Flow - Update record in Trackvia

      Hello, I have a Flow that executes correctly but I only want it to execute once when a particular field on a record is updated in TrackVia. I have the trigger filters setup correctly and I want to add an "update record" action at the end of the flow to

    • Add Comprehensive Accessibility Features to Zoho Desk Help Center for End Users

      Hello Zoho Desk Team, We hope you're doing well. We’d like to submit a feature request to enhance the client-facing Help Center in Zoho Desk with comprehensive accessibility features, similar to those already available on the agent interface. 🎯 Current

    • Rename Record Summary PDF in SendMail task

      So I've been tasked with renaming a record summary PDF to be sent as part of a sendmail task. Normally I would offer the manual solution, a user exports the PDF and uploads it to a file upload field, however this is not acceptable to the client in this

    • in zoho creator Sales Returns form has sub form Line Items return quantity when i upate the or enter any values in the sub form that want to reflect in the Sales Order form item deail sub form field Q

      in zoho creator Sales Returns form has sub form Line Items return quantity when i upate the or enter any values in the sub form that want to reflect in the Sales Order form item deail sub form field Quantity Returned\ pls check the recording fetch_salesorder

    • Estimates with options and sub-totals

      Hi It seems it would be great to be able to show multiple options in an estimate. For instance I have a core product to which I can add options, and maybe sub-options... It would be great to have subtotals and isolate the core from the not compulsory items. Thanks

    • Optional Items Estimate

      How do you handle optional items within an estimate? In our case we have only options to choose with. (Like your software pricing, ...standard, professional, enterprise) How can we disable the total price? Working with Qty = 0 is unprofessional....

    • Important Update : Zendesk Sell announced End of Life

      Hello Zendesk users, Zendesk has officially announced that Zendesk Sell will reach its End of Life (EOL) on August 31, 2027 (Learn more). In line with this deprecation, Zoho Analytics will retire its native Zendesk Sell connector effective October 1,

    • Zoho Sheets

      Hi, I am trying to transition into Zoho sheets, I have attached the issues encountered. Server issues, file trying to upload for more than 30 mins, even once uploaded my data aren't loaded. Simple calculations are not working I have attached the sample.

    • Zoho CRM + Zoho FSM : alignez vos équipes commerciales et techniques

      La vente est finalisée, mais le parcours client ne fait que commencer ! Dans les entreprises orientées service, conclure une vente représente seulement la première étape. Ce qui suit — installation, réparation ou maintenance régulière — influence grandement

    • Top Bar Shifting issue still not fixed yet

      I mentioned in a previous ticket that on Android, the top bar shifts up when you view collections or when you're in the settings. That issue still hasn't been fixed yet. I don't wanna have to reinstall the app as I've noticed for some reason, reinstalling

    • Power of Automation:: Automate the process of updating project status based on a specific task status.

      Hello Everyone, Today, I am pleased to showcase the capabilities of a custom function that is available in our Gallery. To explore the custom functions within the Gallery, please follow the steps below. Click Setup in the top right corner > Developer

    • ZOHO SHEETS

      Where can I access desktop version of zoho sheets? It is important to do basic work If it is available, please guide me to the same

    • Attention API Users: Upcoming Support for Renaming System Fields

      Hello all! We are excited to announce an upcoming enhancement in Zoho CRM: support for renaming system-defined fields! Current Behavior Currently, system-defined fields returned by the GET - Fields Metadata API have display_label and field_label properties

    • Billing Management: #3 Billing Unbilled Charges Periodically

      We had a smooth sail into Prorated Billing, a practice that ensures fairness when customers join, upgrade, or downgrade a service at any point during the billing cycle. But what happens when a customer requests additional limits or features during the

    • No bank feeds from First National Bank South Africa since 12 September

      I do not know how Zoho Books expects its customers to run a business like this. I have contacted Zoho books numerous times about this and the say it is solved - on email NO ONE ANSWERS THE SOUTH AFRICAN HELP LINE Come on Zoho Books, you cannot expect

    • Citation Problem

      I had an previous ticket (#116148702) on this subject. The basic problem is this; the "Fetch Details" feature works fine on the first attempt but fails on every subsequent attempt, Back in July after having submitted information electronically and was

    • Open Sans Font in Zoho Books is not Open Sans.

      Font choice in customising PDF Templates is very limited, we cannot upload custom fonts, and to make things worse, the font names are not accurate. I selected Open Sans, and thought the system was bugging, but no, Open Sans is not Open Sans. The real

    • Failing to generate Access and Refresh Token

      Hello.  I have two problems: First one when generating Access and Refresh Token I get this response:  As per the guide here : https://www.zoho.com/books/api/v3/#oauth (using server based application) I'm following all the steps. I have managed to get

    • Zeptomail 136.143.188.150 blocked by SpamCop

      Hi - it looks like this IP is being blocked, resulting in hard bounces unfortunately :( "Reason: uncategorized-bounceMessage: 5.7.1 Service unavailable; Client host [136.143.188.150] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?136.143.188.150

    • Apply transaction rules to multiple banks

      Is there any way to make transaction rules for one bank apply to other banks? It seems cumbersome to have to re-enter the same date for every account.

    • How to bulk update records with Data Enrichment by Zia

      Hi, I want to bulk update my records with Data Enrichment by Zia. How can I do this?

    • Need Guidance on SPF Flattening for Zoho Mail Configuration

      Hi everyone, I'm hoping to get some advice on optimizing my SPF record for a Zoho Mail setup. I use Zoho Mail along with several other Zoho services, and as a result, my current SPF record has grown to include multiple include mechanisms. My Cloudflare

    • How do I split a large CSV file into smaller parts for import into Zoho?

      Hi everyone, I’m trying to upload a CSV file into Zoho, but the file is very large (millions of rows), and Zoho keeps giving me errors or takes forever to process. I think the file size is too big for a single import. Manually breaking the CSV into smaller

    • Client Script Payload Size Bug

      var createParams = { "data": [{ "Name": "PS for PR 4050082000024714556", "Price_Request": { "id": "4050082000024714556" }, "Account": { "id": "4050082000021345001" }, "Deal": { "id": "4050082000023972001" }, "Owner": { "id": "4050082000007223004" }, "Approval_Status":

    • Sync Issue Between Zoho Notebook Web App on Firefox (PC) and Android App

      Hi Zoho Notebook Community, I'm facing a sync problem with Zoho Notebook. When I use the web version on Mozilla Firefox browser on my PC, I create and save new notes, and I've synced them successfully. However, these new notes aren't showing up in my

    • Messages not displayed from personal LinkedIn profile

      Hello. I connected both our company profile and my personal profile to Zoho social. I do see all messages from our company page but none from my private page. not even the profile is being added on top to to switch between company or private profile,

    • lead convert between modules

      Hello, The workflow we set up to automatically transfer leads registered via Zapier into the Patients module to the Leads module started to malfunction unexpectedly on September 25, 2025, at 11:00 AM. Under normal circumstances, all fields filled in the

    • Flow Task Limits - How to Monitor, Understand Consumption?

      So, I got an email last night saying that I've exhausted 70% of my tasks for this month, and encouraging me to buy more tasks. I started to dig into this, and I cannot for the life of me figure out where to find any useful information for understanding,

    • Cross References Do Not Update Correctly

      I am using cross references to reference Figures and current am just using the label and number, i.e. Figure #. As seen here: When I need to update the field, I use the update field button. But it will change the cross reference to no longer only including

    • Manage control over Microsoft Office 365 integrations with profile-based sync permissions

      Greetings all, Previously, all users in Zoho CRM had access to enable Microsoft integrations (Calendar, Contacts, and Tasks) in their accounts, regardless of their profile type. Users with administrator profiles can now manage profile-based permissions

    • How to Track and Manage Schedule Changes in Zoho Projects

      Keeping projects on track requires meticulous planning. However, unforeseen circumstances can cause changes to schedules, leading to delays. It becomes important to capture the reason for such changes to avoid them in the future. Zoho Projects acknowledges

    • Is there a notification API when a new note is addeding

      Trying to push to Cliq, or email notification when there's a new note added in module. How to implement this?

    • Zoho Sheet - Desktop App or Offline

      Since Zoho Docs is now available as a desktop app and offline, when is a realistic ETA for Sheet to have the same functionality?I am surprised this was not laucned at the same time as Docs.

    • Collaborate Feature doesn't work

      Hello Team. It seems that the collaborate section is broken? I can post something but it all appears in "Discussions". In there is no way how I would mark something as Draft, Approval, post or any of the other filter categories? Also if I draft a post

    • Edit Permission during and after approval?

      When a record is sent for approval Can a user request for edit permission from the approver? We don't want to give edit permissions for all the records under approval Only on a case-by-case basis How can we achieve this?

    • Zoho web and mobile application not workingn

      Both zoho forms web and mobile application aren't working. I have checked my network connections and they are fine.

    • Introducing the revamped What's New page

      Hello everyone! We're happy to announce that Zoho Campaigns' What's New page has undergone a complete revamp. We've bid the old page adieu after a long time and have introduced a new, sleeker-looking page. Without further ado, let's dive into the main

    • Prevent stripping of custom CSS when creating an email template?

      Anyone have a workaround for this? Zoho really needs to hire new designers - templates are terrible. A custom template has been created, but every time we try to use it, it strips out all the CSS from the head.  IE, we'll define the styles right in the <head> (simple example below) and everything gets stripped (initially, it saves fine, but when you browse away and come back to the template, all the custom css is removed). <style type="text/css"> .footerContent a{display:block !important;} </style>

    • Bulk Moving Images into Folders in the Library

      I can't seem to select multiple images to move into a folder in order to clean up my image library and organize it. Instead, I have to move each individual image into the folder and sometimes it takes MULTIPLE tries to get it to go in there. Am I missing

    • Next Page