Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • File Upload field not showing in workflow

      Hi, I have added a field on Zoho CRM. I want to use it in a workflow where that particular field is updated based on another field, however it is not showing up in the field list to select it in the workflow. Why is this please?

    • CRM Custom function updating a module record shows the Super Admin user as the record modifier

      Dear Zoho CRM Team, Is there any way to update this so that when a custom function has updated a record the Super Admin user doesn't become the modifier? This happens on the record as a modifier and shows up in the audit logs. It would be more useful

    • Best practice importing items and matching assemblies

      Hi, I was wondering what would be the best practice to import items and composite items (assemblies) From my backup, what should I import first? The items or the composite items? I am on Zoho one, using inventory and books. Kind regards, Sabine

    • Best way to fetch employee names from Zoho People into Zoho Creator Inventory Stock Form field Employee Name Lookup

      Hi Team, I have a requirement in my Zoho Creator application (Inventory Stock Adjustment) where I need to fetch employee names from Zoho People and use them as a lookup in a form. Currently, I am considering using an integration field to fetch this data

    • get file api is returning Junk data

      I am working on extension development where at one point I need to retrieve attachments on records I found out I can use only invokeconnection and not invokeurl in extension development The invoke connection returns the image in raw binary format. When

    • Need help with message box

      End user updates many records at once. Each have unique serial number. They want a confirmation box that says starting number, ending number and qty. Is there any way to do this? I been searching and asking support but seems like no great solution.

    • [Webinar] The Transformative Power of Gen BI

      Traditional decision-making tools are no longer enough. The integration of generative AI into business intelligence (BI) is proving to be a true game changer—enabling businesses to make faster, smarter, and better informed decisions. Early adopters of

    • Backorder process review - Automating Removal of Sales Order from "On Hold" When PO is Received

      Hello Zoho Inventory Team, Currently, sales orders in On Hold status are released only when the bill for the purchase order is created. In our workflow, it would be much more efficient if the sales order could automatically move out of On Hold as soon

    • How to access the saved Query using API?

      I have created a query in setting. Is it possible to access it from a API? What's the endpoint? I tried /v8/queries/important_deals but didnt' work.

    • création d'une base de données

      base de donnee d'un lycée

    • User Session variables to recall after form submit & redirect to menu

      Hey, Forgive me if this is a simple answer I have overlooked. I have built a big onboarding process for my SAAS platform which is built and managed by an external Dev team. I am hoping to embed my new creator app on the SAAS platform. I have a menu page

    • Zoho Writer Merge Template with data from Zoho Analytics Views

      Hello, Is there any way to pull information from Zoho Analytics and put it into a Merge Template? I am trying to create a bulk export of one report that filters on a field to create a single document for each of the filter fields.

    • Having to enter bill before stock shows as available.

      Hi, Am I right in thinking you must create a bill from the purchase order receipt before the goods are available for shipping?

    • Agent Availability Report

      From data to decisions: A deep dive into ticketing system reports Businesses need to track when their support agents check in and check out from work. This report, titled Agent Availability, is one of the static reports that helps managers track the number

    • Writing by Hand in "Write" Notes

      Hi there! I just downloaded this app a few moments ago, and I was wondering if there was a way to write things by hand in "Write" mode instead of just typing in the keyboard. It would make things a bit more efficient for me in this moment. Thanks!

    • AI-driven construction analytics using Zoho Projects Plus

      Construction projects thrive on timelines; if a project is completed on or before the expected time, it makes headlines, but if it gets delayed, the company’s credibility will be at risk. Such projects require precise planning and constant monitoring,

    • Canadian payroll

      Hello ZOHO, is any updates on when payroll for Canada will be available?

    • Unified customer view: Bringing product intelligence into your CRM data

      For businesses today, applications and digital interfaces are the main touch points in the customer journey. While Zoho CRM data already tells who your customers are and what they buy, Zoho Apptics tells you how they interact and engage with your product.

    • Zoho Creator SaaS app with multiple client AND multiple users

      I am close to finalizing a Zoho Creator app that will be used by different companies to manage sale info, and each company can have multiple users using the app. Things were going perfectly until I tried a search on a report. I use a database field in each table to separate the data for each company, which works great. BUT when I do a search, I get to see ALL the data. This means that Company A could easily see the data from Companies B and C via the built-in search. Is there a way to set up the

    • Project Change Orders and Additions

      We are in the process of migrating from QuickBooks Online to Zoho Books. We have Zoho One and like the ability to sync all of our data across everything. And I like that projects work in a way that's less dumb than QuickBooks. I'm trying to figure out

    • Track online, in-office, and client location meetings separately with the new meeting venue option

      Hello everyone! We’re excited to announce meeting enhancements in Zoho CRM that bring more clarity and structure to how meetings are categorized. You can now specify the meeting venue to clearly indicate whether a meeting is being held online, at the

    • Possible to send Zoom AI Companion transcripts and summaries to contacts in CRM?

      Title says it all. Is it possible to send Zoom AI Companion transcripts and summaries to contacts in Zoho CRM?

    • How do I edit the Calendar Invite notifications for Interviews in Recruit?

      I'm setting up the Zoho Recruit Interview Calendar system but there's some notifications I don't have any control over. I've turned off all Workflows and Automations related to the Calendar Scheduling and it seems that it's the notification that is sent

    • Posibility to add Emoticons on the Email Subject of Templates

      Hi I´ve tried to add Emoticons on the Subject line of Email templates, the emoticon image does show up before saving the template or if I add the Emoticon while sending an Individual email and placing it manually on the subject line. Emoticons also show

    • Clarity on extended contract status

      Clarity on “extended” status- How does Zoho “extend” a contract?  E.g. if client extends by 1 month, can the “end date” be adjusted accordingly in Zoho and tagged as “Extended” with a clear audit trail that also captures the client’s email? Note- Email

    • Customizing contract status

      Can we customize/add/remove status on Zoho contracts ourselves or does Zoho needs to do this? Context- There is a long list of status on Zoho but not all of them are relevant for us . There are few status which are missing and needs to be added. E.g-

    • Set Custom Business Calendars and Holidays for Global Teams

      Managing a project across diverse teams means accounting for more than just tasks and deadlines; it means acknowledging how and when each team actually works. Users might follow different working days or observe region-specific holidays that cannot be

    • [Integration edition] Deluge learning series - Integrations between Zoho apps and Third-party services with Deluge | 28 August 2025

      We’re excited to welcome you to the Deluge learning series: Integration edition! The Integration edition will run for three months: Session 1 – Integrating Zoho Apps with Deluge using inbuilt integration tasks Session 2 – Integrating Zoho Apps with Deluge

    • Critical Vulnerability in all major password manager plugins - is Zoho Vault affected?

      Hi Zoho, a security researcher found a critical clickjacking vulnerability in all major browser password managers: DOM-based Extension Clickjacking: Your Password Manager Data at Risk | Marek Tóth Is Zoho Vault affected as well? Apparently the Vault extension

    • Introducing SecureForms in Zoho Vault

      Hey everyone, Let’s face it—asking someone to send over a password or other sensitive data is rarely straightforward. You wait. You nudge. You follow up once, twice—maybe more. And when the information finally arrives, it shows up in the worst possible

    • Tables from ZohoSheets remove images when updated from source

      I have a few tables from a ZohoSheet in a ZohoWriter document that will remove the images in the cells when I refresh from the source. The source still has the images in the table when I go to refresh. After updating from the source, as you can see the

    • Change eMail Template for Event-Invitations

      Hello ZOHO-CRM Team How I can change the eMail Template for Event-Invitations? I work with the German Version of the Free Version. I know how I can modify eMail alerts or Signature Templates, but where I can other eMails modify you send out? Thank you for your answer. Regards, Juerg

    • Need Your Insights

      Hi Zoho, I'm confused why the flow only sends to one output. setVariable15 is from a list. It doesn't consider the 2nd entry. Any thoughts?

    • Map Dependency Fields - How can can I limit picklist values from a multi-select Parent field?

      I know the "Map Dependency Fields" feature doesn't allow a multi-select parent field. However, I need the functionality. Can I do it through a deluge function, and would it still support color coding picklists? We need the color coding for quick scanning

    • DYK 1 - Color Palette for Enhanced Visual Identification of Status

      Introducing the Did You Know series of posts. The goal of this series is to familiarize users with certain features or enhancements in Zoho Projects that may not be evident at first glance. The first post in this series deals with color palettes for indicating

    • Invalid field in the COQL query

      Dear Zoho Support! I believe that you already helped me with a similar problem a few years ago. One of my clients has a custom field named "LOB" in the "Deals" Module (see the field's metadata below). The COQL query using this field: : "select id, Deal_Name,

    • Automating Employee Birthday Notifications in Zoho Cliq

      Have you ever missed a birthday and felt like the office Grinch? Fear not, the Cliq Developer Platform has got your back! With Zoho Cliq's Schedulers, you can be the office party-cipant who never forgets a single cake, balloon, or awkward rendition of

    • Transferring domain registration to new registrar and switching email hosting at the same time?

      I need to transfer an existing domain uv cure adhesive that's currently with SiteGround to Porkbun. I also need to move the existing custom email addresses from SiteGround to Zoho Mail. I'm not sure if I should transfer the domain first and then tackle

    • Split deposits

      Can Zoho do split deposits. One deposit, two checks for two separate invoices from different customers. This is one of the most common tasks I can imaging. When I mark the two invoices paid, there are two deposits in bank register. When I try to match,

    • Deactivate Desk Contact without Deleting Contat

      We have a client who has multiple tenants for regulatory purposes, and as such, has a few users that have email addresses in both tenants. They've then emailed into the ticketing system, so we have multiple contacts (no big deal, we want to keep their

    • Next Page