Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

    • Recent Topics

    • Allow Multiple Scheduled Appointments with Zoho Support

      Dear Zoho Team, I hope you're doing well. First, thank you for introducing the option to schedule support calls via the Zoho CRM booking link. This has been a fantastic enhancement, eliminating the need for back-and-forth coordination when scheduling

    • Share Edit while in Workflow

      There are times that while a writer document is in the middle of a workflow the user that's responsible for that stage of the workflow needs to bring in other people to work on the document. Currently there is no way for the user to be able to allow other's

    • How to add a custom hyperlinked button in ZohoCRM Canvas Builder?

      I am building custom views in ZohoCRM Canvas builder, with one view that will be used for a client Portal. Trying to add a custom button that when clicked opens a separate window to a ZohoForm. How do I do this?

    • Event Time Zone in meeting invites are confusing users

      When sending calendar invites to internal and external users, the first section "Event Time Zone" is confusing people and they are automatically declining events. Can this section please be removed??? It already shows the correct time zone next to the

    • How do you create an event/meeting in a different time zone?

      Does anyone know how do you create an event/meeting in a different time zone? 

    • Extend GitHub Commit Integration to Tasks in Zoho Projects

      Hello Zoho Projects Team, We hope you are doing well. We would like to submit a feature request regarding the current GitHub integration in Zoho Projects, specifically the ability to link commit information. Current Limitation: As of now, Zoho Projects

    • Add Native GitHub Pull Request Creation & Management from Tasks/Bugs

      Hi Zoho Projects Team, I am submitting a feature request to enhance our development lifecycle by enabling native GitHub Pull Request (PR) creation and management directly within Zoho Projects. The Current Limitation: Currently, the GitHub integration

    • Add Actionable “Smart Commits” for GitHub Integration

      Hi Zoho Projects Team, I am submitting a feature request to introduce Smart Commits support for the GitHub integration in Zoho Projects. The Current Limitation: Currently, Zoho Projects offers a "passive" integration. Including a Task ID in a commit message

    • Add Unified Project-Wide Development & Release GitHub Dashboard

      Hi Zoho Projects Team, I am writing to submit a feature request for a Unified Development & Release GitHub Dashboard that provides centralized GitHub visibility at both the Project and Milestone levels. The Vision: We need a single "Command Center" that

    • Add Native “Create GitHub Branch” Action for Tasks and Bugs

      Hi Zoho Projects Team, I would like to submit a feature request to enhance the GitHub integration in Zoho Projects by adding the ability to initiate a GitHub branch directly from a Task or Bug. The Current Limitation: Currently, Zoho Projects lacks a

    • One-click “Open in VS Code” from Zoho Tasks/Bugs with automatic branch checkout

      Hi Zoho Projects Team, I am submitting a feature request to enhance the development workflow within Zoho Projects, specifically the ability to open a Task or Bug directly in VS Code and automatically check out the related branch, similar to capabilities

    • How to reduce inventory valuation when not tracking individual items?

      We purchase clothing items wholesale, embroider them with a company's logo, then resell. The garment style, size, color, etc. changes according to a customer's desire, so we don't track the individual item's purchase within Zoho Inventory unless there's

    • Merge tag didn't work in the campaign sent to test email list

      Hello, I just sent out a campaign to my list of test emails (including my personal one). The merge tag that's supposed to put a first name of the contact $[UD:FIRST_NAME||]$ didn't work. Where the first name of the contact should be displayed there's

    • Add Camera Background Blur During Recording

      Hi Zoho WorkDrive Team, Hope everything is well. We would like to request an enhancement to the video recording feature in Zoho WorkDrive. Currently, the camera preview displayed during a recording does not support background blur. This is an essential

    • Sync Data from MA to CRM

      Currently, it's a one-way sync of data from the CRM to MA. I believe we should have the ability to select fields to sync from MA to the CRM. The lead score is a perfect example of this. In an ideal world we would be able to impact the lead score of a

    • How can I transfer data from Production to Development environment?

      Hi, I am using Creator V6 and would like to bring all the data in production to the Development and Testing environments? Is there an easy way of doing that or I have to export and import each table?

    • filter on sheets mobile (iOS, iPadOS) does not work

      re-posting this as a question because reporting as a problem netted zero responses... I have this issue on every spreadsheet, whether imported or created natively in Zoho Sheets. I can use on desktop without issue: but on iOS the filter dropdowns are

    • How to map fields from Zoho Recruit to Zoho People

      I've got these fields from my Job Offer that I'm trying to map to the Work information fields in Zoho People, but they arent showing up. For example, how do I get the department name field (in the job post) to map to the work information field in Zoho

    • Mass email from Report output

      Hi, I'd like to send a mass email based on a report output. The report is pulling multiple information from linked modules. Each line of the report ends up with a contact name, email and multiple field values pulled from the linked modules (some are custom

    • Introducing SlyteUI : From Idea to a Working Interface in Minutes

      Hello everyone! Are you spending hours building basic UIs? Does even the smallest customization feel like a major task? CRM customization should feel intuitive and straightforward, not time consuming or exhausting. SlyteUI makes this possible by simplifying

    • Add Comprehensive Accessibility Features to Zoho Writer

      Hello Zoho Writer Team, We hope you are doing well. We would like to submit a feature request to enhance Zoho Writer with a full set of accessibility tools, similar to the accessibility options already available in the Zoho Desk agent interface. 🚧 Current

    • How to see Tickets that require a response

      When an agent checks his work he should be able to see a list of tickets that he has to reply to. Currently he can only see if the ticket is closed or open. However the ticket might be open but we are waiting for the customers reply, so there is nothing

    • Zoho Books' 2025 Wrapped

      Before we turn the page to a new year, it’s time to revisit the updates that made financial management simpler and more intuitive. This annual roundup brings together the most impactful features and enhancements we delivered in 2025, offering a clear

    • [Free webinar] Integrations in Zoho Creator - Creator Tech Connect

      Hello everyone, We’re excited to invite you to another edition of the Creator Tech Connect webinar. About Creator Tech Connect The Creator Tech Connect series is a free monthly webinar featuring in-depth technical sessions designed for developers, administrators,

    • Issue with attaching files to a task through the API

      Hello! I've implemented a function that creates a task for every new bill that is created but I haven't been able to attach to the task the files which are attached to the bill. I have encountered multiple errors but the most common one is error 33003:

    • Trigger workflows from SLA escalations in Zoho Desk?

      Hey everyone, I’m currently working with SLA escalation rules in Zoho Desk and ran into a limitation that I’m hoping someone here has solved more elegantly. As far as I can tell, SLA escalations only support fairly limited actions (like changing the ticket

    • Can I hide empty Contact fields from view?

      Some contacts have a lot of empty fields, others are mostly filled. Is there a way I can hide/show empty fields without changing the actual Layout? I would like to de-clutter my view, and also be able to add information later as I am able. I would be

    • Agents permission per department

      Hi Team, can I setup permission for each agent what they can do in each department, for example I want account department agents to only have view access to support department tickets and not allowed to assign or reply to clients. I am sure this would

    • Ask the Experts - Live Q&A webinar

      Hello Community, We’re excited to host our very first Ask the Experts session! Join us on 7 April 2026 from 11 a.m. to 12 p.m. (IST) for this live webinar Q&A session, where you will have an opportunity to connect directly with our product experts, gain

    • Cliq iOS can't see shared screen

      Hello, I had this morning a video call with a colleague. She is using Cliq Desktop MacOS and wanted to share her screen with me. I'm on iPad. I noticed, while she shared her screen, I could only see her video, but not the shared screen... Does Cliq iOS is able to display shared screen, or is it somewhere else to be found ? Regards

    • Prevent Automatic Milestone Inheritance for Newly Created Task Lists

      Hello Zoho Projects Team, We hope you are doing well. We would like to request an enhancement regarding how new task lists inherit Milestone association in Zoho Projects. Current Behavior: At the moment, when a new task list is created below an existing

    • Hiding / removing Currency field from Layout

      Hello, I'm working on the LEAD standard layout of Zoho CRM. I like to have it as minimal as possible for our team. I could remove most of the field, but Currency is not possible. As lead, I don't quite understansd the necessity to have Currency as mandatory.

    • Updating Sales orders on hold

      Surely updating irrelevant fields such as shipping date should be allowed when sales orders are awaiting back orders? Maybe the PO is going to be late arriving so we have to change the shipment date of the Sales order ! Not even allowed through the api - {"code":36014,"message":"Sales orders that have been shipped or on hold cannot be updated."}

    • Can we have Backorder Management ?

      Can we have Backorder Management ?

    • Accept credit card payment for multiple invoices

      We have a problem with Zoho Books.  While we can charge a customer's credit card for an individual customer invoice, we cannot charge a customer's credit card for more than one invoice. In our business, it is common for a client to phone up and say, "Here

    • Anyone in Australia using Zoho Books AND has their account with NAB?

      Hi I have an account with both NAB and Suncorp. Suncorp transaction come in the next day however NAB transactions take 4-5 business days to appear. eg: A deposit made today in my Suncorp will be imported into Zoho tomorrow. A deposit made today to the NAB account will be imported maybe Saturday (Friday overnight). I have contacted both Zoho and NAB but noone seems to know why. I was just wondering if anyone else in Australia uses NAB and has this issue (or doesn't) maybe we could compare notes and

    • Newby Questions - Vendors, Customers, and Income vs. Other Income Clarifications

      Q1. For Deposits, there does not seem to be an option for "Income". "Other Income" is an option however. What is the process to add the option to assign a deposit to type "Income"? Q2. In many cases Vendors and Customer are the same. Vendors may purchase

    • ISO 3166-2 Compliant Country and Regions

      Zoho should consider implementing ISO 3166-2 standards for countries and regions in address dropdowns. It adds a lot of noise in the data when items added programmatically conform to ISO 3166-2 (e.g., TX for Texas, CA for Canada, etc.), but the few items

    • Tax on Imported goods charged by Shipping Company

      Hi Folks, I imported goods from outside Canada, for better understanding I will give an example data. imported goods value: 2000$ The shipping company sent me an invoice containing the following information: Custom duty on imported goods: 400$ Administration

    • Invoice Import Error: "Either Tax or Tax Exemption Reason is required" - Recurring Across All Attempts

      Issue Description: I'm experiencing a critical issue with invoice imports in Zoho Books. Every import attempt fails with the same error message, regardless of the tax data provided: Either Tax or Tax Exemption Reason is required. This error is blocking

    • Next Page