Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Celebrating the power of visuals

      On World Photography Day, we would like to highlight the power of visuals in customer service. Visuals promote learning and understanding in less time. A photograph can rekindle a memory, convey more than words, and give a fresh perspective. How does

    • Customer address in Zoho Bookings

      Hello,  Is it possible to add customer address information to the Zoho bookings appointment screen? Or have it pull that information automatically from the CRM? We are wanting to use this as a field management software but it is difficult to pull the address from multiple sources when it would be ideal to have a clickable address on the appointment screen that opens up the user's maps.  It would also be advantageous for the "list view" to show appointment times instead of just duration and booking

    • Blocking / black listing customers

      Hi, We have a situation, we observed that certain customers are blocking multiple appointments with our advsiors but not showing up. Some of these are repeat offenders. This leads to those service hours getting blocked and not available for genuine customers.

    • Option to block bookings from specific email address or ip adresss in zoho booking

      Sometime few of our client keep booking irrelevant booking service just to resolve their queries and they keep booking it again and again whenever they have queries. Currently its disturbing our current communication process and hierarchy which we have

    • Feature Request: Email Templates for notifications accross all services

      Currently in Zoho Bookings, email notifications (such as booking confirmations, reminders, and cancellations) must be customized individually for each service. This becomes time-consuming and error-prone when managing multiple services that require consistent

    • Allow customers to choose meeting venue and meeting duration on booking page

      My business primarily involves one-to-one meetings with my clients. Given the hybrid-work world we now find ourselves in, these meetings can take several forms (which I think of as the meeting "venue"): In-person Zoom Phone call I currently handle these

    • Approval-based booking with Zoho Creator and Zoho Bookings

      Hi community members, We have developed a workaround for approval-based booking using Zoho Creator and Zoho Bookings! This provides a temporary solution as we work on the native feature, and it's useful for anyone needing an approval workflow when confirming

    • Member Accounts in Related List

      Hi Team, Currently, when a parent account is associated with an account in FSM, there is no related list displaying the associated member accounts under the parent account’s related list section. To view member accounts, I have to manually search using

    • how do i remove a specific Zoho Service from my account

      I no longer need Zoho CRM, ZRM Assist nor ZRM BugTracker. How do I remove them from the list of apps for my account?

    • I Want migarte all invoice details to zoho sheets

      I want to migrate all existing invoice details to Zoho Sheet, and automatically update the sheet whenever a new invoice is created.

    • were can i find my invoices i need this for my accountant

      were can i find my invoices i need this for my accountant, how can i get id direct to my email?

    • ONLY email field not populating Writer fillable document (randomly)

      I have a Zoho Writer fillable document that has pulled all my data from my Zoho Sheets file, EXCEPT the email column. It pulled every data before and after that column with no issues. Screenshots attached. It's not my first time using the app or the feature,

    • My number is marked as spam

      Hello Zoho Mail Support, My phone number was incorrectly flagged as “spam” during sign-up. This is my personal number, and I have not engaged in any spam activities. Kindly review and verify my account so I can proceed with my email setup. Thanks.

    • Personnalisation des paramètres dans Zoho Mail

      Pourquoi cela compte-t-il ? La personnalisation des paramètres dans Zoho Mail permet aux administrateurs de configurer l’environnement de messagerie en fonction des besoins spécifiques de leur organisation. Que ce soit pour alléger l’interface pour certaines

    • How To Save Data Into Zoho CRM Sandbox

      Hi Community, I want to save data into my zoho sandbox , for this I am using this api endpoint - https://www.zohoapis.com/crm/v8/Patients but I am getting this error - { "success": false, "message": "Zoho API request failed", "error": { "code": "INVALID_MODULE",

    • Automate pushing Zoho CRM backups into Zoho WorkDrive

      Through our Zoho One subscription we have both Zoho CRM and Zoho WorkDrive. We have regular backups setup in Zoho CRM. Once the backup is created, we are notified. Since we want to keep these backups for more than 7 days, we manually download them. They

    • Question about retrieving unsubscribed contacts (outside of lists) via API

      Hello, I am currently using Zoho Marketing Automation and would like to integrate it with our company’s core system. For this purpose, I am exploring the API options available to retrieve contact information. Specifically, I would like to know if there

    • Getting “mandatory field missing: Service_Line_Items” When Creating Work Order via Zoho Flow Deluge

      Hi Team, I’m trying to create a Work Order in Zoho FSM with only a Service Line Item (no Parts). However, I keep getting this error: Work Order Response: {"code":"MANDATORY_NOT_FOUND","details":{"api_name":"Service_Line_Items"},"message":"required field

    • How to customize the colors of the Client Portal login screen and add the company logo?

      As title, how to customize the colors of the Client Portal login screen and add the company logo?

    • Daily updates/fixes and how to see what was changed?

      When I receive the notification that zoho was updated and I need to refresh it. How can I see what was changed or fixed? Sometimes they change things that effect my books and I need to know what they did. For example over this past weekend something was

    • Upcoming Change: Snowflake Username/Password Authentication Deprecation – Action Required

      Hello Users, Snowflake has officially announced that username and password-based authentication will be deprecated by November 2025. You can find the official announcement [here]. If you're using a Snowflake connection in Zoho Analytics to import data,

    • Why should I choose Zoho Inventory vs Odoo?

      Hello there! I have used Zoho in different companies I've worked in, and I have a positive perception of it. I am starting a new import business for pipes, tubes, fittings, valves, elbows, etc., which all have serial numbers, cast numbers, etc., so I

    • Product Updates in Zoho Workplace applications | July 2025

      Hello Workplace Community, Let’s take a look at the new features and enhancements that went live across all Workplace applications this July. Zoho Mail Import bookmarks from Pocket Worried about losing your Pocket bookmarks? Don't worry we have got you.

    • PLEASE FIX YOR BUGS

      PICTURES ARE BEING REJECTED DESPITE THEM FOLLOWING THE GUIDELINES ON DIMENTIONS.

    • Kaizen# 204 - Answering Your Questions | Perform Field Updates before Blueprint transition via Client Script

      Hello everyone! Welcome back to another exciting Kaizen post. One of the questions we received through your Kaizen feedback was: “How can I update fields before Blueprint transition and how to prevent a transition based on a condition using Client Script?”

    • Create online meetings for Booking Pages with Zoho Meetings and Zoom

      Greetings, We hope you're all doing well. We're excited to share some recent enhancements to Bigin's Booking Pages. As you know, Booking Pages let you create public pages to share your availability so that your customers can easily book time slots with

    • Filters in audit logs

      Greetings, I hope all of you are doing well. We're happy to announce a few recent enhancements we've made to Bigin. We'll go over each one in detail. Previously, there were no filters available to narrow down data in audit logs. Now, we've introduced

    • Enhanced help options in Bigin

      Greetings, We're excited to introduce a new enhancement to Bigin's Help section: a comprehensive Help Options panel that brings together all your support resources in a single, well-organized space. Previously, the Need Help? menu provided only a limited

    • Zoho FSM API Developer Needed

      Hi, I’m looking for a developer with experience using Zoho FSM APIs. Scope: Connect WordPress website booking form to Zoho FSM Check availability (date, time, region) Create Work Orders + Service Appointments automatically Notify both customer and scheduler

    • Revenue Management: #4 What if there are uncertainties in project or service delivery?

      Our previous post taught us how Zoho Billing makes life easy for businesses with its automated revenue recognition rule. However, certain businesses have more challenges that an automated system cannot handle, and there are certain situations where automated

    • This mobile number has been marked spam. Please contact support-as@zohocorp.com

      Bom dia, estou tentando colocar o número 11 94287-6695 e esta com erro "This mobile number has been marked spam. Please contact support-as@zohocorp.com" pode me ajudar, por favor?

    • Items Serial Tracking Issue

      We enabled Zoho Items inventory tracking then disabled it after some time now we want to enable it again When I check the missing serial number reports I see one item But I cant see any option to Add the serial numbers Where and how to add the serial

    • Composite Services and Account Tracking

      I am looking to garner support/request the ability to make composite services. A quick search in the forums brings up multiple requests for this feature. I fail to see why an item is mandatory while services are optional. I also would like to see the

    • Zoho Payroll integration with Zoho Books - unable to match multiple bank feeds to one wage payment

      For one employee's wage, I make two partial payments. Those bank feed transactions come into Zoho Books via bank integration. I make one pay-run for the month in Zoho Payroll and that comes into Zoho Books via the Zoho integration. Zoho Books doesn't let me match multiple bank feed transactions against a single wage item. Please fix urgently. I can't complete my books because of this.

    • Add Checkbox Selection & Bulk Actions to Delivery Challans Module

      Hi Zoho Team, I’ve noticed that in the Sales Orders module, there are checkboxes beside each entry that allow users to select multiple records for bulk actions such as print, email, or delete. However, in the Delivery Challans module, this option appears

    • Can't be able to check-in in laptop

      even after giving location access still i can't be able to check-in in laptop.

    • Compensation Cess on Coal ₹400 per tonne. ?????

      The compensation cess rate varies by the type of product. And the cess is calculated based on the value of the product without GST. Coal, for example, comes with a cess of ₹400 per tonne. That means that if you sell 2 tonnes of coal that have a value

    • 7 month over zoho book purchase but still not immpliments Golive

      7 month over zoho book purchase but still not immpliments Golive one problems zoho team short out then other problems come still very poor mangments and immliments team . struggling with the templates in ZOHO Books. Especially with the placement of some

    • Average Costing / Weighted Average Costing

      Hello fellow maadirs. I understand Zoho Books uses FIFO method of dealing with inventory costing, but do you guys have any plans to introduce average costing? We indians need average costing. It's part of our culture. Please. I beg thee. Thanks.

    • SMS to customers from within Bigin

      Hi All, Is there anyone else crying out for Bigin SMS capability to send an SMS to customers directly from the Bigin interface? We have inbuilt telephony already with call recordings which works well. What's lacking is the ability to send and receive

    • Next Page