Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • Reverse payment on accidentally closed invoice.

      An invoice was closed accidentally with the full payment added. However, only partial payment was paid. How can I reopen the invoice and reverse this to update it to show partial payment?

    • New integration: Track booking page appointments in Google Analytics 4

      Hello all, Greetings from the Zoho Bookings team! We’re excited to introduce our new Google Analytics 4 (GA4) integration designed to help you track booking activity, understand customer behavior, and measure marketing performance, all in one place. What

    • How to list emails in a folder, e.g. Inbox, on multiple pages when using Zoho mail webpage?

      Something as shown in the figure. There are totally 50 emails in Sent folder. If "Mail per page" equals 20, then the Sent folder is split into 3 pages. When I wander through Sent folder, I can just select a specific page to jump to. BTW, it seems that

    • Zoho Calendar soft bounce on @hotmail.com and @yahoo.com email addresses

      Hello, our Zoho calendar recently does not send the calendar invites to emails with hotmail and yahoo domains and comes back with a "soft bounce". other domains like Gmail works fine. Also sending "email" to the same emails to the above domains work well

    • ERROR CODE :512 - 5.4.4 DNS error:NXDOMAIN.

      Suddenly we cant send mail, we are getting this error for all outbound mail to multiple domains.

    • Can Zoho Flows repeat Actions more than once?

      I'm attempting to make an intentional Zoho Flow loop using the below layout. However, when "WithinLimit" condition is met, the program fails to execute the action "Get & Add Request Co..." again. Is this by design? Is Zoho Flows unable to repeat actions

    • Unveiling Cadences: Redefining CRM interactions with automated sequential follow-ups

      Last modified on 01/04/2024: Cadences is now available for all Zoho CRM users in all data centres (DCs). Note that it was previously an early access feature, available only upon request, and was also known as Cadences Studio. As of April 1, 2024, it's

    • customer data security

      We are exploring ways to enhance our within Zoho CRM. Our Goal: We want to fully integrate RingCentral with Zoho CRM to enable click-to-call functionality for our sales team. However, to comply with data privacy regulations and protect customer contact

    • Zoho Cliq not working on airplanes

      Hi, My team and I have been having this constant issue of cliq not working when connected to an airplane's wifi. Is there a reason for this? We have tried on different Airlines and it doesn't work on any of them. We need assistance here since we are constantly

    • Sync CRM inventory data with Zoho Books

      I just switched everything over to ZoHo books, but I am trying to find out why the CRM Estimates, Invoices, and Sales Orders created in ZoHo CRM are not then duplicated in ZoHo Books? I had Quickbooks before, and had to do everything twice, I thought

    • mask Customer phone number and agents cant see customer phone number

      Is there any way we can integrate Zoom Phone with Zoho CRM while ensuring that customer phone numbers remain masked? We need a solution where agents can make outbound calls but cannot see customer phone numbers. Please let us know if there is any solution

    • Email Reminders on Shared Calendars

      How do we turn off the setting that emails reminders to everyone who has accepted or declined a calendar invite? If 8 of us have been invited to the same meeting, we receive 8 notifications for every step of the process, from invitation to decision.

    • WebDAV / FTP / SFTP protocols for syncing

      I believe the Zoho for Desktop app is built using a proprietary protocol. For the growing number of people using services such as odrive to sync multiple accounts from various providers (Google, Dropbox, Box, OneDrive, etc.) it would be really helpful if you implemented standard protocols such as WebDAV / FTP / SFTP so that alternative inc clients can be used.

    • What's New in Zoho Inventory | Q2 2025

      Hello Customers, The second quarter have been exciting months for Zoho Inventory! We’ve introduced impactful new features and enhancements to help you manage inventory operations with even greater precision and control. While we have many more exciting

    • How to refresh a ticket view ?

      I am doing a widget where I send a rest api call to make a new draft to the ticket I am viewing. The issue is sometimes it refresh a ticket view and I can see inserted draft right away, but sometimes I do not see it even if it is inserted correctly and

    • Ugh! - Text Box (Single Line) Not Enough - Text Box (Multi-line) Unavailable in PDF!

      I provide services, I do not sell items. In each estimate I send I provide a customized job description. A two or three sentence summary of the job to be performed. I need to be able to include this job description on each estimate I send as it's a critical

    • Merge Items

      Is there a work around for merging items? We currently have three names for one item, all have had a transaction associated so there is no deleting (just deactivating, which doesn't really help. It still appears so people are continuing to use it). I also can't assign inventory tracking to items used in past transactions, which I don't understand, this is an important feature moving forward.. It would be nice to merge into one item and be able to track inventory. Let me know if this is possible.

    • Supervisor Rules - Zoho Desk

      Hi, I have set up a Supervisor Rule in Zoho Desk to send an email alert when a ticket has been on hold for 48 hours. Is there a way to change it so that the alert only sends once and not on an hourly basis? Thank you Laura

    • ResponseCode 421, 4.7.0 [TSS04] Messages from 136.143.188.51 temporarily deferred due to user complaints

      Had email bounce. Let me know if you can fix this. Thanks. Michael

    • Zoho Canvas - Custom templates for related lists

      Hi, I see that the example pages load always one of our related lists in a custom template, but I dont know how to work with that:  1) How can i make my own custom templates for related lists?  2) Where and how can i check out existing custom templates?

    • Automation #15: Automatically Adding Static Secondary Contacts

      Rockel is a top-tier client of Zylker traders. Marcus handles communications with Rockel and would like to add Terence, the CTO of Zylker traders to the email conversations. In this case, the emails coming from user address rockel.com should have Terence

    • New Zoho triggers Google Dangerous flag due toabnormal charcters

      Just signed up and doing my first email test. I sent it to my google email account but it got flagged as Dangerous" due abnormal characters. My DNS setup looks ok. Page snips attached Help Please Thanks, Rick DC PowerWorld

    • Is there a API to fetch tasks in a Board/Section

      I am writing a scheduled function that retrieves all the tasks and send an reminder on cliq. I cannot seem to find a API to fetch tasks (by user / board / section) What are the way to fetch tasks?

    • Having trouble fetching contents of Zoho Connect Feeds using the API, requesting alternative API documentation.

      I'm trying to retrieve feed/post data from Zoho Connect using the API but facing challenges with the current documentation. What I've tried: OAuth authentication is working correctly (getting 200 OK responses) Tested multiple endpoints: /pulse/nativeapi/v2/feeds,

    • Adding an Account Name to Tasks/Reminders

      Does anyone know how to add the related account name to a task?  When we look at the list of activities and when the reminders pop up, there is no way of quickly seeing who the account is. 

    • Triggering Zoho Flow on Workdrive File Label

      Right now Im trying to have a zoho flow trigger on the labeling/classification of a file in a folder. Looking at the trigger options they arent great for something like this. File event occurred is probably the most applicable, but the events it has arent

    • SendMail to multiple recipients

      Hi, I'm trying to send an email to a list of recipients.  Right now the "to" field is directed to a string variable. (List variables won't work here). In the string variable, how can I make it work? trying "user@app.com;user2@app.com" or "user@app.com; user2@app.com" just failed to send the emails. Ravid

    • Populate drop down field from another form's subform

      Hello, I found how to do that, but not in case of a subform. I have a Product form that has a subform for unit and prices. A product might have more than one unit. For example, the product "Brocoli" can be sold in unit at 3$ or in box of 10 at 25 $. Both

    • Usar o Inventory ou módulo customizado no CRM para Gestão de Estoque ?

      Minha maior dor hoje em usar o zoho é a gestão do meu estoque. Sou uma empresa de varejo e essa gestão é fundamental pra mim. Obviamente preciso que esse estoque seja visível no CRM, Inicialmente fiz através de módulos personalizados no próprio Zoho CRM,

    • Signup forms behaviour : Same email & multiple submissions

      My use case is that I have a signup form (FormA) that I use in several places on my website, with a hidden field so I can see where the contact has been made from. I also have a couple of other signup forms (FormB and FormC) that slight differences. All

    • getting error in project users api

      Hello, I'm getting a "Given URL is wrong" error when trying to use the Zoho Projects V3 API endpoint for adding users to a project. The URL I'm using is https://projectsapi.zoho.com/api/v3/portal/{portalid}/projects/{projectid}/projectusers/ and it's

    • Change total display format in weekly time logs

      Hi! Would it be possible to display the total of the value entered in the weekly time log in the same format that the user input? This could be an option in the general settings -> display daily timesheet total in XX.XX format or XX:XX.

    • Different Company Name for billing & shipping address

      We are using Zoho Books & Inventory for our Logistics and started to realize soon, that Zoho is not offering a dedicated field for a shipping address company name .. when we are creating carrier shipping labels, the Billing Address company name gets always

    • How to display historical ticket information of the total time spent in each status

      Hi All, Hoping someone can help me, as I am new to Zoho Analytics, and I am a little stuck. I am looking to create a bar chart that looks back over tickets raised in the previous month and displays how much time was spent in each status (With Customer,

    • Zoho Projects iOS app update: Global Web Tabs support

      Hello everyone! In the latest version(v3.10.10) of the Zoho Projects app update, we have brought in support for Global Web Tabs. You can now access the web tabs across all the projects from the Home module of the app. Please update the app to the latest

    • Zoho Community Weekend Maintenance: 13–15 Sep 2025

      Hi everyone, We wanted to give you a heads-up that Zoho Community will undergo scheduled maintenance this weekend. During this period, some community features will be temporarily unavailable, while others will be in read-only mode. Maintenance Window:

    • Agent Performance Report

      From data to decisions: A deep dive into ticketing system reports An agent performance report in a ticketing system provides a comprehensive view of how support agents manage customer tickets. It measures efficiency and quality by tracking key performance

    • Show both Vendor and Customers in contact statement

      Dear Sir, some companies like us working with companies as Vendor and Customers too !!! it mean we send invoice and also receive bill from them , so we need our all amount in one place , but in contact statement , is separate it as Vendor and Customer, 

    • URL validation

      We use an internal intranet site which has a short DNS name which Zoho CRM will not accept.   When attempting to update the field it says "Please enter a valid URL". The URL I am trying to set is http://intranet/pm/ Our intranet is not currently setup with a full DNS name and given the amount of links using the shortname probably isn't a feasible change for us.

    • Pourquoi dans zohobooks version gratuite on ne peut ajouter notre stock d'ouverture??

      Pourquoi dans zohobooks version gratuite on ne peut ajouter notre stock d'ouverture ??

    • Next Page