Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Dear Zoho CEO: Business Growth is about how you prioritise!

      All of us in business know that when you get your priorities right, your business grows. Zoho CRM and Zoho Books are excellent products, but sadly, Zoho Inventory continues to lag behind. Just this morning, I received yet another one-sided email about

    • Payroll In Canada

      Hi, When can we expect to have payroll in Canada with books 

    • Please review and re-enable outgoing emails for my domain

      Hello Zoho Support, I have recently purchased a new domain and set up email hosting with Zoho. However, my account shows "Outgoing Email Blocked". I am a genuine user and not sending bulk/spam emails. Please review and re-enable outgoing emails for my

    • Payroll without tax integrations (i.e. payroll for international)

      It seems as though Zoho waits to develop integrations with local tax authorities before offering Zoho Payroll to Zoho customers in a country. Please reconsider this approach. We are happy Zoho Books customers, but unhappy that we have to run payroll in

    • goingout e mail block

      info@ozanrade.com.tr

    • Incoming mails blocked

      Zoho User ID : 60005368884 My mail Id is marketing#axisformingtechnology.com .I am getting following message "Your Incoming has been blocked and the emails will not be fetched in your Zoho account and POP Accounts. Click here to get unblocked." Please

    • Assistance Needed: Ticket Status Not Updating and Sorting by Last Customer Reply in Zoho Desk

      Hello, I’m facing two issues in Zoho Desk that I’d like your guidance on: Ticket Status Not Updating: When a customer replies to a ticket, the status does not change to Reopened. Instead, it remains in Waiting on Customer, even after the customer’s response

    • Configuring Email Notifications with Tautulli for Plex

      Hi I'm new to Zoho. I am from Canada and I have a I use a web based application called Tautulli for Plex that monitors my Plex media server. It also sends a newsletter to my followers. To set this up they require a "From" email address., a smtp server

    • Is there a way to automatically add Secondary Contacts (CCs) when creating a new ticket for specific customers?

      Some of our customers want multiple contacts to receive all notifications from our support team. Is there a way to automatically add secondary contacts to a ticket when our support team opens a new ticket and associates it with an account? This would

    • How to Set Up Zoho Mail Without Cloudflare on My Website

      I'm having some trouble with Cloudflare here in Pakistan. I want to configure Zoho Mail for my domain, but I'm not sure how to set it up without going through Cloudflare. My website, https://getcrunchyrollapk.com/ , is currently using CF, but I'd like

    • Spam is Being Forwarded

      I am filtering a certain sender directly to the Trash folder. Those messages are still being forwarded. Is this supposed to happen?

    • IMAP Block

      My two accounts have been blocked and I am not able to unblocked them myself. Please respond to email, I am traveling and this is urgent.

    • "DKIM not configured"

      Hello. I have been attempting get the DKIM verified but Toolkit keeps sending the message that it is not configured, but both Namecheap and Zoho show it as configured properly. What am I missing?

    • Zoho mail with custom domain not receiving email

      i registered zoho mail with my own domain. I can login and access the mail app. I tried to send email from an outlook email account and an icloud email account. Both emails were not received. My plan is free. I also tried to send email from this zoho

    • Set connection link name from variable in invokeurl

      Hi, guys. How to set in parameter "connection" a variable, instead of a string. connectionLinkName = manager.get('connectionLinkName').toString(); response = invokeurl [ url :"https://www.googleapis.com/calendar/v3/freeBusy" type :POST parameters:requestParams.toString()

    • Waterfall Chart

      Hello, I would like to create a waterfall chart on Zoho analytics which shows the movement in changes of budget throughout a few months, on a weekly basis. Should look something like the picture below. Does anyone know how to?

    • Help Center and SEO: Any Benefit to My Domain-Mapped Website Ranking?

      First of, I love the Help Center which I've just decided to integrate into my website to replace its old-fashioned FAQs. So much more to achieve there now! Lots of new benefits to the site visitors and to me in terms of organizing and delivering all the

    • Issue with Importing Notes against Leads

      Hi, I am attempting to import some Notes into Zoho CRM which need to be assigned to various Leads. I have created a csv file which I am testing that contains just one record. This csv file contains the following columns: Note Id Parent Id       (I have

    • Trigger a Workflow Function if an Attachment (Related List) has been added

      Hello, I have a Case Module with a related list which is Attachment. I want to trigger a workflow if I added an attachment. I've seen some topics about this in zoho community that was posted few months ago and based on the answers, there is no trigger

    • Option to Hide Project Overview for Client Profiles

      In Zoho Projects, the Project Overview section is currently visible to client profiles by default. While user profiles already have the option to restrict or hide access to the project overview, the same flexibility isn’t available for client profiles.

    • Creator Add Records through API - Workflows not triggered ?

      Hi Everyone, I am trying to add records to my Creator application through a third party app that I am developing. Currently, I am testing this through Postman. The records I need to add have a lot of workflows to run to populate dropdowns, fields, use

    • Important Update: Changes to Google Translate Support in Zoho SalesIQ

      We’re updating our default chat translation options across all Zoho SalesIQ data centres to offer a more secure, in-house solution for your translation needs. What’s changing? We will be offering Zoho Translate (our in-house tool) as the default translation

    • Zoho CRM Community Digest - July P1 | 2025

      Hey everyone, The start of July 2025 marked a special milestone: 200 posts in our Kaizen series! For those new here, Kaizen is a go-to series for Zoho CRM developers, where we regularly share best practices, tips, and expert insights to help you build

    • What’s New in Zoho Expense (April – July 2025)

      Hello users, We're happy to bring you the latest updates and enhancements we've made to Zoho Expense over the past three months, which include introducing the Trip Expense Summary report in Analytics, extending Chatbot support to more editions, rolling

    • Weekly Tips: Insert frequently used phrases in a jiffy using Hotkeys

      You often find yourself using similar phrases in an email —like confirming appointments or providing standard information. Constantly switching between the mouse and keyboard interrupts your flow and slows you down.Instead of typing the same phrases over

    • Undo Command in Notebook

      It would be handy to have an Undo in the row of icons at the bottom of the notes.

    • Improved UI for a Seamless User Experience - Calls, Tasks, and Meetings

      Hello all, We are making UI unification across CRM so that the UI experience is seamless across the product. As part of that effort, we have made changes to the details page of activity-based module records—Meetings, Calls, and Tasks. Let's look at these

    • New Customization options in the module builder: Quick Create and Detail view

      Hello everyone, We have introduced two new components to the module builder: Quick create and Detail view. The Quick Create Component It is a mini form used to create a record and associate it to the parent record from a lookup field. For example, if you have a Deals lookup in the Contacts module, then you can associate existing deals or create a deal and associate it with the contact. You can customize this Quick Create form by adding standard as well as custom fields. There is no limit to the number

    • Narrative 9: GC—Meaningful conversations that benefit your customers

      Behind the scenes of a successful ticketing system - BTS Series Narrative 9: GC—Meaningful conversations that benefit your customers Customers often seek immediate solutions, which is why self-service options are essential. Guided Conversations provides

    • FSM App Oddity

      We recently rolled out FSM to our technicians, and only one technician is having an issue. He's using an iPhone with iOS 18.6 installed. When he goes to service appointments using the calendar icon at the bottom of the app, he gets a list view only. Typically,

    • Cliq Not Working !

      Zoho Cliq has been experiencing connectivity issues since this morning. The app is unable to establish a connection with the server.

    • Injecting CSS into ZML pages

      Is there a way to inject CSS into ZML pages? Use case: 1. Dashboard displays 'Recent Activities' card displaying a list of newly added records 2. Each item in list links to the record onClick 3. When a user points the cursor over an item in the list,

    • Power of Automation :: Automated Weekly Notifications for Unapproved Timesheets

      Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate

    • Extracting Data from Sitelink APIs

      Hi All, I'm working to pull data on Sitelink using API calls but i can't seem to extract it. If you click expand, you will see it has a lot of data in it. I tried extracting from the map but it's not giving me anything. Is there a workaround on this or

    • Export a list of fields for all modules in a spreadsheet with specific field data

      Many of my clients are using spreadsheets to create lists of fields for all modules when starting a new implementation or when updating an existing setup. This is a useful process but also a very time consuming one. It would be good a list of fields could

    • [Important announcement] Zoho Writer will mandate DKIM configuration for automation users

      Hi all, Effective Dec. 31, 2024, configuring DKIM for From addresses will be mandatory to send emails via Zoho Writer. DKIM configuration allows recipient email servers to identify your emails as valid and not spam. Emails sent from domains without DKIM

    • 【Zoho Backstage】2025年7月のアップデート紹介

      本投稿は、本社のZoho Desk コミュニティに投稿された以下の記事を翻訳し、一部抜粋したものです。 What's New - July 2025 | Zoho Backstage ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 本投稿では、Zoho Backstage の直近のアップデートを3点抜粋して、ご紹介します。 目次 1.参加可否の確認：RSVP 2.証明書作成ツール：Certificate Builder 3.登録の承認 ：Registration Approval 1.参加可否の確認：RSVP

    • Option to select location?

      As a business coach, I meet with clients at various public locations. I have two or three pre-determined locations that I meet at. I would like the client to choose the location when booking an appointment. Is there a way to do that with a single service, or is the best way to accomplish this by creating one service for each location offered?

    • Can we have Backorder Management ?

      Can we have Backorder Management ?

    • Display a field of an Account on the Ticket view

      Hi there, We would like to display of the Account of the user submitting a ticket on the ticket view. See for example: How can this be achieved? It doesn't really matter where it will be placed as long as it is shown there. Thanks, Snir

    • Next Page