Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

    • Recent Topics

    • Add RTL and Hebrew Support for Candidate Portal (and Other Zoho Recruit Portals)

      Dear Zoho Recruit Team, I hope you're doing well. We would like to request the ability to set the Candidate Portal to be Right-to-Left (RTL) and in Hebrew, similar to the existing functionality for the Career Site. Currently, when we set the Career Site

    • App for Mac OS X please!

      It would be awesome to have a mail app for Mac OS X that included all the cool features such as steams, calendar, tasks, contacts, etc. Most people prefer native apps, rather than running it through a web browser. I know that we can use the IMAP, CalDAV,

    • How to integrate Zoho CRM, Zoho Forms and a WIX Web Site

      Attached video demonstrates how to use Zoho Forms included in Zoho One, to design a Contact Us form to be embedded into a WIX web site and integrated into Zoho CRM.

    • CRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive

      Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage files

    • Introducing Workqueue: your all-in-one view to manage daily work

      Hello all, We’re excited to introduce a major productivity boost to your CRM experience: Workqueue, a dynamic, all-in-one workspace that brings every important sales activity, approval, and follow-up right to your fingertips. What is Workqueue? Sales

    • Default Reminder Time in New Tasks or New Event?

      Any way to change this from 1:00am? Thanks, Gary Moderation Update (February 2026): With the Calendar preferences, the default reminder time for Meetings, Appointments and All-Day Meetings can be set. Read more: Calendar preferences in Zoho CRM Regarding

    • Dynamic Field Folders in OneDrive

      Hi, With the 2 options today we have either a Dynamic Parent Folder and lots of attachments all in that one folder with only the ability to set the file name (Which is also not incremented so if I upload 5 photos to one field they are all named the same

    • Right Shift key not sending to Zoho Assist environments

      I'm unable to use Right Shift key in Zoho environments. Zoho environments are Win10. Computer I access from is Win 11. Issue started when I changed to Win 11. Have tried: - Multiple browsers - web client AND desktop client - 3rd party mapping tools to

    • Unable to Filter Retail Sales Orders & Need Paid/Unpaid Filter – Zoho Books

      Hi Zoho Team, Recently you introduced Retail – Standard and Retail – Premium templates for Sales Orders. However, in the Sales Order module we still cannot filter or segregate Retail Sales Orders separately from normal B2B sales orders. Because of this,

    • About Meetings (Events module)

      I was working on an automation to cancel appointments in zoho flow , and in our case, we're using the Meetings module (which is called Events in API terms). But while working with it, I'm wondering what information I can display in the image where the

    • PDF Annotation is here - Mark Up PDFs Your Way!

      Reviewing PDFs just got a whole lot easier. You can now annotate PDFs directly in Zoho Notebook. Highlight important sections, add text, insert images, apply watermarks, and mark up documents in detail without leaving your notes. No app switching. No

    • Ability to assign Invoice Ownership through Deluge in FSM

      Hi, As part of our process, when a service appointment is completed, we automated the creation of the invoice based on a specific business logic using Deluge. When we do that, the "Owner" of the invoice in Zoho FSM is defaulted to the SuperAdmin. This

    • How do you do ticket add ons in Backstage?

      Hi Everyone, If you wanted to have general admin tickets and allow for add ons, like camping, or car or Carbon offset. What would you do? Peace Robin

    • From Zoho CRM to Paper : Design & Print Data Directly using Canvas Print View

      Hello Everyone, We are excited to announce a new addition to your Canvas in Zoho CRM - Print View. Canvas print view helps you transform your custom CRM layouts into print-ready documents, so you can bring your digital data to the physical world with

    • validation rules doesn't work in Blueprint when it is validated using function?

      I have tried to create a validation rule in the deal module. it works if I try to create a deal manually or if I try to update the empty field inside a deal. but when I try to update the field via the blueprint mandatory field, it seems the validation

    • Pull cells from one sheet onto another

      Hello all! I have created an ingredients database where i have pricing and information and i want to pull from that database into a recipe calculator. I want it to pull based on what ingredient I choose. The ingredients database has an idea and i want

    • Emails Failing with “Relaying Issues – Mail Sending Blocked” in ZeptoMail

      Hello ZeptoMail Support Team, We are facing an email delivery issue in our ZeptoMail account where emails are failing with the status “Process failed” and the reason “Relaying issues – Mail sending blocked.” Issue Details Agent Name: mail_agent_iwwa From

    • How does Zoho handle support operations for our EU users?

      Dear Zoho users, We have some news and reminders about how we handle support operations for Zoho users in the EU Data Center. This is part of our broader commitment to ensuring GDPR compliance and providing secure data handling for all users. We'd like

    • Sender Email Configuration Error.

      Hello Team, Hope you are all doing well. We are in the process of creating the Zoho FSM environment in the UAE. When we try to add the sender email address “techsupportuae@stryker.com”, we receive the error message: “Error occurred while sending mail

    • Layout Adjustment.

      Hello Latha, Hope you are doing well. I wanted to check with you about adjusting the layout columns in the Service Report from the backend, as currently there is no capability for us to customize the layout from our end. Below is the output that I am

    • Let us view and export the full price books data from CRM

      I quote out of CRM, some of my clients have specialised pricing for specific products - therefore we use Price Books to manage these special prices. I can only see the breakdown of the products listed in the price book and the specialised pricing for

    • Anyone in Australia using Zoho Books AND has their account with NAB?

      Hi I have an account with both NAB and Suncorp. Suncorp transaction come in the next day however NAB transactions take 4-5 business days to appear. eg: A deposit made today in my Suncorp will be imported into Zoho tomorrow. A deposit made today to the NAB account will be imported maybe Saturday (Friday overnight). I have contacted both Zoho and NAB but noone seems to know why. I was just wondering if anyone else in Australia uses NAB and has this issue (or doesn't) maybe we could compare notes and

    • Zoho Often Lag

      Desk Team, We've noticed that Desk, Creator & other apps seems to lag more these days. We often refresh multiple times.

    • Offer already made- but I withdrew it

      I made an offer letter, but made a mistake on it. I withdrew the offer but now I can't recreate the correct offer. Zoho keeps saying that "A same offer has already been made". I look in the "offers" and there are NO offers (this is the first time I've

    • Importing into the 'file upload' field

      Can you import attachments into the file upload field. I would expect it to work the same way as attachments do, But can't seem to get it to work. Moderation Update: The option to import into File Upload and Image Upload fields has been supported. Read

    • Layout Rules Don't Apply To Blueprints

      Hi Zoho the conditional layout rules for fields and making fields required don't work well with with Blueprints if those same fields are called DURING a Blueprint. Example. I have field A that is used in layout rule. If value of field A is "1" it is supposed to show and make required field B. If the value to field A is "2" it is supposed to show and make required field C. Now I have a Blueprint that says when last stage moves to "Closed," during the transition, the agent must fill out field A. Now

    • Why is the ability Customize Calls module so limited?

      Why can't I add additional sections? why can't I add other field types than the very limited subset that zoho allows? Why can I only add fields to the outbound/inbound call sections and not to the Call Information section?

    • Zoho Desk blank screen

      opened a ticket from my email, zoho desk comes up blank, nothing loads. our receptionist also gets the same thing under her login on her computer. our sales rep also gets same thing on zoho desk at his home on a different computer. I tried clearing cache/history/cookies,

    • Optimum CRM setup for new B2B business

      Can some advise the most common way to setup Zoho CRM to handle sales for a B2B company? Specifically in how to handle inbound/outbound emails. I have spent hours researching online and can't seem to find an accepted approach, or even a tutorial. I have

    • filter on sheets mobile (iOS, iPadOS) does not work

      I have this issue on every spreadsheet, whether imported or created natively in Zoho Sheets. I can use on desktop without issue: desktop screenshot on iOS the filter dropdowns are not there and there is no way to add them mobile iPadOS looks the same

    • 【Zoho CRM】商談タブへのデータインポート

      Zoho使用前にエクセルで管理していた商談情報を、Zoho一括管理のため、商談タブにインポートしたいのですが、お客さまの氏名だけが紐づけられませんでした。 「Zoho CRMにインポートする項目を関連付ける」のところが画像のようになっています。 （弊社では、「姓」を「★個人データ名」という項目名に変更し、フルネームを入れて使用しています。） どのようにしたら氏名をインポートできるかご存じの方がいらっしゃいましたら、ご教示いただきたく、よろしくお願いいたします。 （投稿先が間違っていましたらご指

    • Add Custom Reports To Dashboard or Home Tab

      Hi there, I think it would be great to be able to add our custom reports to the Home Tab or Dashboards. Thanks! Chad

    • Canva Integration

      Hello! As many marketing departments are streamlining their teams, many have begun utilizing Canva for all design mockups and approvals prior to its integration into Marketing automation software. While Zoho Social has this integration already accomplished,

    • Can't edit Segments

      Happening with 2 different Zoho One environments, in different browsers. Please fix.

    • Zoho Sheet for Desktop

      Does Zoho plans to develop a Desktop version of Sheet that installs on the computer like was done with Writer?

    • Transfer ownership of files and folders in My Folders

      People work together as a team to achieve organizational goals and objectives. In an organization, there may be situations when someone leaves unexpectedly or is no longer available. This can put their team in a difficult position, especially if there

    • Consumer Financing

      Does Zoho currently have a payment gateway (such as Stripe, Square, etc) which offers financing for customers? So, let's say the estimate we give the customer is greater than what they can afford at the time, but we can sell the service now, letting them

    • Subform edits don't appear in parent record timeline?

      Is it possible to have subform edits (like add row/delete row) appear in the Timeline for parent records? A user can edit a record, only edit the subform, and it doesn't appear in the timeline. Is there a workaround or way that we can show when a user

    • Zoho CRM - Option to create Follow-Up Task

      When completing a Zoho CRM Task, it would be very helpful if there was an option to "Complete and Create Follow-Up Task" in the pop-up which appears. It could clone the task you are closing and then show it on the screen in edit mode, all the user would

    • Zoho CRM for Everyone's NextGen UI Gets an Upgrade

      Hello Everyone We've made improvements to Zoho CRM for Everyone's Nextgen UI. These changes are the result of valuable feedback from you where we’ve focused on improving usability, providing wider screen space, and making navigation smoother so everything

    • Next Page