Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • OS X Notebook quits immediately upon launch

      NoteBook for OS X (Sequoia, but also under Sonoma) always quits immediately upon launch (so I cannot use the "Attach user log" option). I've restarted my MacBook but the problem persists. If it helps, attached is a diagnostic report from Library>Log

    • Can External users upload files or images to WorkDrive?

      I want to know if it is possible for someone externally through a link and PW be able to upload files and images onto WorkDrive?

    • Multi-line fields character limits

      Is there a way to set the character limit higher on multi-line fields so that we are not losing information pasted into the field? When the text is entered or pasted, there is no error to say that the text is too large. After saving and going back to view most of the text is gone.   Also, when viewing the resume, the text is not wrapped in the multi line fields and can t be read without scrolling across the page.

    • Taz bot not working — What should I do to resolve this issue?

      I am experiencing issues with the Taz bot in Zoho Cliq—not receiving responses or it does not seem to work as expected. Could you please explain why the Taz bot might not be functioning and what steps I should take to resolve this issue? Thank you!

    • OneNote Migration

      I am trying to migrate two notebooks from OneNote. For five days now I have had no notification that migration has completed and the migration page show 50% complete - one notebook completed one not finished. It just stays like this. I am unable to cancel

    • Set various time slot reminders for task and event

      Hi I would like known if there is possibility to set various time reminders for a task or event like we have the possibility to do in google calendar or google task. For example I'am creating an event or task and I want to be alerted before 30min and

    • Zoho Calendar not functioning---cannot edit, add attendees, etc.

      Hello, My calendar is acting strange. I'm using Chrome as my browser and pop ups are not blocked (the calendar worked fine yesterday). When creating an event, I'm not able to "edit" the event and add attendees, etc. The link is non-responsive. I cannot

    • Spilt Axis for stacked column and line graph

      Each month around this time I prepare a business review deck. One of the biggest annoyances I have with Zoho, also happens to be something that most other platforms have provided for a long time now, and that is being able to create a chart with stacked

    • Pin a note on top

      Dear Zoho Notebook Team, Please highly consider adding ability to pin a note on top and arrange pinned notes. I have a lot of notes that I want to pin important ones on top and this feature is missing badly. Thank you.

    • Windows 11 app ver 2.2.8

      I have been trialing Zoho Notes syncing between my PC and iphone. No problems with iphone but the Windows 64 bit app ver 2.2.8 was very buggy and would lock up completely, needing a Control Alt Delete etc to close it down. It seemed to be connected to

    • Deluge Script: Onboarding → Access Form

      Hello everyone, Edit: Sorry, I think I put this topic in the wrong section. I’ve recently run into an issue that’s been giving me a bit of a headache for the past few weeks. Context: This is my first time using Deluge to create a script, but it’s not

    • Windows Device Authentication

      We have recently started using ManageEngine, and my boss saw a device management feature in the Zoho One directory. We thought it would either help give us more intergration into Zoho one through bringing ManageEngine services through Zoho one. Or, it

    • How can I change spell check language?

      I cannot find the way to change spell check language. My "display language" is English, and I want to have the same one for Spell Check, but it is Russian!!! How can I change it? In one of your explanations you mentioned that I have to choose it from

    • Mail is so slow - doesn't even work!

      Mail has been getting slower and slower - and today it's not even pulling up emails in either Inbox or Unread. This is beyond frustrating since email is a big part of business. Sent a request through the useless help portal - no response. Called the useless

    • User marked as SPAMMER. Mail Fetch has also been disabled for any active POP accounts.

      I am the administrator for joelles.com One of our accounts has been blocked saying this: User marked as SPAMMER. Mail Fetch has also been disabled for any active POP accounts. I cannot change the disabled account in the control panel as it says that it

    • My domain did not activate

      Hi, my domain (apsaindustrial.com.ar) did not activate, and the phone verification message never arrived. Please would you solve this problem? Thanks.

    • ME SALE ESTE ERROR: No fue posible enviar el mensaje;Motivo:554 5.1.8 Email Outgoing Blocked

      Ayuda!! Me sale este error al intentar enviar mensajes desde mi correo electronico de Zoho! Tampoco recibo correos pues cuando me envia rebotan. Ayuda, Me urge enviar unos correo importantes!! Quedo atenta MAGDA HERNANDEZ +5731120888408

    • Is there a way to sync Tags between CRM and Campaigns/Marketing Hub?

      I wonder if there is a way to synch the tags between CRM and Marketing-Hub / Campaigns?

    • how to see if a specific contact opened an email in zoho campaign?

      how to see if a specific contact opened an email in zoho campaign?

    • Revenue Management: #8 Revenue Recognition in Educational & Training Institutions

      Educational Institutions and training centres typically collect course fees at the time of enrolment, sometimes for a one-day workshop and sometimes for a year-long certification course. You might also charge separately for course materials or evaluation.

    • How to Customize Task Creation to Send a Custom Alert Using JavaScript in Zoho CRM?

      Hello Zoho CRM Community, I’m looking to customize Zoho CRM to send a custom alert whenever a task is created. I understand that Zoho CRM supports client scripts using JavaScript, and I would like to leverage this feature to implement the alert functionality.

    • Using files from Zoho CRM in Gemini/ChatGPT/Claude

      Hi all, I’ve got subscriptions to Gemini and a few other AI tools which I use for tasks like data enrichment, email composition, etc. In our workflow, we often receive various documents from clients — such as process workflows, BRDs/requirement documents

    • Enhancements to the formula field in Zoho CRM: Auto-refresh formulas with the "Now" function, stop formula executions based on criteria, and include formulas within formulas

      Dear Customers, We hope you're well! By their nature, modern businesses rely every day on computations, whether it's to calculate the price of a product, assess ROI, evaluate the lifetime value of a customer, or even determine the age of a record. With

    • Maximum file limit in zoho people LMS

      Dear Team, I am having approximately 4.9 GB of material, including PPTs and videos for uploading in zoho people LMS course. May I know what is the maximum limit limit for the course files Thanking you, With regards, Logeswar V Executive _ Operations

    • Dynamically prefill ticket fields

      Hello, I am using Zoho Desk to collect tickets of our clients about orders they placed on our website. I would like to be able to prefill two tickets fields dynamically, in this case a readonly field for the order id, and a hidden field for the seller

    • Optimize your Knowledge Base for better visibility by allowing search engine crawling and indexing

      All you need to know about no-follow and no-index in KB. What are article crawlability and indexability? Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it

    • Has anyone created a public ASAP Guide that I can check out?

      I am thinking of adding an ASAP guide to my web application, but I have noticed that the ASAP widget itself can be really slow to load sometimes. Has anyone created a public ASAP Guide that I can check to see how performant it is? I don't want to spend

    • Zoho Desk Partners with Microsoft's M365 Copilot for seamless customer service experiences

      Hello Zoho Desk users, We are happy to announce that Zoho Desk has partnered with Microsoft's M365 to empower customer service teams with enhanced capabilities and seamless experiences for agents. Microsoft announced their partnership during their keynote

    • What’s New in Zoho Analytics – September 2025

      Hello Users!! In this month’s update, we’re raising the bar across multiple touchpoints, from how you bring in data, plan and track projects to how you design and brand your dashboards. We’ve added the all-new Gantt chart for project visualization, expanded

    • Zoho MCP has no tools for Creator or 3rd Party Apps?

      I don't see a Zoho MCP community forum so putting this here. Two big problems I see: 1) Although Zoho advertises "over 950 3rd party apps" as available through their MCP, when I go to "Add Tools" there are ZERO 3rd party apps available to choose from.

    • Zoho Forms - Zoho Drive connection - Shared Drives not supported

      Hello i am stuck with Google Drive Connection There is no supported shared drives Connection is not support shared drives boolean Query Parameters - supportsAllDrives=true&supportsTeamDrives=true to activate fetch files from the shared drives. Ahat need

    • Can't create package until Bill created?

      I can't understand why we cannot create a package until a Bill is created? We are having to created draft Bills to create a package when the item is received, but we may not have received a Bill from the supplier. Also, Bill # is required, but we normally

    • Whats the Time out Limit for API Calls from Deluge?

      Hi Creator Devs, We are making API calls to third party server via Deluge. Getting this error message: Error at line : 24, The task has been terminated since the API call is taking too long to respond. Please try again after sometime. Whats the default

    • How to get the authorization code for domain transfer from Zoho?

      I have transferred my domain from zoho to godaddy, and that needs a authorization code. I haven't got any mail containing that. How to get the authorization code from domain transfer? Thanks, Arun

    • I have error AS101 when I try to add paypal@mydomain.com to Zoho

      Please help me with this. I tried to call the help line 4 times but don't get any response.

    • Tip #3: How to change your booking page language

      Displaying your booking page in your target audience's language can greatly increase customer satisfaction. By speaking their language, you will help customers feel more comfortable scheduling with you and create a stronger connection with them. Let's

    • How can I optimize a Zoho Site page for SEO when embedding external menu or restaurant links?

      Hi everyone, I’m experimenting with building small content hubs on Zoho Sites and want to make sure I’m doing it in an SEO-friendly way. For example, I tried creating a page that highlights restaurant menu items and linked out to a resource like this:

    • Diff signature for compose new email and replies

      Hi,   How do i have different signature for replies and new emails. its inconvenient to have one large signature for replies. Usually on Outlook we have the option to keep separate signatures for new emails and for replies.

    • Zoho Website Site Speed Up & Setting

      We are experiencing slow loading speeds on our Zoho website and would like assistance in optimizing its performance. Kindly review the site and suggest or implement necessary improvements to enhance speed, especially related to: > Caching mechanisms >

    • Clickjacking: Zoho Vault's Response

      Issue: Password manager browser extensions are found to be vulnerable to clickjacking security vulnerabilities that could allow attackers to steal account credentials, TFA codes, and card details under certain conditions. Reported by: Marek Toth, Independent

    • Next Page