Data Synchronization using Bulk Read and Notification APIs

Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    Access your files securely from anywhere



                          Zoho Developer Community




                                                  • Desk Community Learning Series


                                                  • Digest


                                                  • Functions


                                                  • Meetups


                                                  • Kbase


                                                  • Resources


                                                  • Glossary


                                                  • Desk Marketplace


                                                  • MVP Corner


                                                  • Word of the Day


                                                  • Ask the Experts





                                                            Manage your brands on social media



                                                                  Zoho TeamInbox Resources



                                                                      Zoho CRM Plus Resources

                                                                        Zoho Books Resources


                                                                          Zoho Subscriptions Resources

                                                                            Zoho Projects Resources


                                                                              Zoho Sprints Resources


                                                                                Qntrl Resources


                                                                                  Zoho Creator Resources



                                                                                      Zoho CRM Resources

                                                                                      • CRM Community Learning Series

                                                                                        CRM Community Learning Series


                                                                                      • Kaizen

                                                                                        Kaizen

                                                                                      • Functions

                                                                                        Functions

                                                                                      • Meetups

                                                                                        Meetups

                                                                                      • Kbase

                                                                                        Kbase

                                                                                      • Resources

                                                                                        Resources

                                                                                      • Digest

                                                                                        Digest

                                                                                      • CRM Marketplace

                                                                                        CRM Marketplace

                                                                                      • MVP Corner

                                                                                        MVP Corner





                                                                                          Design. Discuss. Deliver.

                                                                                          Create visually engaging stories with Zoho Show.

                                                                                          Get Started Now


                                                                                            Zoho Show Resources


                                                                                              Zoho Writer Writer

                                                                                              Get Started. Write Away!

                                                                                              Writer is a powerful online word processor, designed for collaborative work.

                                                                                                Zoho CRM コンテンツ










                                                                                                  Nederlandse Hulpbronnen


                                                                                                      ご検討中の方




                                                                                                            • Recent Topics

                                                                                                            • Issue with ticket replies via Slack: '+' symbols replacing spaces in emails

                                                                                                              Hello, support team! We're experiencing an issue when replying to tickets directly through Slack. When the reply is sent to the email, spaces are being replaced by '+' symbols. This makes the message harder to read and understand. Is there any solution
                                                                                                            • Allow 2 logos for Branding, one for Light Mode and one for Dark Mode?

                                                                                                              Our logo has a lot of black text on it. If we leave the background transparent, per recommendation of Zoho, when a user is viewing a file and turns on dark mode, our logo is not really visible and looks really weird. It would be really great if we could
                                                                                                            • Zoho Creator Populate radio field with values with all the created rows subfor

                                                                                                              I have Main Form where i have a lookup field where i get brewery names and the number of tanks as a multiline text field with a list of beer names Based Brewery selected and bbt_tanks number i create rows in the subform and now i want to populate list
                                                                                                            • Currency column showing $ symbol

                                                                                                              Hello, I'm importing data from Zoho Projects to Zoho Analytics and I was wondering why "Budget amount" column is set in dollars even if the "Currency" column = EUR: Is there a way to get the budget amount as "EUR" + nnnnnn? Thank you
                                                                                                            • Android notifications not working

                                                                                                              I've set push notifications to 'on' in ZohoMail for android settings but nothing doing. Can anyone help? I do use a VPN.
                                                                                                            • my clients are not receiving mails

                                                                                                              Hi, My clients are not receiving my mails sent . may we know the reason My dns server and imap settings are perfect
                                                                                                            • Múltiple Deals when converting a Lead

                                                                                                              Hello!!! I hope someone can help me figure out the best way to handle this scenario. I have a multi-select field named “Service” in the Leads module that captures either Service A, Service B, or both. When converting a lead, Zoho CRM currently creates
                                                                                                            • zoho mail and crm is very slow

                                                                                                              I have recently employed Zoho in our organisation. Even after taking high speed internet, mail and CRM takes many minutes to even load. Its really slow and faces lot of downtime.
                                                                                                            • How to use if_case with expressions other than equals

                                                                                                              I'm trying to define a formula column that implements logic like this case statement would: case when numfld1 is null then null when numfld2 > 0 then 100*numfld2 when numfld2 < 0 then numfld2 else 0.0 end In formula columns, the docs say you need to use
                                                                                                            • Zoho CRM's V8 APIs are here!

                                                                                                              Hello everyone!!! We hope you are all doing well. Announcing Zoho CRM's V8 APIs! Packed with powerful new features to supercharge your developer experience. Let us take a look at what's new in V8 APIs: Get Related Records Count of a Record API: Ever wondered
                                                                                                            • Create global project dashboard for all users

                                                                                                              Would like to be able to create a custom dashboard for projects with certain widgets that are default for all new projects.  right now, I have to modify each project dashboard per project per user.  This is not practical.  
                                                                                                            • What's New in Zoho Inventory | January - March 2025

                                                                                                              Hello users, We are back with exciting new enhancements in Zoho Inventory to make managing your inventory smoother than ever! Check out the latest features for the first quarter of 2025. Watch out for this space for even more updates. Email Insights for
                                                                                                            • Inline images are not shown on iPhone

                                                                                                              When I add an image inline it gets displayed on a Zoho's computer software or web browser, but not on Zoho's iPhone app - the image appears to be broken and cannot be copied neither saved. What's the problem with displaying images inline when reading
                                                                                                            • Kaizen #186 : Client Script Support for Subforms

                                                                                                              Hello everyone! Welcome back to another exciting Kaizen post on Client Script! In this edition, we’re taking a closer look at Client Script Support for Subforms with the help of the following scenario. " Zylker, a manufacturing company, uses the "Orders"
                                                                                                            • Viewing Live data

                                                                                                              Where can I see the live data that is sent from the device?
                                                                                                            • Canvas templates can now be shared with different CRM organizations

                                                                                                              ----------------------------------------Moderated on 14th February, 2023------------------------------------------- Dear all, This feature is now open for all users in all DCs. To learn more about importing and exporting canvas templates, read our help
                                                                                                            • Formatting Mailing Labels

                                                                                                              I want to use the "Print Mailing Labels" function on the drop down list, but I am not seeing a way to change the formatting on the mailing labels. At the moment, the information that appears on the mailing labels ARE NOT mailing addresses, but random information.  I would also like to change be able to change the size of the labels.  At the very least I would like to know what type of labels I can get that would be the correct size.  
                                                                                                            • CRM to Writer Mail Merge Preview not working

                                                                                                              When performing a mail merge from CRM to writer the preview function does not work. I get the following error. I am a Zoho one user on a ChromeOS. I have been successfully using mail merge from CRM to Writer about 4 years. This error seemed to coincide
                                                                                                            • Best practice : when to convert lead to Deal

                                                                                                              Hello, I'm new to Zoho and run my own business. To make sure I'm using Zoho correctly, when do I press convert, from Lead to Deal, at what stage in the conversion funnel/conversation. I want to make sure I can a) monitor status of all pending lead or
                                                                                                            • Show Call History During a Blueprint Transition in Leads Module

                                                                                                              Hi all, I have a Blueprint set up in the Leads module with a transition to Reattempt Call, which updates the lead status to Attempted Contact. I’d like to know if there’s a way to show the call history or at least a summary of how many call attempts have
                                                                                                            • How do I filter contacts by account parameters?

                                                                                                              Need to filter a contact view according to account parameter, eg account type. Without this filter users are overwhelmed with irrelevant contacts. Workaround is to create a custom 'Contact Type' field but this unbearable duplicity as the information already
                                                                                                            • How to delete more than 100 leads at a time.

                                                                                                              We are a call center and we need to upload fresh leads daily.  Is there any way to delete all leads only at once.  Currently we are deleting 100 at a time. Please anyone who can help. Thank you.
                                                                                                            • The Next Chapter for CRM for Everyone: Moving from Early Access to Phased Rollout for Customers

                                                                                                              #CRM25Q1 Hello Everyone, Until now, CRM for Everyone has been available in early access mode exclusively for users who opted to try the new version. We are now transitioning to a phased release, starting with the basic edition. We are thrilled to announce
                                                                                                            • Canvas for related lists

                                                                                                              Hey, we would like to customize our related lists. For us, it would make more sense to present the data from an assigned record vertical instead of horizontal. Can we get a related list Canvas view?
                                                                                                            • Standalone custom function not generating logs

                                                                                                              Why dont't standalone custom functions generate logs when the're called from another function? I have some functions (workflow, buttons and blueprint) that have common parts, so I put that part in a standalone function which is called from the others.
                                                                                                            • Show Zoho Books Retainer Invoice in Zoho CRM

                                                                                                              Hi Support, How can I get Retainer Invoices created in Zoho Books to show in Zoho CRM? If a sales person needs to collect an upfront deposit, they should be able to see that the retainer invoice has been created and paid. Thanks, Ashley
                                                                                                            • Feature Request - Zoho Books - Add Retainer Invoices to CRM/Books integration

                                                                                                              Hi Books Team, My feature request is to include Retainer Invoices in the finance suite integration with Zoho CRM. This way we will be able to see if retainer invoices have been issued and paid. I have also noticed that when the generate retainer invoice
                                                                                                            • ACR Phone mobile app for logging phone calls into Zoho CRM

                                                                                                              ACR Phone is an Android app recording voice calls with additional features such as blacklist, cloud upload, call log and more. Use the ACR Phone extension for Zoho CRM to generate Leads and Contacts with the voice recording attached right after your phone
                                                                                                            • New in Zoho Sign: Allow recipients to review and change associated recipients

                                                                                                              Greetings! We are happy to announce the addition of a new recipient role, Manages recipients, in our signature workflows. This role enables a recipient to review, modify, or add details for any recipients associated with them in the workflow. This ensures
                                                                                                            • How can I throw an error / terminate the flow from within a custom function?

                                                                                                              As the subject says. I would like to be able to terminate a flow from within a custom function if certain conditions are not met. I know I could hook a decision box to the output of my custom function and check return variable, but hoping there is a more
                                                                                                            • Multiple Zoho Attendees in a Customer meeting

                                                                                                              We are having constraints with having to log duplicate meetings when we have 2 Zoho users attending a customer meeting. What are the options to resolve this? You can add participants, but you cannot report on them. What can be done to avoid creating so
                                                                                                            • iOS 18 is here! Discover the enhanced Bigin app with iOS 18, iPadOS 18 and macOS Sequoia.

                                                                                                              Hello, everyone! We are excited to be back with new features and enhancements for the Bigin app. Let us take a look at the new iOS 18 and iPadOS 18 features. The following is the list of features covered in this update: Control widgets. New app icons.
                                                                                                            • Multi-Select lookup field has reached its maximum??

                                                                                                              Hi there, I want to create a multi-select lookup field in a module but I can't select the model I want the relationship to be with from the list. From the help page on this I see that you can only create a max of 2 relationships per module? Is that true?
                                                                                                            • Password change restriction

                                                                                                              Can the administrator restrict users from changing their email password?
                                                                                                            • NOW Zoho Creator still cannot bulk download Image or File Upload Field

                                                                                                              The filedownloader has been deprecated for 5 years. Until now, we still cannot have a replacement tool. How can we bulk download the file that we uploaded to Zoho Creator. Previously, it was so simple to bulk download all those files. But now failed to
                                                                                                            • CRM API Search Record for Last Name equals "."

                                                                                                              When using the CRM API to look for all contacts with a lastname = "." The API returns an Invalid Query Reponse I have tried (Last_Name:equals:.) (Last_Name:equals:%5C.) (Last_Name:equals:\.) We have a scenario where the Last Name may not be known for
                                                                                                            • MS PowerApps Custom Connector

                                                                                                              Has anyone successfully connected to the API from MS PowerApps? The OAuth2 seems impossible to use from a platform like this.
                                                                                                            • Zdk-cli

                                                                                                              As i have tried to login to zdk cli as it returns this error ✔ Success! Logged in as JAYANTHAN ✖ Error during initialization of zdk api supported only in sandbox environment
                                                                                                            • Se puede mover un Cliente de un modulo a otro personalizado

                                                                                                              Hola, tengo una duda que me gustaría resolver: Actualmente trabajo con dos módulos: Seguimiento de Venta y Cierre de Ventas . Mi objetivo es que, cuando desde el módulo de Seguimiento se marca una venta como "Venta Efectiva" , el cliente sea movido al
                                                                                                            • Its 2022, can our customers log into CRM on their mobiles? Zoho Response: Maybe Later

                                                                                                              I am a long time Zoho CRM user. I have just started using the client portal feature. On the plus side I have found it very fast and very easy (for someone used to the CRM config) to set up a subset of module views that make a potentially extremely useful
                                                                                                            • Next Page