Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • Possible to generate/download Quote PDF using REST API?

      See title. Is there any way after a quote has been created to export to a PDF using a specified template and then download it? Seems like something that should be doable. Is this not supported in the API v2.0?

    • Creating an invoice to be paid in two installments?

      Hi there, I own a small Photographic Services business and have not been able to find a way to fit my billing system into Zoho, or any other Accounting software. The way my payments work is: 1. Customer pays 50% of total price of service to secure their

    • Bug in allowing the user to buy out of stock items

      Hi i want to allow the user to buy out of stock items, according to the commerce documentation if i disable Restrict "Out of stock" purchases it will, but it doesnt work, so i want to know if it had any relation with zoho inventory, and if theres any

    • Replace Lookup fields ID value with their actual name and adding inormation from subforms

      Hi everyone,  I wanted to see if someone smarter than me has managed to find any solutions to two problems we have. I will explain both below.  To start we are syncing data from Zoho CRM to Zoho Analytics and I will use the Sales Order module when giving

    • Can a Zoho Sites page be embedded into another website (outside Zoho)

      Hi All, We have a request from a client - they'd like to take one of our information pages created in Zoho Sites and embed it into their own website? I was told through an email with Zoho that this was possible >>Thank you for your patience regarding

    • Bug in allowing the user to buy out of stock items

      Hi i want to allow the user to buy out of stock items, according to the commerce documentation if i disable Restrict "Out of stock" purchases it will, but it doesnt work, so i want to know if it had any relation with zoho inventory, and if theres any

    • Transition Criteria Appearing on Blueprint Transitions

      On Monday, Sept. 8th, the Transition criteria started appearing on our Blueprints when users hover over a Transition button. See image. We contacted Zoho support because it's confusing our users (there's really no reason for them to see it), but we haven't

    • Zoho CRM Sales Targets for Individual Salespeople

      Our organistion has salespeople that are allocated to different regions and have different annual sales targets as a result. I am building an CRM analytics dashboard for the sales team, which will display a target meter for the logged in salesperson.

    • All new Address Field in Zoho CRM: maintain structured and accurate address inputs

      The address field will be available exclusively for IN DC users. We'll keep you updated on the DC-specific rollout soon. It's currently available for all new sign-ups and for existing Zoho CRM orgs which are in the Professional edition. Managing addresses

    • Transfer all Related Data to new Account Owner

      Currently when I change the account Owner I only see the option to change only the open deals But I want the new account owner to take over all the related modules and all the deal stages Is it not possible right now? Am I missing something? Do I really

    • Can i connect 2 instagram accounts to 1 brand?

      Can i connect 2 instagram accounts to 1 brand? Or Do i need to create 2 brands for that? also under what subscription package will this apply?

    • How to Calculate MTTR (Mean Time to Resolve)

      We want to calculate MTTR (Mean Time to Resolve) in our Zoho Analytics report under Tickets. Currently, we are using the following fields: Ticket ID Ticket Created Time Ticket Closed Time Ticket On Hold Time We are planning to calculate MTTR (in days)

    • How to export project tasks, including the comments

      Hi, how can I export the project tasks, whereby I can also see the comments associated to a specific task? The use-case is that often we use comments to discuss or update a task related ideas. I would like to export the tasks, where we can also see the

    • How to Install Zoho Workdrive Desktop Sync for Ubuntu?

      Hi. I am newbie to Linux / Ubuntu. I downloaded a tar.gz file from Workdrive for installing the Workdrive Desktop Sync tool. Can someone give me step by step guide on how to install this on Ubuntu? I am using Ubuntu 19.04. Regards Senthil

    • Does Zoho Sheet Supports https://n8n.io ?

      Does Zoho Sheet Supports https://n8n.io ? If not, can we take this as an idea and deploy in future please? Thanks

    • Bigin Android app update: User management

      Hello everyone! In the most recent Bigin Android app update, we have brought in support for the 'Users and Controls' section. You can now manage the users in your organization within the mobile app. There are three tabs in the 'Users and Controls' section:

    • Share records with your customers and let them track their statuses in real time.

      Greetings, I hope everyone is doing well! We're excited to introduce the external sharing feature for pipeline records. This new enhancement enables you to share pipeline records with your customers via a shareable link and thereby track the status of

    • Live webinar: Discover Zoho Show: A complete walkthrough

      Hello everyone, We’re excited to invite you to our upcoming live webinar, Discover Zoho Show: A Complete Walkthrough. Whether you’re just getting started with Show or eager to explore advanced capabilities, this session will show you useful tips and features

    • Deal Stage component/widget/whatever it is... event

      Deal Stages I am trying to access the event and value of this component. I can do it by changing the Stage field but users can also change a Deal Stage via this component and I need to be able to capture both values. Clicking on 'Verbal' for instance,

    • Create advanced slideshows with hybrid reports using Zoho Projects Plus

      Are your quarterly meetings coming up? It’s time to pull up metrics, generate reports, and juggle between slides yet again. While this may be easier for smaller projects, large organizations that run multiple projects may experience the pressure when

    • Add an option to disable ZIA suggestions

      Currently, ZIA in Zoho Inventory automatically provides suggestions, such as sending order confirmation emails. However, there is no way to disable this feature. In our case, orders are automatically created by customers, and we’ve built a custom workflow

    • Email Integration - Zoho CRM - OAuth and IMAP

      Hello, We are attempting to integrate our Microsoft 365 email with Zoho CRM. We are using the documentation at Email Configuration for IMAP and POP3 (zoho.com) We use Microsoft 365 and per their recommendations (and requirements) for secure email we have

    • Formula field with IF statement based on picklist field and string output to copy/paste in multi-line field via function

      Hello there, I am working on a formula field based on a 3-item picklist field (i.e. *empty value*, 'Progress payment', 'Letter of credit'). Depending on the picked item, the formula field shall give a specific multi-line string (say 'XXX' in case of 'Progress

    • CRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive

      Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage files

    • Zoho CRMの流入元について

      Zoho CRMとZoho formsを連携し、 formsで作成したフォームをサイトに埋め込み運用中です。 UTMパラメータの取得をformsを行い、Zoho CRMの見込み客タブにカスタム項目で反映される状況になっています。 広告に関してはUTMパラメータで取得できているため問題ないのですが、オーガニック流入でフォーム送信の場合も計測したいです。メールやGoogle、Yahoo、directなどの流入元のチャネルが反映されるようにしたいのですが、どのように設定したら良いでしょうか。 また、

    • Error While Sign in on Zoho Work Drive

      Dear Team, I hope this email finds you well. I have recently created a Zoho account and started using it. But while I am trying to log in to Zoho work drive it won't log me in its crashing every time I try it. I have tried it on android app, phone browser

    • Choosing a portal option and the "Unified customer portal"?

      I am trialling Zoho to replace various existing systems, one of which is a customer portal. Our portal allows clients to add and edit bookings, complete forms, manage their subscriptions and edit some CRM info. I am trying to understand how I might best

    • Elevate your CX delivery using CommandCenter 2.0: Simplified builder; seamless orchestration

      Most businesses want to create memorable customer experiences—but they often find it hard to keep them smooth, especially as they grow. To achieve a state of flow across their processes, teams often stitch together a series of automations using Workflow

    • Unified Directory : How to Access ?

      I signed in to Zoho One this morning and was met with the pop up about the upgraded directory (yay!) I watched the video and pressed "Get Started" ... and it took me back to the standard interface. How do I actually access the new portal/directory ?

    • Translation support expanded for Modules, Subforms and Related Lists

      Hello Everyone!   The translation feature enables organizations to translate certain values in their CRM interface into different languages. Previously, the only values that could be translated were picklist values and field names. However, we have extended

    • Unified task view

      Possible to enable the unified task view in Trident, that is currently available in Mail?

    • Bigin, more powerful than ever on iOS 26, iPadOS 26, macOS Tahoe, and watchOS 26.

      Hot on the heels of Apple’s latest OS updates, we’ve rolled out several enhancements and features designed to help you get the most from your Apple devices. Enjoy a refined user experience with smoother navigation and a more content-focused Liquid Glass

    • Importing data into Assets

      So we have a module in Zoho CRM called customers equipments. It links to customers modules, accounts (if needed) and products. I made a sample export and created extra fields in zoho fsm assets module. The import fails. Could not find a matching parent

    • Allow instruction field in Job Sheets

      Hello, I would like to know if it is possible to have an instruction field (multi line text) in a job sheet or if there is a workaround to be able to do it. Currently we are pretty limited in terms of fields in job sheets which makes it a bit of a struggle

    • Streamlining Work Order Automation with Zoho Projects, Writer & WorkDrive

      Hello Community, Here is the first post in 'Integration & Automation' Series. Use Case :: Create, Merge, Sign & Store Documents in Zoho WorkDrive. Scenario :: You have a standard Work Order template created in Zoho Writer. When a task status is chosen

    • The dimensions of multilingual power

      Hola, saludos de Zoho Desk. Bonjour, salutations de Zoho Desk. Hallo, Grüße von Zoho Desk. Ciao, saluti da Zoho Desk. Olá, saudações da Zoho Desk. வணக்கம், Zoho Desk இலிருந்து வாழ்த்துகள். 你好，来自 Zoho Desk 的问候。 مرحباً، تحيات من Zoho Desk. नमस्ते, Zoho

    • Multi-line address lines

      How can I enter and migrate the following 123 state street Suite 2 Into a contact address. For Salesforce imports, a CR between the information works. The ZOHO migration tool just ignores it. Plus, I can't seem to even enter it on the standard entry screen.

    • Accessing Zoho Forms

      Hi all, We're having trouble giving me access to our company's Zoho Forms account. I can log in to a Forms account that I can see was set up a year ago, but can't see any shared forms. I can log into Zoho CRM and see our company information there without

    • Archiving Contacts

      How do I archive a list of contacts, or individual contacts?

    • Cost of good field

      Is there a way we can have cost of good sold as a field added to the back end of the invoicing procedure and available in reports?

    • Next Page