Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Compensation Cess on Coal ₹400 per tonne. ?????

      The compensation cess rate varies by the type of product. And the cess is calculated based on the value of the product without GST. Coal, for example, comes with a cess of ₹400 per tonne. That means that if you sell 2 tonnes of coal that have a value

    • 7 month over zoho book purchase but still not immpliments Golive

      7 month over zoho book purchase but still not immpliments Golive one problems zoho team short out then other problems come still very poor mangments and immliments team . struggling with the templates in ZOHO Books. Especially with the placement of some

    • Average Costing / Weighted Average Costing

      Hello fellow maadirs. I understand Zoho Books uses FIFO method of dealing with inventory costing, but do you guys have any plans to introduce average costing? We indians need average costing. It's part of our culture. Please. I beg thee. Thanks.

    • SMS to customers from within Bigin

      Hi All, Is there anyone else crying out for Bigin SMS capability to send an SMS to customers directly from the Bigin interface? We have inbuilt telephony already with call recordings which works well. What's lacking is the ability to send and receive

    • How to update/remove file in zoho creator widgets using javascript API

      Hi Team, I have developed a widget which allows inserting and updating records I have file upload field with multiple file upload. Now while doing insert form record, I am using uploadFile API to upload files for that record. I am using updateRecord API

    • Parent & Member Accounts (batch updating / inheritance)

      Hello, I find the Parent Account functionality very useful for creating custom views and reports, but was wondering if I can also carry out batch editing on all members (aka children) of a Parent Account at the same time. Alternatively, can I set members to automatically inherit the values of the parent? For example: We have a chain of supermarkets that buy our products. These supermarkets are all members of a Parent Account in our CRM. We release a new product and all of the member stores wish to

    • Edit Legend of Chart

      I would like to edit the legend of the chart. Every time I enable the legend, I get a very unhelpful (1), and when I try to type to change to what I would desire, nothing happens, which is very frustrating. I've gone through your online tutorials and nowhere can I find a legend settings button. This seems a simple fix, where can edit the legend? Thanks.

    • Extended timeouts for APIs beyond 40secs for to accomodate LLMs

      A 40 second max response time for API calls is fine when connecting to most services, however is unsuitable when dealing with LLMs (ChatGPT/Claude/Gemini) where the response timing is very uncertain. Is there any way to increase this? It would be great

    • Deletion of Zoho Account

      To whom it may concern, Good day, My account has been created incorrectly in Zoho and I am not able to join my Company's Zoho account - attached screenshot for your kind reference Alphatronmarine - Portal Kindly advise procedure to delete this current

    • Workflow for deposit to bank account

      Hello, Is it possible to make a workflow when a deposit is made to your bank account which is coupled to Zoho books? I want Zoho to sent an email each time a deposit is made to our bank account via a workflow. Regards, Steven

    • Marking Retainer invoice paid through Deluge

      Hey Everyone, We have a scenario where we are collecting deposit payments on our website. Now, in zoho books, we need to create a retainer invoice and mark it as paid automatically using deluge just like we can mark normal invoices as paid. I have tried

    • Create a new record in custom module vi custom button

      I have zoho books premium plan . I have 2 custom modules in zoho books. 1. Goods Receipt 2. Delivery Order, I need to select multiple records from Goods Receipt and create a new Delivery order from these multiple records. (like multilple sales order into

    • Profile date settings

      At present I have "EEE, MMMM dd, yyyy" but this takes an exessive amount of column space, we should be able to input our own format. I would like to use "EEE, MMM dd, yy" - a much shorter version of the above but with the same abbreviated info, requiring

    • Delivery Method Field in Sales Order Module

      In Books and in Sales orders, the "Delivery Method" field seems to allow for anything to be entered and it seems to store those entries for future use.  When you chose to convert a sales order to a purchase order, the related field is now called "Shipment

    • Editing / Removing stages for pipeline

      Hello, I'm trying to create a new pipeline. I created a new stage and made an error when entering the probability. How can I edit fields in stages that I created? Can I delete these stages from "Add Stages" list?

    • Dynamically Filter User Lookup in CRM Subform

      We have a subform called Pricing Calculator in the Zoho CRM Opportunity module and need some assistance. Current Setup: First column: Picklist (Level) Second column: User Lookup field When a Level is selected, we want the User lookup to display only users

    • change time zone

      can't seem to figure out how to change the time zone of the project

    • Bigin iOS app update: Built-in telephony and RingCentral support

      Hello everyone! We are excited to introduce Built-In Telephony and RingCentral support in the latest iOS version(v1.11.13) of the Bigin mobile app. Once the integration is completed on the Bigin desktop site(bigin.zoho.com), you can choose the Built-In

    • Add Image or Update Image API - for Items Module

      I am trying to add new Items to Zoho Inventory from Zoho Creator. I achieved this using Zoho Inventory Create Item API, but how to add or update the item image from Zoho Creator to Zoho Inventory Item Module?

    • Introducing Booking Pages—a topping for your Calendar Scheduling needs!

      Greetings, We're here with a new topping for Bigin! Let's dive into the details. What does this topping do? Scheduling appointments with customers is one of the most common challenges small businesses face on a daily basis, as it often involves frequent

    • Debugging `try` blocks : Tip

      I find it annoying that if one line inside a `try` block has an error, the Deluge arser points the beginning of the block to the location of the error. BUT, if you temporarily comment out the initial `try {`  The parser goes through the whole block and

    • [Product Update] TimeSheets module is now renamed as Time Logs in Zoho Projects.

      Dear Zoho Analytics customers, As part of the ongoing enhancements in Zoho Projects, the Timesheets module has been renamed to Time Logs. However, the module name will continue to be displayed as Timesheets in Zoho Analytics until the relevant APIs are

    • Use approval workflow comments in record scripts

      Greetings, i'm running an approval workflow for my records, during approval/rejection there is a step where comments are entered. i want to add there comments to the record and to use them in various deluge scripts like sending emails and so on.  how

    • ZOHO Store

      Not able to make a payment We are using Zoho One, and we are from India. The payment currency, which shows for us, is in USD. But the system says we can choose Country/Region India if it shows INR only. Attaching screenshots for more info.

    • Support Migration into Aliases in Zoho Mail

      Hello Zoho Mail Team, How are you? We are in the process of migrating some of our users from Google Workspace (Gmail and Google Drive) to Zoho. During this process, we noticed that Zoho Mail currently only supports migration into a primary mailbox and

    • API for Z Workdrive Flow Make-Integromat ?

      We are zoho workdrive fans Also we would like to have an api to work with Zoho Flow or with Make better known by its old name INTEGROMAT Is it planned and when? 3 months -6 months or more?

    • Apps Pane no longer visible

      I have read all the info and help and nothing works, I do not have a "show apps" anywhere and I can no longer see my Apps pane in the left hand side of mail, please advise how to get this back

    • Canvas View - Print

      What is the best way to accomplish a print to PDF of the canvas view?

    • 5名限定 課題解決型ワークショップイベント Zoho ワークアウト開催のお知らせ(8/21)

      ユーザーの皆さま、こんにちは。Zoho ユーザーコミュニティチームの藤澤です。 8月開催のZoho ワークアウトについてお知らせします。 今回はZoomにてオンライン開催します。 ▷▷参加登録はこちら：https://us02web.zoom.us/meeting/register/eVOEnBsSQ2uvX4WN5Z5DeQ ━━━━━━━━━━━━━━━━━━━━━━━━ Zoho ワークアウトとは？ Zoho ユーザー同士で交流しながら、サービスに関する疑問や不明点の解消を目的とした「Zoho

    • New in Zoho Forms: Inline OTP Verification

      Hello form builders, We are excited to announce the launch of Inline OTP Verification in Zoho Forms, a smarter way to ensure the authenticity of the contact details you collect. Until now, OTP Verification in Zoho Forms worked as a pre-access step: respondents

    • Zoho Mail : Associate emails with Meeting records and allow multiple emails to be assocaited at once

      Is there a workaround that would allow either of these? I want to associate emails with Meeting records. I also would like to be able to select multiple emails at once for association with a record.

    • Create task from email

      Is there a way on mobile to create a task from an email? I use this feature a lot and when traveling now I read email on mobile. By the time I get to my office I forget about them since I didn't add it to a task. Is this feature missing on moble?

    • Zoho Socials - Unable to view Channels and SmartQ

      Hi, The channel Facebook has been added by the admin, however, it is not visible on the User level (employee). Other channels are visible. Also, we have the premium account, and SmartQ is not working. Can anyone help? Regards, Priyanka

    • Eliminating Manual Consolidation: Automating Currency Field Sync from Task to Project

      Hello Everyone, A Custom function is a user-written set of code to achieve a specific requirement. Set the required conditions needed as when to trigger using the Workflow rules (be it Tasks / Project) and associate the custom function to it. Requirement:

    • We want to set the "Converted from Lead" value in Deals using a Workflow or via a Deluge script. How?

      For use in Zoho Analytics, we need the field "Converted from Lead" filled in our deal records. This field is empty everywhere, because we do not create deals directly when converting a lead to a contact. We want to do that using the API or a workflow

    • Sales Orders: Quoted_Items + items in another subform -> into Ordered_Items ?

      hello, When creating Sales Orders, is it posible to inherit/fill the Ordered_Items with all the items from Quoted_Items + all the items from a customized subform with similar fields? Since you can create a sales order in different ways (convert, new -

    • How to cancel the GSTR1 pushed to GSTN

      How to cancel the GSTR1 Pushed to GSTN, some rectifications to be done in HSN & SAC code

    • Zoho Books API — Invalid Operation Type / Scope does not exist

      Hello Team, We are unable to use the Zoho Books API from our registered application. We’ve already: Created a client in Zoho API Console using (Admin in Books) Generated the OAuth code and token successfully Used the correct scopes: ZohoBooks.fullaccess,ZohoOauth.userinfo.READ

    • Enhancements in Canvas

      Dear All, Greetings! Canvas lets you design the record details page to suit your brand or business preferences. We are glad to introduce the following enhancements to uplift your design experience. Reusable Components Style Presets Let's go! Reusable

    • Minimum order quantity

      Is there a way to enforce a minimum order quantity - ie has to have a minimum of 250?

    • Next Page