Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Kaizen #203 - Answering Your Questions | Handling API Limits and Error Responses

      Hi Everyone, Welcome back to yet another post in the Kaizen Series! We appreciate your keen participation in the 200th milestone celebration of the Kaizen series. We will continue to answer the queries we received through the feedback. When working with

    • How to verify website ownership with google search console

      Hi, I am having a free .in domain provided by Zoho I have created a website on it now I want to verify my ownership to google webmaster. Can you please help me how to do that.

    • Kaizen #89 - Color Coding using Client Script

      Hello everyone! Welcome back to another exciting Kaizen post. Today let us see how you can apply color codes to the List and Detail Pages of Zoho CRM using Client Script. Need for color code in Zoho CRM When you mark things with different colors as a

    • "SPF record exceed the allowed limit of 10"

      Hi, I was wondering if there were a "universal Zoho SPF record" that would allow all my configured Zoho services that email things to meet this limitation ? or do I have to have an entry for mail, com, billing, etc?

    • Zoho Mail android app update: Block & reject future emails, Report phishing, Spam alerts, Suspicious URL detection

      Hello everyone! In the most recent Zoho Mail Android app update, we have brought in support for the following features: Block & reject future emails Report Phishing Spam alerts in the mail details screen. Suspicious URL detection. Block & reject future

    • Animated GIF Images in Chat

      I know this seems to be a small feature request but in a recent Cliq update Zoho disabled autoplay for animated GIFs posted in Cliq conversations. We think this was not a good change. In our organization, animated GIFs in a chat play a role in expressing

    • I can't seem to login in to Mail Apps of MacOS /IOS

      Hi, i'm having trouble in signing in to mail apps from IOS. It's always come back to wrong passwords. But i already changed my password like 3 times. But still it says wrong credentials

    • Profile Image Showing Incorrectly in Zoho Mail

      Hi everyone, I’m facing a serious issue with Zoho Mail. The profile image showing for my email address is incorrect — it’s not my image. When I send an email to my Gmail account, it displays someone else’s image. This looks very suspicious and can make

    • I need access to my old email

      I need access to my old email bromzeman@zoho.com to verify it for forwarding. I can’t access the inbox to get the confirmation code. Please assist with recovery or forwarding. as you might already know, they made alot of the email addresses to have that

    • Set Defaults for Email Groups

      Is there a way to set defaults for all of the groups that I establish as a moderator? For example, I want every group I establish to have the following settings: 1. Moderator is <user> 2. User is <user>, <user> 3. Notifications for new group turned

    • LOGS FOR RECEIVED MAIL ( READ DATE & TIME)

      In Zoho mail we can extract the read logs of received mails so that we can see when we have read the mail and at what time & we have read it.

    • Domain verification probem

      Hello, i use a domain from Namecheap with hosting from Cinfu when i try to verify my domain on zoho i get "TXT Verification failure" i even tried the HTML verification and the code appears but also giving me the verification failure error.

    • Switching to Cloudflare email routing from Zoho Mail

      Hello, I'm currently working on migrating from Zoho Mail to Cloudflare's email routing solution. This requires changing MX and TXT records for our custom domain - when we do this, will our users still be able to log into their accounts and access an archived

    • Un Subscription Button

      How can i Add the Un Subscription Button in Zoho mail

    • Documents unable view and need to downlad

      I can't view .doc files in Zoho mail unless I download them, but I can view PDF files without downloading.

    • we encountered a problem while sending your email. please try again later.

    • Adding and removing people from groups

      We're having problems adding people to a group. Apparently Zoho has one email address and will not recognize a different email address.

    • MAIL SEARCH IS NOT WORKING

      THIS ISSUE HAS BEEN BUGGING ME SINCE MORNING, PLEASE RESOLVE THIS AT THE EARLIEST

    • URL Parameter on Help Center SIgn in link

      Is it possible to add a url parameter to the sign in link on the Help Center?

    • migrating from HelpScout

      I am attempting to import a conversation file from helpscout into desk and am receiving size errors. What is the current file size restriction. Does anyone have any tips for a successful migration?

    • Layout Rules Don't Apply To Blueprints

      Hi Zoho the conditional layout rules for fields and making fields required don't work well with with Blueprints if those same fields are called DURING a Blueprint. Example. I have field A that is used in layout rule. If value of field A is "1" it is supposed to show and make required field B. If the value to field A is "2" it is supposed to show and make required field C. Now I have a Blueprint that says when last stage moves to "Closed," during the transition, the agent must fill out field A. Now

    • Article Name Sorting in Zoho Desk Knowledge Base (agent / admin side)

      Dear Zoho Desk Support, We are writing to request an enhancement to the Knowledge Base management feature within Zoho Desk. Currently, there is no option to sort articles by their name, which significantly hinders efficient article management, especially

    • How to parse JSON data with SQL in Zoho Analytics?

      Hi all, I have a column with JSON data. I want to show this column in a chart, but it is very messy, and no JSON parsing function is supported on Zoho Analytics. data example: {"id": 5, "status": "false", "date": "15/10/22"} what I want to do in SQL is

    • Ability to turn off "Would you like this amount to be reflected in the Payment field?" message

      Team, Is there any way to turn off the message" Would you like this amount to be reflected in the Payment field?" when I make a payment? This is so annoying. This happens EVERY TIME I put an amount in the Payment Made field.

    • Add an Equation Field (Or update the Formula Field)

      Hi, I would like to be able to have one field as a Text Field with QR Code, and then have multiple Equation/Formula Fields that then take parts of that fields data with LEFT, MID, RIGHT, REGEX, etc. Thanks Dan

    • How to parse column having JSON data using SQL?

      We have a daily sync from a PostgreSQL database that brings data into Zoho Analytics. Some of the columns store raw JSON data. We need to build SQL queries on top to parse data from JSON and store them in discrete columns. There is no option for "Data

    • Enable report button based on the current user role

      Greetings  i have a report that contains action buttons, i want these buttons to appear as enabled only when the current logged in user has a certain role, for example only CEO role users will be able to use this button. but when setting the conditions

    • 500 Internal Error In Mail API

      I'm getting 500 Internal Error when using mail API. I'm getting this error for this one account, it works fine for other Account IDs which I have in my system.

    • Piss poor service in Support in Domains and email

      Srijith Narayanan B contacted me today. Very pleasant fellow. Just didn't want to tell him how bad your support service is. You help the person, but you leave before we can finish the next stage. Which causes a lot of frustration. It's been 8 days now

    • Zoho live chat widget in React Js

      I am trying to test Zoho live chat widget code in react js, below is the sample code void(0)} onClick={()=>window.$zoho.salesiq.floatwindow.visible("show")}>LIVE CHAT window.$zoho = window.$zoho || {};window.$zoho.salesiq = window.$zoho.salesiq

    • Are there any plans to add Triggers for Subform edits?

      By The Grace of G-D.  Hi, How are you? Can you tell me if you have any plans to support subform edit as a workflow trigger? And what about have them trigger an "onChange" client script?

    • Zoho commerce

      i am facing issue with order summary emails.i am getting 1 continuous email for order received yesterday and today.ideally 1 email should be received for a particular date ie for 02/08 i should received 1 email from 12.01am till 11.59pm but it is being

    • Feature Request: Improve Category Page Sorting for "Out of Stock" Products

      Hi there, I'm writing to request a new feature that I believe would significantly improve the user experience in my online store. Currently, on category pages, products are sorted by popularity. However, when a popular product goes "Out of Stock," it

    • POSTMAN - There was an error in evaluating the Pre-request Script:Error: Cannot read properties of undefined (reading 'json')

      I am beginning the journey to learn how to use the API for Zoho Sign. I am getting the following error when I try to use postman. To walk you through how I am getting this error... I wanted to start with a simple GET and expand my learning from there.

    • How do i integrate shipstation with zoho inventory

      Wanting to set up my own delivery driver in ship station so we can get real time tracking of where the package is but then i want it to automatically update zoho inventory packages/shipments how can i do this

    • Invalid value passed for salesorder_id

      Hi, I am using sales return API, details are given below: API: https://inventory.zoho.com/api/v1/salesreturns?organization_id=700571811 Post Json Data: { "salesreturn_number": "", "date": "2020-11-12", "reason": "Testing from API", "line_items": [ { "item_id":

    • Create Invoice and Invoice Items from Sales Order via API

      Currently, when creating an Invoice associated with a Sales Order via the API, it appears that I must manually include all of the items (line_items) even though they are already part of the Sales Order. My question is this: is it possible to raise an Invoice via the API based on all of the information associated with a Sales Order--such as the  items? In other words, do I always have to manually include the items (line_items) when raising an Invoice via the API when the Invoice is associated with

    • Outlook 2013 Calendar Syncs but "Related To" Field in Zoho is blank

      Outlook 2013 Calendar Syncs but Related To Field in Zoho is blank I expect the "Realted To" field to be populated with the calendar participants

    • Export a Course

      Is it possible to export a course from Zoho Learn to a SCORM file?

    • Add and Remove Agents from Departments and Groups in Zoho One

      Hi Zoho Flow Team, We hope you're doing well. Currently, Zoho Flow provides an action to add an agent to a group in zoho one, but there is no action to remove an agent from a group or a department. Another action that we find missing is the option to

    • Next Page