Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • CRM - Site/Location

      Hi Zoho, One massive oversight of the Zoho CRM is the inability to add multiple sites or locations to one account. Many organisations have several (in some cases, hundreds) of sites across the country or even the globe. The workaround to this, of course,

    • An unknown error occurred. Please contact support

      Whenever I visit this page I see this. I changed browser and still the same. Can someone from Zoho help me please?

    • How to Streamline Pick & Ship

      Is there a way to streamline the built-in pick and ship process? I guess mostly on the ship side. Our current process allows us to choose shipping super fast. It's an EasyPost plugin for WooCommerce. You have to populate the boxes / weights / shipping

    • How to add, delete or rename the "payment method" drop down?

      When we recorded payment in invoice, there's a "payment method" drop down with choices like cash, bank transfer... We want to add and delete or rename some new selection. How to do such adjustment? Please advice.

    • Smarter data gathering with Query component in Wizards

      Dear All, Introducing the Query Component in Wizards for CRM! A smart search field that saves you time and effort, and helps you manage and gather data more efficiently than ever before. Long and complex record entries can be overwhelming and prone to

    • Introducing teamspaces and team modules in Zoho CRM mobile app

      Hello everyone, We have an exciting update to share in the Zoho CRM mobile app. As part of CRM For Everyone—a new set of features that reflect our vision to democratize CRM and make it accessible for all—teamspaces and team modules are now available on

    • Client Script - change page and/or section background colours

      Hello, Would anyone be willing to help me with a bit of Client Scripting code? I have had a go myself but only been able to alter the form field styles, not the overall page/section/heading styles. I want to set a different page background colour for

    • Zoho SalesIQのチャットボット、ブロックのコピー機能

      Zoho SalesIQのチャットボットの構築でドラッグアンドドロップで作成を行っているます。 内容は同じブロックのコピーペースト機能がないみたいなのですが、同一のブロック、同一の複数のブロックをいくつも作成する場合は、皆様はどのように行われていますか？ 例えば添付の４つのブロックをまとめてコピーして、別のフローの先につなげる場合です。 教えていただけますと幸いです。よろしくお願いいたします。

    • Questions About Zoho Commerce Member Portal

      Hello, A couple questions about the Zoho Commerce Member Portal: 1. Can I add only specific pages be added to the Members Portal, and not the entire website? 2. When a customer signs up on my Zoho Commerce site, is there a setting that gives me a chance

    • Cancel and refund

      Hi, Yesterday I paid for the Zoho email subscription. Within a few minutes, I realized that the subscription counts one email ID as one user. To make another email ID for the same domain name, I'd have to pay for another user. I emailed right away to

    • Online Payment Fees

      We don't take many online credit card payments so the merchant service provider (PayPal) charges us the 2.9% fee for processing the amount. I would like the ability for the fee to be automatically added to the total amount for "ease of payment". We'd

    • Error occured. please try again!!!

      I created a new list. Added two new contact's email addresses. I try to EDIT either one of them to add their first name, last name and phone number. Whatever I try to edit and update, I get the error message  Error occured. please try again!!! And it doesn't update any of their info. I tried just editing first and last name. Still get the error. No matter what I try to update, I get the error. I tried creating a new list, and adding just one email address and then edit it and also get the error.

    • "Send email as" not work

      Hi team, I currently use ZOHO MAIL services on the MAKE platform (formerly INTEGROMAT). In MAKE I entered my ZOHO MAIL credentials but the problem is that when the emails are sent from the automation in MAKE, the emails do not come in with the name set

    • *UPDATE: ISSUE RESOLVED, SEE HOW FOR HELP* Issue imbedding Youtube Videos

      **UPDATE: RESOLVED** For anyone that is having a similar issue, try this workaround before attempting to have the Zoho techs fix it. I have been in communication with them for weeks about this and they can't seem to fix the issue. Thinking it may have just been my computer or my browser, I tried two different computers on two different networks, each running different versions of Windows, and tried it on Chrome, Firefox and IE; nothing worked. So, needless to say, it's an issue with Zoho somewhere.

    • Backorder For Composite Items

      Hello If i released SO for composite item and use backorder feature of zoho inventory then it should backorder child item items of composite and not composite item.This is basic of backordering.I conveyed this to zoho call center but no solution yet.

    • Paid Support

      We are in the process of creating tiers for our support offerings. There will be three tiers of support available - 2 paid and 1 free. The paid tiers will be purchased in allotted hours. I need to figure out how to a) work in the out of the box SLA, Contracts

    • Automatically CC supervisor

      Hello,  in our organization agents are replying to tickets and they are required to CC their supervisor on all emails to and from customers.  is there a way to have an email address automatically populate in the CC field when reply to a ticket?  tha

    • Zoho One - White Label

      Releasing a white-label feature for Zoho One, or any software or service, can offer several advantages and benefits for both the company providing the software (Zoho in this case) and its users. Here are some key reasons for releasing a white-label feature

    • Are this Features included in the Road Map

      1. Sync between description/short description in Zoho Books/Inventory and Zoho Commerce: I am unsure how it works now. I belive the description is not pulled. 2. Pick Up: This feature is great. It is added on Zoho Commerce but not on Books/Inventory which

    • Reconciliation - Show Transactions After End Date

      I did a quick search, and I've only seen this brought up indirectly. In the reconciliation interface, transactions dated up to a week after the end date should be available for being reconciled. The use case is pretty simple... many times, transactions

    • Auto populate email field in zoho form

      Hi,  Is there a way the email field in zoho forms can be auto populated based on the name selected from a drop down list?

    • Option to accept and pay for estimate from Public Estimate link?

      Is there any way that our customers can get the Estimate in their email, click on "View Estimate" and from the public link, click Accept and Pay? When they click that, in the background Zoho would create the invoice record in the so that the payment would

    • Kaizen #125 Manipulating Multi-Select Lookup fields (MxN) using Zoho CRM APIs

      Hello everyone! Welcome back to another week of Kaizen. In last week's post in the Kaizen series, we discussed how subforms work in Zoho CRM and how to manipulate subform data using Zoho CRM APIs. In this post, we will discuss how to manipulate a multi-select

    • Easily track referrals with Zoho Forms

      Referral tracking can be a powerful way for businesses to understand where their leads are coming from. With Zoho Forms, tracking the referral sources of your leads is an easy and straightforward process. Here are some tips to help you make the most of

    • Add an email to an existing ticket is not working

      I noticed that in Zoho Desk the funcionality to add an email to an existing ticket is not working using the syntax [##12345##], has the method changed? In red is the syntax we use to add email to an existing ticket As you can see, he did not add the email

    • New CRM to Campaigns Sync Doesn't Continue Making Updates

      Changes made in CRM are not appearing in mapped fields in matching Campaign records after migrating to the new sync process. The only way we've found to get records to update is to call into Support and point out the problem. After convincing Support

    • Zoho Integration with UPS

      I have 2 questions: Firstly, is there a way to notify UPS that we have a package to collect once we have done the shipping label? Secondly, how do I get the tracking number and shipment method onto the Invoice and Package Slip for the customer? Than

    • Cannot log in to IMAP account as of last night

      Hey I've been using MFA with an authenticator for a while and have had to use application passwords for Outlook and Edison Mail on my Android devices. Last night the app passwords started to be rejected on my Android devices so I created new ones for

    • Low Stock View

      We use the Low Stock view frequently as a guide to inform us when to reorder items, but the view is misleading because it does not take into account Purchase Orders that have already been raised. Unless you are aware and check item by item, this can lead

    • Improve History Feature in Zoho Inventory

      At present there is a "history" tab on Zoho Inventory Items, however this only shows a date and time stamp along with the users name. It doesn't say what was changed. What is the value of this if you can't see what was changed? My Ideal is to include

    • Show backordered items on packing slip

      We send out a lot of large orders, and often there are one or two things backordered. How can I fix the packing slips to show quantity ordered  & quantity packed There should also be the ability to "ship" 0 of an item so the receiver knows that things

    • Document | Files

      The vendor "Partial matches" still not fixed here after years of putting up with having to select most vendors manually ! ( again, the banking "Transaction rules" would solve a lot of these issues ) Some unwanted, irrelevant pdf's also arrive. It would

    • books+POS+ tap to pay+ stripe

      So in the UK we now have tap to pay with stripe. So we can use the stripe app as a POS terminal. Brilliant news. Can we hope that the Books App might add this feature ASAP. It would be great to have one system rather than using the not very good Square

    • Canvas Form View - Client Script Page on load - Url params not working

      We have a custom module with a canvas page for form view. We have a button in Customers module to open this canvas page using invokeurl function. We send few parameters as in the URL as query parameters. https://crm.zoho.in/crm/orgxxxxxxxx/tab/CustomModule12/create/canvas/64333200000261xxxx?layoutId=643332000002605001&c=${Customers.Customer

    • Sending Recruit SMS's to Zoho Cliq - Or tracking in the Messages module in Recruit?

      Is there any way to send SMS Gateway messages in Recruit to ZOho Cliq? We use 2-way SMS massages a lot in Zoho Recruit to speed up communication with Candidates. However the only way to keep track of received SMS's is by keeping a look out for the Email

    • This mobile number has been marked spam. Please contact support.

      Hi Support, Can you tell me why number was marked as spam. I have having difficult to add my number as you keep requesting i must use it. My number is +63....163 Or is Zoho company excluding Philippines from their services?

    • Zohomail does not support additional fields in mailto links

      Hello, I set up Zohomail as default composer in Firefox according to manual here: https://www.zoho.com/mail/help/defaultcomposer.html#alink2 Later, I wanted to use this functionality to answer in a Linux mailing list thread using mailto link: mailto:xxxxx@kernel.org?In-Reply-To=%3C727o0521-q24p-s0qq-66n0-sn436rpqqr1p@example.com%3E&Cc=xxxxx%example.org&Subject=Re%3A%20%5BPATCH%20v2%28%29

    • Is it possible to set a customer context variable in Zobot

      Hi, I want to use a context variable to route users down different paths in my Zobot chat flow. I know I can do this when the user enters data. But I want to know if I can use a variable that is 'hard coded' on the card, that the user is unaware of. Use

    • Cannot change Blog Title

      There is nowhere to change the blog title. You can change the blog URL but that is making no difference to the text "Enter Your Post Title" am I missing something?

    • Kaizen #203 - Answering Your Questions | Handling API Limits and Error Responses

      Hi Everyone, Welcome back to yet another post in the Kaizen Series! We appreciate your keen participation in the 200th milestone celebration of the Kaizen series. We will continue to answer the queries we received through the feedback. When working with

    • Next Page