Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Add and Remove Agents from Departments and Groups in Zoho One

      Hi Zoho Flow Team, We hope you're doing well. Currently, Zoho Flow provides an action to add an agent to a group in zoho one, but there is no action to remove an agent from a group or a department. Another action that we find missing is the option to

    • Zoho learn Custom portal - networkurl & CustomPortalId

      I want to get my individual account’s networkurl and customportalId to use in this API: https://learn.zoho.com/learn/api/v1/portal/<networkurl>/customportal/<customportalId>/manual How can I retrieve the networkurl and customportalId using the API? I

    • Consumer Financing

      Does Zoho currently have a payment gateway (such as Stripe, Square, etc) which offers financing for customers? So, let's say the estimate we give the customer is greater than what they can afford at the time, but we can sell the service now, letting them

    • Intégration de la gestion des Passkeys dans Zoho Vault

      Zoho Vault est depuis plus d’une décennie une solution fiable pour les entreprises : pour la gestion, le partage et le stockage des mots de passe. En 2018, nous avons fait un pas en avant en proposant la connexion unique (SSO). Nous sommes fiers de franchir

    • Scan & Fill with double quote key/value pairs

      Hi, An old Ticket moved to a Topic/Idea: I love the idea of the new Scan & Fill as it nearly covers my previous request for a QR Scanner to read a multi-part QR Code. My QR Codes are hard-coded as below: {"key1":"value1","key2":"value2","key3":"value3"}

    • Analytics SQL Queries should allow # as comment

      # and // are very common for commenting in SQL. Not sure why analytics only allows /* and */ for commenting. Especially when # grays the line as if it's being commented out. This should be added for sure.

    • SalesIQ Operator Activity Reports in Zoho Analytics

      I'm busy building a dashboard in Zoho Analytics and I want to include SalesIQ stats in the dashboard, but I'm unable to get the statistics mentioned in the attached image. Any idea where I can get the stats for Operator Activity?

    • Default in fields on Form B based on the user selection in Form A

      Hi Everyone, I have added an action button to a form report to bring up a new form based on user selection, see it indicated in red below: Then when the ne form loads, I want to default in some of the fields based on the record the user was selected on.

    • No longer can indent

      Hey there! Is it just me or were we used to be allowed to used tab or indent when writing. It’s not working right now, has this always been the case?

    • Free webinar alert! Seamless Transition with Lossless Migration: Zoho One + Zoho Mail

      Hello Zoho Mail Community! 🚀 Attention IT Admins and Email Administrators! Are you planning to migrate your organization's email to Zoho Mail within the Zoho One ecosystem? 📧 Join our exclusive webinar, Seamless Transition with Lossless Migration: Zoho

    • Add Resource to Export

      The Export Data feature does not include a column for the Resource field. Without this column, Zoho Bookings cannot be used by any business for resource-based services or event types e.g. room bookings, equipment bookings. It seems to be an oversight,

    • Client Script | Update - Client Script Support For Custom Buttons

      Hello everyone! We are excited to announce one of the most requested features - Client Script support for Custom Buttons. This enhancement lets you run custom logic on button actions, giving you greater flexibility and control over your user interactions.

    • Mandatory field via deluge code

      I would like to ask you if it is possible to make a field mandatory via deluge script. For example, if I have a decision box and I click on it then I want a single line field to be mandatory. If uncheck the decision box then to do the single line as optional. I think it is not possible to do that and I have to do it via validation in 'on validate' field. 

    • Revenue Management: #1 What does it mean to "recognize" revenue?

      Earning revenue isn't just about collecting cash from your customers. It's about recording the income correctly and consistently. Revenue recognition is the process of deciding when and how to record revenue in financial statements so that they reflect

    • Power of Automation :: Auto-Populate Integration Field in Projects with CRM Account Data

      Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate

    • Zoho Forms and ChatGPT - populating a field using AI.

      I have a form where I would like the user to enter a response or query, and have another field populated using AI. For example, user enters Field 1, AI populates Field 2 in response. I want to be able to wrap some additional instruction text around the

    • campo tag para api

      debo conectarme a una api de zoho inventory y ocupo tomar el campo tag para poder asi jalar el articulo que cuente con el campo correcto en tag ejemplo que tag existen carro y avion que cuando busque los articulo con tag carro arroje solo estos por mas

    • Uploading file as attachment to Zoho CRM

             Hi,   I am trying to attach a file to a Zoho CRM contact using Zoho Flow. Right now, I try to do it through the “Upload File” field in Zoho CRM (In my screenshots, it’s called Téléchargement du fichier 1).   Here is what I tried:   Case 1: Webmerge document The Flow is called “Custom Function” (see screenshot 101).   Step  1: Creating a Webmerge document (screenshot 99)   Step 2: I use “Update module entry” to upload the created file. I upload Webmerge’s “Document” in my “Téléchargemet du

    • Zia Answer Bot - Create Ticket

      Surprisingly, the current iteration of Zia will try to answer a question and unless you have "transfer to SalesIQ chat" enabled, it won't create a ticket for the user or offer them a method to create a ticket. We don't want it to create chats for us,

    • meassure leads phases

      Hi, I need to create a table to meassure the time that a lead stay in blueprint phases. the phases are first contact, second contact, lead spam, contacted, appointment. any idea? I have attached an example

    • Zoho Desk API Documentation missing a required field

      We are trying to create a section using this information. Even after preparing everything based on that page, we still get an error. The error we get is this: {"errorCode":"INVALID_DATA","message":"The data is invalid due to validation restrictions","errors":[{"fieldName":"/translations","errorType":"missing","errorMessage":""}]}

    • In the Custom Module I have 500 Records , this 500 record only want to view to the specific user only example user A ,

      In the Custom Module, I have 500 Old records that should only be visible to a specific user, for example, User A. Any new records created from today onwards should be visible to Record owner in the Custom Module. Pls help how i achive this .

    • Invoice template, how to change the text under "Notes" and "Terms and Conditions"

      In "Invoice templates", there are two text/info sections at the bottom:"Notes" and "Terms and Conditions". It is possible to change the names of these two headings, but how is it possible to change/alter the text under it. As a standard it says "Thank you for your business" under Notes - I need to change it into something different- How? Thank you.

    • How to reply to thread via API

      We have built a webapp for our customers that uses the Zoho Desk API to enable each customer to view their full list of tickets, view individual tickets and raise new tickets. The Zoho Desk API doesn't have the ability to reply to a ticket/thread. Replies

    • Sending merged mail templates for signatures fail since today

      We have ZOHO one, we use merge templates in CRM to edit in ZOHO Writer, and from there send it for signature through zoho sign. This all worked up until today, suddenly we read in the log that the merge is succesfull but the sending for signature failed.

    • Feature Request - Make Lead List Larger and Adjustable

      Hi LandingPage team, I recently started using LandingPage and I am happy to share my feedback to help improve the app. I've noticed on the Leads page, there is no option to make the columns wider. It would be great if the comlumns expanded to fit the

    • Zoho Projects - Pin Recent Projects

      Hi Projects Team, It would be great if I could "pin" projects on the Recent Projects list in Zoho Projects. We have some internal projects which we regularly have to add time and some regular client projects. It would be great if I could pin those projects

    • ZDK Error

      I get this error when trying to trigger a CRM Function from Client Script: Uncaught (in promise) ZDKError: {"code":"NOT_ACTIVE","details":{"api_name":"activate_client_from_prospect"},"message":"api is inactive for the given custom function","status":"error"}

    • "Disbursing product components in phases, monitoring them, and displaying only the final product."

      i have a product composed of multiple components, and these components will be delivered to the customer in batches. However, the final invoice should only show the finished product. How can I issue (or release) the components and track their delive

    • Followed Subtasks doesn't show up in the Subtasks Section

      I have a task assigned to me now in the same task, there's a subtask and I am added as the follower on that task Even though I am a follower I still don't see that in the subtasks section The view permission for the profile is Related It's supposed to

    • Tip #39- Strengthen account security with Multi-factor Authentication (MFA) – ‘Insider Insights’

      Securing your organization's data begins with verifying that only the correct individuals have access to it. One of the simplest yet most effective ways to accomplish this is to enable Multi-factor Authentication (MFA) within Zoho Assist. MFA introduces

    • Automate timeout chat tracking with Workflows in SalesIQ

      With our feature-packed Nova release, Workflows has become one of the most powerful tools in Zoho SalesIQ. They let you automate follow-up actions when key events occur, such as when a chat ends, a visitor leaves a bad/good rating, or a lead is updated.

    • Mass edit / Mass update products

      Hi, Is there any way to mass update or bulk edit product fields in Zoho Inventory?

    • Automatic Verification of IMAP Integration Status

      Our sales staff have their O365 email integrated with CRM, over time this integration requires re-authentication via the UI. I can manually check the integration status by accessing Settings -> Channels -> Email -> Email Sharing -> "Configuration Type"

    • Tip of the Week #68– Share and access files faster with Zoho WorkDrive extension.

      Have you ever wasted time searching for the right file to attach to your emails—or worried whether the right people could access it? Without proper sharing settings, files might end up inaccessible to teammates or, worse, visible to people who shouldn’t

    • CV-Library: The Newest Source Booster in Zoho Recruit!

      We’ve expanded your sourcing toolkit — CV-Library, one of the UK’s largest and most trusted online job boards, is now available as a Source Booster in Zoho Recruit. This gives recruiters instant access to millions of UK-based candidate profiles, all without

    • Zoho AI Translate Task as Rest API

      I cant find any docs on how to use Zoho AI Translate Task from a rest api call https://www.zoho.com/deluge/help/ai-tasks/translate.html I am working on a custom Widget and I dont think I can execute zoho deluge ai translate task from a custom widget.

    • Is there the possibility to book less than 250 customer portal users?

      If you use the Creator, which is included in Zoho ONe, you can create a customer portal and give access to a maximum of 3 external people / customers, right? On the Creator website I saw that you can add 250 users for 100€/month. However, we don't need

    • Display Parts custom fields in the Work Order

      Hello, I see that is it possible to add custom fields in the Parts module. Would it be possible to also add those custom fields in the Work Order Module under Parts? This would be very useful for customer who want to showcase more information about parts.

    • Unable to edit Saved Estimates

      We are facing issue with editing the price or discount for a saved estimate for nearly 1 week.

    • Next Page