Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • How do you print a refund check to customer?

      Maybe this is a dumb question, but how does anyone print a refund check to a customer? We cant find anywhere to either just print a check and pick a customer, or where to do so from a credit note.

    • Notebook

      I have purchased the monthly pro subscription of Notebook. But it does not support my XP-Pen to write something in it. So it is not useful to me. Hence I am requesting you to help me to discontinue this subscription.

    • Domain Mapping & Image Publishing Issues on Zoho Sites

      Hello, I am facing two issues with my Zoho Sites account: 1. Images not visible after publishing. 2. Domain mapping error: "Domain already exists". I am a paid customer. Please connect me with Live Chat Support or Zoho Assist so I can show my issue

    • Prevent duplicate with custom fields?

      I was wondering something about custom field/custom modules in Zoho Desk. For some reason you can make a custom field mandatory but not unique? For example, if I create a custom module to manage equipment and renewal and make a field serial number no

    • Round robin

      Hi, I'm trying to set up a round robin to automatically distribute tickets between agents in my team but only those tickets that are not otherwise distributed by other workflows or direct assignments. Is that possible and if so which criteria should I

    • "Is Zoho CRM customer" vs "Is linked with Zoho CRM"

      Recently while building a Flow, I was setting up a Decision action following a Zoho Invoice Fetch record action. There were 2 choices that I had not seen as something I could manually action in Zoho Invoice: "Is Zoho CRM customer" and "Is linked with

    • Two currencies

      More and more I am finding that internattional payments' fees are unpredictable. I would like, on my invoices that are in a foreign currency (eg. USD$ or EUR€) for there to be a GBP£ TOTAL display alongside the invoice's currency total. This would make

    • Automatic Matching from Bank Statements / Feeds

      Is it possible to have transactions from a feed or bank statement automatically match when certain criteria are met? My use case, which is pretty broadly applicable, is e-commerce transactions for merchant services accounts (clearing accounts). In these

    • Generate leads from instagram

      hello i have question. If connect instagram using zoho social, it is possible to get lead from instagram? example if someone send me direct message or comment on my post and then they generate to lead

    • Zoho PDF editor has a lot of issues.

      Zoho PDF editor needs a lot of work. It hangs and glitches a lot. Deletes annotations and clearings randomly.

    • Where is the desktop app for Zoho Projects???

      As a project manager, I need a desktop app for the projects I manage. Yes, there's the web app, which is AWESOME for cross browser and platform compatibility... but I need a real desktop app for Projects that allow me to enter offline information where

    • How to Automate Monthly PDF Reports with Filters in Zoho Creator

      Hi everyone, I’m trying to build an automated monthly reporting process in Zoho Creator and would appreciate suggestions or best practices from anyone who has done something similar. What I’m trying to do: I have a form called New_Customer with fields

    • Feedback: Streamlining Note Management in Zoho Notebook

      Dear Team/Support, I would like to share some feedback regarding the note management system that could help improve usability and accessibility for users like myself. Notebook 1 (screenshot attached): Currently, the system does not allow selecting and

    • showing Limit exceeded

      Good afternoon...trust you're good. I've been having issues working with but it's not responding. it's showing Limit exceeded, sorry it seems like too many people are working on the sheet right now please try again later. meanwhile no one is working on

    • Converting Sales Order to Invoice via API; Problem with decimal places tax

      We are having problems converting a Sales Order to an Invoice via API Call. The cause of the issue is, that the Tax value in a Sales Order is sometimes calculated with up to 16 decimal places (e.g. 0.8730000000000001). The max decimal places allowed in

    • Sorting columns in Zoho Projects

      Hi, In project management best practice, sorting columns (ascending, descending) is an important tool. Sorting dates to see the order of tasks starting, sorting on priority or even on planned hours is a must for an efficient project control. Currently,

    • Business Continuity - Disaster Recovery

      I know about the Zoho CRM backup .zip files, however, this doesn't include any of the infrastructure with like custom fields or custom modules. I am curious on what everyone has in place for a true backup or what your plan is if your Zoho instance were

    • Upload API

      I'm trying to use the Upload API to upload some images and attach them to comments (https://desk.zoho.com/DeskAPIDocument#Uploads#Uploads_Uploadfile) - however I can only ever get a 401 or bad request back. I'm using an OAuth token with the Desk.tickets.ALL

    • Losing description after merging tickets

      Hello, We merge tickets when they are about the same topic from the same client. It happens sometimes. We recently noticed that after the merger only the description from the master ticket is left in a thread. And the slave-ticket description is erased.

    • Option to Empty Entire Mailbox or Folder in Zoho Mail

      Hello Zoho Mail Team, How are you? We would like to request an enhancement to Zoho Mail that would allow administrators and users to quickly clear out entire folders or mailboxes, including shared mailboxes. Current Limitation: At present, Zoho Mail only

    • update linked contacts when update happens in account

      Hi, I have a custom field called Licence in the Accounts module. When someone buys a licence, I’d like to update a custom field in the related Contacts. How can I achieve this? I noticed that workflows triggered on Accounts only allow me to update fields

    • Problem Management Module

      I am looking for a Problem Management module within Zoho Desk. I saw in some training videos that this is available, and some even provided an annual price for it. I want an official confirmation on whether this is indeed available. This is not a particularly

    • Deluge sendmail in Zoho Desk schedule can't send email from a verified email address

      I am trying to add a scheduled action with ZDesk using a Deluge function that sends a weekly email to specific ticket client contacts I've already verified the email address for use in ZDesk, but sendmail won't allow it in its "from:" clause. I've attached

    • Unable to explore desk.zoho.com

      Greetings, I have an account with zoho which already has a survey subscription. I would like to explore desk.zoho.com, but when I visit it while logged in (https://desk.zoho.com/agent?action=CreatePortal) I just get a blank page. I have tried different

    • Offline support for mobile app

      Accessing your files and folders from your mobile devices is now quicker and simpler, thanks to the power of offline support. Whether on an Android or iOS device, you can use the Offline function to save files and folders, so you can review them even

    • Zoho Desk KB article embedded on another site.

      We embed KB articles from Zoho Desk on another site (our application). When opening the article in a new tab, there is no issue, but if we choose lightbox, we are getting an error "To protect your security, help.ourdomain.com will not allow Firefox to

    • Transitioning to API Credits in Zoho Desk

      At Zoho Desk, we’re always looking for ways to help keep your business operations running smoothly. This includes empowering teams that rely on APIs for essential integrations, functions and extensions. We’ve reimagined how API usage is measured to give

    • List of packaged components and if they are upgradable

      Hello, In reference to the article Components and Packaging in Zoho Vertical Studio, can you provide an overview of what these are. Can you also please provide a list of of components that are considered Packaged and also whether they are Upgradable?

    • Does Attari Messaging app have Bot option and APIB

      Hi, Does Attari also have Bot and API as we use in WhatsApp??

    • How to add application logo

      I'm creating an application which i do not want it to show my organization logo so i have changed the setting but i cannot find where to upload/select the logo i wish to use for my application. I have seen something online about using Deluge and writing

    • Introducing Keyboard Shortcuts for Zoho CRM

      Dear Customers, We're happy to introduce keyboard shortcuts for Zoho CRM features! Until now, you might have been navigating to modules manually using the mouse, and at times, it could be tedious, especially when you had to search for specific modules

    • Empowered Custom Views: Cross-Module Criteria Now Supported in Zoho CRM

      Hello everyone, We’re excited to introduce cross-module criteria support in custom views! Custom views provide personalized perspectives on your data and that you can save for future use. You can share these views with all users or specific individuals

    • Send Automated WhatsApp Messages and Leverage the Improved WhatsApp Templates

      Greetings, I hope all of you are doing well. We're excited to announce a major upgrade to Bigin's WhatsApp integration that brings more flexibility, interactivity, and automation to your customer messaging. WhatsApp message automation You can now use

    • Verifying Zoho Mail Functionality After Switching DNS from Cloudflare to Hosting Provider

      I initially configured my domain's (https://roblaxmod.com/) email with Zoho Mail while using Cloudflare to manage my DNS records (MX, SPF, etc.). All services were working correctly. Recently, I have removed my site from Cloudflare and switched my domain's

    • Zoho Analytics Regex Support

      When can we expect full regex support in Zoho Analytics SQL such as REGEXP_REPLACE? Sometimes I need to clean the data and using regex functions is the easiest way to achieve this.

    • Change of Blog Author

      Hi, I am creating the blog post on behalf of my colleague. When I publish the post, it is showing my name as author of the post which is not intended and needs to be changed to my colleague's name. How can I change the name of the author in the blogs?? Thanks, Ramanan

    • Show Attachments in the customer portal

      Hi, is it possible to show the Attachments list in the portal for the particular module? Bests.

    • Zoho CRM Formula - Current Time minus Date/Time field

      Hello, I am trying to prevent duplicate emails going to clients when more than 1 deal is being updated. To do this, I would like to create a formula to identify if a date/time field is >= 2 hours ago. Can someone please help me write this formula? Example:

    • Does Zoho Docs have a Line Number function ?

      Hi, when collaborating with coding tasks, I need an online real time share document that shows line numbers. Does Zoho's docs offer this feature ? If yes, how can I show them ? Regards, Frank

    • Feature Request - Insert URL Links in Folders

      I would love to see the ability to create simple URL links with titles in WorkDrive. or perhaps a WorkDrive extension to allow it. Example use case: A team is working on a project and there is project folder in WordDrive. The team uses LucidChart to create

    • Next Page