Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • How to create auto populate field based on custom module in Zoho CRM?

      Hello, i'm still new to Zoho CRM and work as administrator in my company. Currently, I'm configuring layout for Quotes Module. So, the idea is, I've created a read-only field in Quotes called "Spec". I want this field automatically filled with Specification

    • Rich Text For Notes in Zoho CRM

      Hello everyone, As you know, notes are essential for recording information and ensuring smooth communication across your records. With our latest update, you can now use Rich Text formatting to organize and structure your notes more efficiently. By using

    • Change Last Name to not required in Leads

      I would like to upload 500 target companies as leads but I don't yet have contact people for them. Can you enable the option for me to turn this requirement off to need a Second Name? Moderation update (10-Jun-23): As we explore potential solutions for

    • Office 365 and CRM mail integration: permission required

      Has anyone run into this weird problem? My email server is Office 365. When I try to configure Zoho CRM to use this server, a Microsoft popup window opens requesting user and password. After entering that, I get a message in the Microsoft window saying

    • Tables improvement ideas / features

      Heya, I've been using Zoho Tables for a few months now and wanted to post some features that I think will be greatly beneficial for the tool: 1. Ability to delete a record in automation or move a record in automation. - Usecase: I move a record from one

    • Deluge - Can't get phone number SalesIQ

      Hey folks, I’m building a custom plug for SalesIQ that’s supposed to register leads into Zoho CRM. The SalesIQ chat is being implemented on WhatsApp, and in my plug I’m using this line: mobile_clean = session.get("phone").get("value"); From what I understand,

    • Zoho Desk - Community

      As a regular user of Zoho Cares Community I would really love to see the publish date of articles. For example, when I look at Announcements, it would be very beneficial to see which ones were posted recently, over those which have just and a recent comment.

    • access to quartz for my customers

      Hi how can I have access to the application quartz you use for us to send you screen rocording, this feature would be immensely useful for our customers support https://quartz.zoho.com/

    • Issue with Inline Images in Email Reply via Zoho Desk API

      Hi, I am attempting to send inline images in an email reply using the Zoho Desk API, but the images are not being displayed inline for the recipient. I have followed this documentation: https://desk.zoho.com/DeskAPIDocument#Uploads https://desk.zoho.com/DeskAPIDocument#Threads#Threads_SendEmailReply

    • How to search a value stored in a subform?

      Hello, We store serial numbers in subforms but now we would like to be able to search the values to be able to easily find the record with the serial number. I saw that it's not possible to search such values through global search but is it possible to do it an other way? Thank you,

    • Field Dependency Not Working on Detail Page in Zoho Desk

      Hi Support Team, I’ve created field dependencies between two fields in Zoho Desk, and they are working correctly on the Create and Edit layouts. However, on the Detail page, the fields are not displaying according to the dependencies I’ve set — they appear

    • How do the keyword critera work?

      Hi, I'm working on automated assignment of tickets based on keywords. How does this feature work? Where does this criteria look for keywords - email address, subject, email body? Can you please clarify this as I want to avoid overlapping with criteria

    • Error: Unsupported content type: text/html;charset=UTF-8 after tryeing to get the token for n8n automation

      I am working on ZOHO Desk automation and need to get the ZOHO auth token for n8n I have created the app in ZOHO Desk API, got client id and client secret. Added all data required to get a token in n8n. After I sign in with my ZOHO credentials in ZOHO

    • Improving Collaboration Features in Zoho Portal

      Hello Zoho Community, I’ve recently started exploring Zoho services and I’m really impressed with the wide range of features. However, I feel there is still room for improvement in the collaboration area. For example, it would be really helpful if we

    • Automated Shopify Emails Not Being Delivered

      I have an ecommerce store with Shopify. I recently set up my email to be served through Zoho. Since doing this, customers are not receiving some of our automated emails from Shopify itself. Our initial email that confirms their purchase goes through but our Shipping Notification that is automatically sent out upon fulfillment is not going through. Sometimes we get a notice that it's been classified as spam, sometimes nothing. I can send/receive email via Outlook on my desktop and I can send/receive

    • I can't log in to my account on Thunderbird

      I've just had to rebuild my PC (calamitous mess from Microsoft with Win10/Win 11 'upgrade' - they confirmed I had to start with a new build). I have used Zoho mail for years via Mozilla Thunderbird, but now I've had to download the latest version of TBird,

    • Send a campaign to one recipient.

      Very often I speak to a customer and they say they didn't see my email (maybe it went in Junk, maybe they deleted it). Anyway, I just want to go into the Campaign and send it to one person. You already have a feature very close to this - when sending a Test. While developing a campaign, I can send tests to anyone. Why can't we have this AFTER  the campaign has been sent? I know, there's a caveat, and that's in the use of merge tags. Most of the time I only use FNAME, but maybe you could check if

    • Try FSM again for our business

      We already have our customers individual equipment in CRM with serial numbers, install dates, warranty length and importantly next service which is generally 2 years. a month before the service date is due we get get a report and send out service reminders.

    • Use Zoho Books to bill for work done in Zoho Desk??

      I'm trying to see if something is possible (and if yes, how). We use Zoho One to manage our business. We have a lot of clients that will put in a ticket (via portal) to have work done. Out techs will pick up the ticket, do the work, and then log the time

    • Get Cliq Meetings in my O365 calendar

      Hi, we are currently evaluating to replace the Teams Messaging and Meetings with Cliq. We currently still have all our email and calendars in O365. What i want to achieve is, to create a (ZOHO) meeting from Cliq and have this meeting added to my Outlook/O365

    • Issue with Zoho Help Portal – Tickets Missing or Not Answered

      Hi, How are you? I think there may be an issue with the Zoho Help Portal. I opened a few tickets directly in the help portal a some time ago but never received any response I also opened ticket 148356451 by email. I did receive a reply to it, but the

    • DUNS & Bradstreet and Credit risk monitoring integration with Zoho books

      Small businesses not being paid by bigger clients and clients of all sizes is a huge problem. It will be nice if Zoho develops integration with DUNS & Bradstreet(D&B) and Credit risk monitoring integration with Zoho books. That have small businesses can

    • Zoho Forms - Form Rules based on attachment fields

      Many businesses use forms to collect documents and images from customers. In many cases, you may want to trigger a notification or other automation based on whether or not an attachment was added. I've noticed that attachment fields do not appear in Rules

    • Add multiple users to a task

      When I´m assigning a task it is almost always related to more than one person. Practical situation: When a client request some improvement the related department opens the task with the situation and people related to it as the client itself, the salesman

    • Can't open draft email for editing

      Last night I started composing an email and I let it save in drafts. This morning I want to continue working on the email. It is in my Drafts folder but it will not open. The only option there is to delete it. This is not the first time it has happened. On previous occasions I have just deleted the draft and started afresh, but I really want this one back. Windows 10 with Pale Moon 28.10.0 browser.

    • Open filtered deals from campaign

      Do you think a feature like this would be feasible? Say you are seeing campaign "XYZ" in CRM. The campaign has a related list of deals. If you want to see the related deals in a deal view, you should navigate to the Deals module, open the campaign filter,

    • Change scheduling emails time

      When sending an individual email there is a great feature to schedule them to send later. I could only use the one time that is suggested. Is there a way to select another time? Regards, Glenn

    • Zoho CRM: how can I control which contacts to sync with Outlook?

      I was just playing around syncing contacts from Zoho to MS Outlook (MS365 account.) The problem is our firm has hundreds of thousands of contacts and I don't want to bury my contacts list in outlook. Any help with this is greatly appreciated.

    • How to overcome limitations in meetings

      As a company, one of our deliverables is a meeting between two other companies, where we act as facilitators. So, if we recorded this meeting  in Zoho CRM, it should be connected to 2 accounts, 2 contacts, and 1 campaign (a campaign, in our use, is the

    • Different MRP / Pricing for same product but different batches

      We often face the following situations where MRP of a particular product changes on every purchase and hence we have to charge the customer accordingly. This can't be solved by Batch tracking as of now so far as I understand Zoho. How do you manage it as of now? 

    • Add a 'Log a Call' link to three dot icon in Canvas

      Hi, There's a three dot element when creating a canvas called 'More'. I would like to modify this to add a link that says 'Log a Call' in order to quickly record the details of a cellphone call. I'd also like this to be a simple 'contact' selection and

    • Syncing Zoho Forms with Bigin - Embedding issue?

      Hello everyone, I created a Zoho Form for a page on my GoDaddy website to collect leads, which then transfers the data to Bigin. However, I'm facing an issue where it doesn't seem to work properly. I've integrated Zoho Forms with Bigin and tried embedding

    • Can not add fields to a Section

      I feel like I'm missing something obvious: I can add new Sections to my form but I can not add fields to the Sections. I've tried fields already on the form as well as dragging and dropping new fields into the Section but nothing will go into it. What

    • Record Logged in User while using CRM lookup field

      Is it possible, while using the Zoho CRM lookup field, to automatically use the user account logged into Zoho CRM in a hidden field? I was hoping to add employee accounts to my current plan. But would like a record on the Form submission of who submitted

    • Form Rules for Suburb Categories to alternate landing pages or Making a Fields Contents ALL CAPS

      I need to send differentform submissions to two to three different thank-you URLs (for Meta/Google pixels) depending on which suburb a user selects in a form. I have ~400 suburbs split into two categories (A and B, based on business value). Current challenges:

    • Collaps Notes

      There are times when long/large notes are added to a record i.e. Accounts or Deals etc. Currently, the full note is displayed in the notes related list section. It would be great if by default only 5 to 10 rows of the note are displayed when the note

    • Zoho Down

      I have a drop in my Zoho One services.

    • Runing RPA Agents on Headless Windows 11 Machines

      Has anyone tried this? Anything to be aware of regarding screen resolution?

    • Problem for EU users connecting Zoho CRM through Google Ads for Enhanced conversions

      Has anyone else experienced this problem when trying to connect Zoho CRM through Google Ads interface to setup enhanced conversions? Did you guys get it fixed somehow? The Problem: The current Google Ads integration is hardcoded to use Zoho's US authentication

    • Why am I getting event Pop-up Notification for events that have been cancelled?

      Why is Calendar Notification still popping up for events that have been cancelled or changed? Each time events are cancelled or changed, I have observed that I am still getting notifications for them. Below is a sample pop-up notification for one of the

    • Next Page