Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Transferring domain registration to new registrar and switching email hosting at the same time?

      I need to transfer an existing domain uv cure adhesive that's currently with SiteGround to Porkbun. I also need to move the existing custom email addresses from SiteGround to Zoho Mail. I'm not sure if I should transfer the domain first and then tackle

    • Split deposits

      Can Zoho do split deposits. One deposit, two checks for two separate invoices from different customers. This is one of the most common tasks I can imaging. When I mark the two invoices paid, there are two deposits in bank register. When I try to match,

    • Deactivate Desk Contact without Deleting Contat

      We have a client who has multiple tenants for regulatory purposes, and as such, has a few users that have email addresses in both tenants. They've then emailed into the ticketing system, so we have multiple contacts (no big deal, we want to keep their

    • Delete my store of Zoho commerce

      Hi Team, I want to delete my stores of commerce. Please help me asap. Looking for the positive response soon. Thanks Shubham Chauhan Mob: +91-9761872650

    • Ability to add VAT to Retainer Invoices

      Hello, I've had a telephone conversation a month ago with Dinesh on this topic and my request to allow for the addition of VAT on Retainer Invoices.  It's currently not possible to add VAT to Retainer Invoices and it was mutually agreed that there is absolutely no reason why there shouldn't be, especially as TAX LAW makes VAT mandatory on each invoice in Europe!   So basically, what i'm saying is that if you don't allow us to add VAT to Retainer Invoices, than the whole Retainer Invoices becomes

    • [Free Webinar] Learning Table Series - Zoho Creator for Asset Management with AI Enhancements

      Hello Everyone! We’re excited to invite you to another edition of Learning Table Series, where we showcase how Zoho Creator empowers industries with innovative and automated solutions. Each month highlights a specific sector, and this time our focus is

    • Menu Building is completely broken

      I have been 3 hours, I have not been able to edit the menu. Either it is completely broken, very little intuitive or I do now know anything... There is no way to create a megamenu, no way to create a menu. Despite the fact I go to menu configurartion

    • Can you sell Subscriptions using Zoho Commerce?

      In addition to physical products and the apparently coming soon 'Digital Products', it is possible to sell Subscriptions using Zoho Commerce?

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Multiple Languages for Product Names

      Hi, I use 2 languages: spanish and english. I want to have for every product a name in spanish and a name on english. I want to have to possibility of choosing one of these languages when making an invoice or a purchase order. Is there any way to do

    • Item with name in different languate

      Hello, is there a way to have an item with its name in different languages? For example: I sell an item in different markets and I'd like to have a Proposal and the Invoice with the Item Name in a specific language. Rino Bertolotto Zoho Specialist, STESA srl

    • Contacts with most tickets? Alarm for multiple tickets?

      Is it possible to see through the analytics/reports which contacts are creating the most tickets (not the most discussed ones)? Also, is there a way to set up a notification if a contact creates multiple tickets within a certain time frame?

    • Issue with Template Subject Line Format in Zoho CRM

      Hi Team, I’ve noticed that when I update the subject line of an email template in Zoho CRM, it sometimes appears in an incorrect format when used. Please see the attached screenshot for reference. Kindly look into this issue and fix this issue from backend

    • Two Data Labels in Bar Chart

      I need to create a bar chart that has both the SUM and COUNT. I've concatenated them into a formula but it converts it into a stacked bar / scattered chart. The bar chart is no longer accessible. Since i'm comparing YOY, it would be best to have it in

    • Disable field on subform row

      Hi, Is it currently possible to disable a row item on a subform? I was just trying to do something whereby until another value is entered the field is disable but for the deluge scripting interface threw up an error saying such a function is not supported on a subform. Thanks in advance for your help. Shaheed

    • Leads - Kanban view fit to screen

      Hey guys, I created a custom layout for my leads, staged by lead status. I have 10 types of status. In Kanban view I see only 4 columns/stages and need to scroll to the right to see the rest. Is there a way to make columns/stages be displayed all together?

    • Request to Differentiate Auto-Closed WhatsApp Conversations in SalesIQ

      Hi Zoho Support, I’d like to raise a request related to the way WhatsApp conversations are auto-closed in SalesIQ. Every Monday, our Sales team has to manually review each closed WhatsApp conversation from the weekend to identify which ones were automatically

    • Kanban View UI gets a revamp

      Hello everyone, In the coming week you will notice design related enhancements in Kanban View. The UI has been changed and a new option is introduced under Kanban View Settings that allows to change the color of the category headers.  Please, note that the functionality is not changed. These changes will not apply to the Activities and Visits modules. Here are the details of the changes: 1. The column widths have been fixed to 300 px. The records will have a box around them for clear distinction.

    • Can you stop Custom View Cadences from un-enrolling leads?

      I'm testing Cadences for lead nurture. I have set un-enroll properties to trigger on email bounce/unsubscribe, and do NOT have a view criteria un-enroll trigger. However, help documents say that emails are automatically un-enrolled from a Cadence when

    • Issue with Anchor Link on Zoho Landing Page (Mobile/Tablet View)

      Hi Team, I have created a landing page using Zoho Landing Page and added an anchor link to it. The anchor link is working fine on desktop view; however, it does not work properly on mobile or tablet view. I’ve tried debugging this issue in multiple ways,

    • Simplest way to convert XML to a map?

      I've reviewed the help info and some great posts on the forum here by Stephen Rhyne (srhyne). At the moment I'm using XPath to generate a list of xml nodes, iterating through that to fetch the field name/value pairs and adding them to a map (one map for each record in the data). I then convert the row map to a string and add it to a list. Here's the function: list xml.getRecordListFromXML(string xml_data, string ele_name) {     result = List();     // get list of record nodes     rec_list = input.xml_data.toXML().executeXPath("//"

    • Introducing Creator Simplified: An exclusive learning series to enhance your app development skills

      Hey Creators! Welcome to Zoho Creator's new learning series, Creator Simplified. In this series, we'll dive into real-world business use cases and explore how to translate your requirements into solutions in your Creator application. You can also expect

    • [Product update] Updated Data Synchronization Process for QuickBooks - Zoho Analytics Integration.

      Dear QuickBooks integration users, We’re making an important update in the way data is currently synced in your QuickBooks integration within Analytics workspace. What’s changing: Previously, with every data synchronization, Zoho Analytics used to fetch

    • Zoho CRM new calander format cannot strikethrough completed task

      Hi, Recently there is a new format for calendar within Zoho CRM However, found out that a completed task will not cross out or strikethrough like previous format. Without strikethrough, it will be difficult to identify which task is still in Open status.

    • How to edit form layout for extension

      I am working on extension development. I have created all the fields. I want to rearrange the layout in Sigma platform. But there is no layout module in Sigma. How can I achieve this for extensions other than Zet CLI and putting the fields into widget

    • Employees not Users

      Hello, We are a construction company that has +180 employees and most of them are in remote location working onsite with no access to internet. Is it possible that we have data stored for all employees but have only 5-10 users who will be in charge of entering employees data? or do we have to pay for all +180 employees? even though they won't be using the system?

    • Zoho people generatimg pdf

      Hello , now i want to make a customm button in zoho people that is inside a deduction module , that fetches all the records and generate a pdf with a template that i have done in the mail merges template , i was told that i have to upload template on

    • Ability to Filter Alias Mailboxes in Zoho Recruit

      Dear Zoho Recruit Team, I hope you are doing well. We would like to request a feature enhancement regarding the handling of alias mailboxes in Zoho Recruit. Currently, when we connect an alias mailbox (e.g., jobs@domain.com) from our Zoho One account

    • zohorecruit.com career form postcode bug

      Dear, When I select a postcode from the drop down on a zohorecruit.com career form, the street text field is automatically filled with the name of the city, which should not happen. Any idea how I can fix this? Thanks, Bart

    • Office-365-agenda and Microsoft Teams Integration

      Dear, I have a trial version of Zoho Recruit and trying to evaluate the Microsoft Teams Integration in Zoho Recruit. After registering with my Office 365 account and checking the result of the registration/sign-in at https://mysignins.microsoft.com/ (which

    • Delegate Access - Mobile iOS/iPad

      We’re over the moon that delegate access is now available in Zoho Mail as we were nearly ready to switch platforms because of it! Is there a timeline on when delegate mailboxes will be accessible from the iOS and iPad OS applications? Thanks, Jake

    • How to add Connector in developer platform zoho?

      Hi, I am working on creating an Extension, and part of the development is to retrieve Email templates. In my CRM instance I can invokeURL by creating Zoho OAuth connection and get the template. But developer platform does not provide Zoho OAuth or any

    • How to archive Lost/Junk Leads so sales reps don’t see them, but keep them for reporting?

      Hi everyone, In our Zoho CRM we have two Lead Status values: Lost Lead and Junk Lead. What I want to achieve is: When a lead is marked as Lost or Junk, it should disappear from my sales reps’ Lead views (so they only see active leads). At the same time,

    • Zoho CRM Canvas Copy Original Layout

      Hello all, I want to use Canvas to make small changes to certain views, not to make huge changes. Is it possible to copy the original Zoho layout and set-up and start from there? I checked and all I can find are some templates which are far from the original

    • Revenue Management: #5 Revenue Recognition in SaaS

      If you're building or running a SaaS business, you've probably encountered this. You get paid upfront for a subscription and a one-time onboarding fee, but you end up with confusion about when to consider it revenue. Can I book all of it now? Should I

    • MS Teams for daily call operations

      Hello all, Our most anticipated and crucial update is finally here! Organizations using Microsoft Teams phone system can now integrate it effectively with Zoho CRM for tasks like dialling numbers and logging calls. We are enhancing our MS Teams functionality

    • Zoho Learn Course Access Issue

      One of the learners in a specific course can't see any lessons. They are registered as both a user and learner for this course in Zoo Learn. What could be the reason?

    • ZOHOLICS Japan 2025 開催のお知らせ（再投稿）

      【コミュニティユーザーの皆さまへお知らせ】 Zoho 最大のユーザーイベント「ZOHOLICS Japan 2025」を9月19日（金）に開催します。 AI活用に関する特別講演、ユーザー事例、Zoho 製品の活用例のご紹介など、Zoholicsならではのセッションをご用意しています。 Zoho コミュニティ開催のMeetupとはまた違った雰囲気のイベントです。 ご都合のつく方はお気軽にご参加ください✨ 詳細はこちら https://events.zoho.jp/zoholics2025#/?affl=forumpost2

    • Phone Number format for Bulk Upload via csv for Zoho Sign

      What is the phone number format that we need to use for a bulk recipient upload via csv in zoho sign, should the country code be included for ex if its US should it be +18889007865 ior 18889007865 or without country code 8889007865? the sample csv provided

    • Filter Page Elements By Selectable Date Range

      I have created a basic Page that will serve as a client dashboard with elements that will provide simple counts & sums of data in reports. There are no reports or forms added to the page, just elements. It currently functions as needed, but shows the

    • Next Page