Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Has anyone verified if Zoho is PCI compliant?

      We are planning on using Zoho to process payments via Authorize.net. We have everything set up and are attempting to complete the PCI DSS SAQ-A requirement for our merchant account. This requires us to prove Zoho has completed the SAQ-D for Service Providers. We need a way to verify compliance, or a copy of an attestation of compliance signed by the appropriate officer at Zoho. I assume I'm not the first person to use Zoho to process payment, and therefore not the first to require this information

    • Bigin Plugin for Outlook

      Could we get this added? The Gmail version already exists, and I would like to avoid having to make a switch.

    • Date does not fit the field

      Hi There. I am having fun learning zoho sign API. Today I noticed the "Signed Date" field does not fit, or alternatively the font is to large for the auto field space. See screenshot below. The signed date field is created by putting {{Signdate}} on the

    • Tip of the Week #69 – Automate your Zoho TeamInbox tasks with n8n integration.

      Don’t waste time repeating the same tasks—like sending follow-up emails or adding new contacts. Let automation save the day. With n8n, an open-source automation tool, you can connect your favorite apps and let them handle the busywork for you. You don’t

    • Multi Page/Step Forms in creator

      Greetings i was wondering if it's possible to create multipage/step forms on creator similar to what we have on zoho forms. is that possbile? Thanks

    • Package Geometry

      how can i add the dimensions and weight capacity of the available boxes to be default in the system everytime we use it ?

    • How to create a Master Kanban Board that syncs with Child Projects?

      Hello, We're currently using Zoho Sprints for managing our interdepartmental teams, and we're looking to enhance our workflow using Kanban boards as part of a company-wide productivity improvement initiative. Our goal is to implement a project structure

    • Writer.. Broken?

      Hello,  Writer has been really good to me during the months I've used it, up until now.  I usually launch the app by tapping the icon and I could immediately pick up where I left off.  Now I'm greeted by a loading circle not reaching 100% and I only have the option to create a new account.  By pressing that button it now switches to a login screen and I can access my account. However, it seems (only speculating ofc) to be stuck in cell-phone mode? everything looks scrambled.  I can't access any of

    • How to access Recruit Variables in a Deluge function?

      I have set up Recruit Variables in Zoho Recruit, and I would like to know how to retrieve these variables from within a Recruit custom function (Deluge). Could someone please explain the correct way to access them? I tried the following code, but it did

    • Upon De activate a user what name doe sthe contacts candidates go under?

      When deactivating a user, does the user name remain the same, as the candidate owner? If not what/who, does it change to? Do I need to change the user name in contacts and candidates before I deactivate the user?

    • Weekly Tips: Customize alerts from your Priority Users

      You might receive hundreds of emails daily, but messages from your manager, clients, or team leads often require immediate attention, as they may contain urgent requests or critical updates. How would you ensure you never miss important messages from

    • Maximum 100 records in Sheet View is limiting. How can I increase this?

      Thanks in advance for any help with this. There was a similar post that showed answered but it did not help with increasing the number of records you see in a Sheet View. Editing in the Sheet View is fast and efficient but I have 3500 records and I need

    • Revenue Management: #3 Revenue Recognition Simplified

      In continuation of the previous post on how to compute revenue recognition, let's explore a solution that helps businesses handle real-world complexities. While the Accounting Standards provide a clear framework for recognizing revenue, the real challenge

    • Tip #40- Strengthen Remote Support with IP-based Restrictions in Zoho Assist– ‘Insider Insights’

      Protecting sensitive data and preventing unauthorized access is a top priority for any organization. With IP-based restrictions in Zoho Assist, you can ensure that only users from trusted networks can initiate remote support sessions. Say your IT team

    • Push Invoices to Xero Manually

      Hi guys, I'm wondering if anyone has wanted to do this and has a workaround or knows of an app that may be able to help with this. I sell B2B and B2C. The customers can purchase on our website or through marketplace, all of which send sales to zoho. The

    • OpenAI error code: 1010 in a Zobot

      Please see short linked screen recording. Insights welcome. Please and thank you! https://workdrive.zohoexternal.com/external/f3247ba9c872639157b707700c0300c433c7664aea924a034f4da3c3ad2e355f

    • Ability to Create Sub-Modules in Zoho CRM

      We believe there needs to be a better, more native way to manage related records in Zoho CRM without creating clutter. Ideally, Zoho would support "sub-modules" that we can create and associate under a parent module. Our use case: We have a custom module

    • Installing EMAIL Setup in New Domain

      Respected Support team, I'm facing an issue with cloudflare in Pakistan, I want to setup Zoho Mail Setup but I Don't know how to enable Zoho mail setup without cloudflare. My Website https://stumbleguymod.com/ is using CF, and I want a different Zoho

    • Signature change

      I cannot see how to change signature or out of office details easily now in the new format.

    • Inventory API - Retrieve all uploaded product / item images

      I know that I can get the primary image for each product / item or composite item, by using the /image endpoint.  https://inventory.zoho.com/api/v1/compositeitems/<item-id>/image?authtoken=<TOKEN> This will return only one photo, even if the item has multiple images uploaded. Is there a way to retrieve all images stored for an item via the Zoho Inventory API?

    • Ebay Integration malfunction

      My eBay integration in Inventory has always worked well. It suddenly malfunctioned. It is creating its own parts in Inventory that are unavailable instead of selling the parts I've always sold. Tech help was unable to resolve this. The latest sale attempt

    • Introducing Bin Locations In Zoho Inventory

      Hello users, We are excited to let you know that your wait for the Bin Locations feature has now come to an end! Yes, you heard us right! We are here to introduce the much-awaited Bin Locations now in Zoho Inventory. But before we dive into the feature

    • how to get all the records in the custom View more than 200 records , Without using the page Concept

      how to get all the records in the custom View more than 200 records , Without giveing page as default in the Loop Concept Pls help how We can Achive this void schedule.Lead_Attempt_To_contact_schedule_10_30() { pages = {1,2}; for each pg in pages { query_map

    • The way that Users can view the ticket

      I have created users. What I would like to achieve is the following: All users under the same company account should be able to view each other’s tickets.

    • Zoho UAE SMS/WHATSAPP

      Hello everyone, so I have a question as regards DC and their impact on automation, integration and app usage. For example I am working with a UAE clientniw but each time I tried to connect their WhatsApp and sms then automate their process I tend to receive

    • Looking to Flag or Tag contacts/ accounts on Zoho Desk?

      I am looking for a way to flag certain accounts and make it obvious on the views pages. So for example if a has a certain package or needs extra attention it is clear before even clicking on the ticket. This could be via adding a tag or flag onto an account,

    • setting date-time field from string

      hello everyone, i hope someone could help me. i have a date-time field in a form that i want to fill in from two separate fields of date, and time. i need to combine the two fields to a one date-time field but can make it work. i tried to convert the

    • Calendars and CRM Contacts

      I'm finding having multiple calendars in Zoho One so confusing. I have a few questions so I can get this straight. We have a meeting room that we have set up as a resource in Calendar. Can this be set up in Bookings and the CRM Calendar? Using Zoho Calendar,

    • Announcing new features in Trident for Mac (v1.23.0)

      Hello everyone! Trident for macOS (v.1.23.0) is here with interesting features and thoughtful enhancements to elevate your workplace communication and productivity. Here's a quick look at what's new. Record your meetings. You can now record audio and

    • Applying a record template

      Hi all, I can't figure this out. I hope you can help. The scenario: We have learners who have to complete a 'digital' journal with tasks in order to qualify. Those tasks, once completed, need a final signature from their 'Mentor', which will trigger their

    • Quickbooks invoice with Zoho Creator

      Is it possible to push data from Zoho Creator directly to an invoice on QuickBooks? If so, where can I find information on how to do this?

    • Help: Capture full page URL in hidden field when same Zoho Form is embedded on multiple pages (iframe)

      Hi all, Goal Use one Zoho Form across multiple pages and record the exact page URL (incl. subdomain + path + hash) where the user submitted it. Example pages: https://www.example.com/cargo/ https://www.example.com/cargo/containers/#contact https://cargo.example.com/auto/

    • Automatically Populate fields - HELP!

      There have been many discussions on this but I still can't seem to get it to work for me. I am trying to create a lookup field and have other fields automatically populate. Based on the instructions in the Help Center, I should be using the "on user input". It's just not working, here is the layout...   Both forms are in the same application. Current form is called Add Note, form to fetch records from is called Add Client. Lookup field is called Select_Client_ID related field in fetch form is called

    • Push notifications to portal users

      Hi all, it is possible to send push notifications to portal users?

    • Rename the attachment from record template pdf in the sendmail deluge script

      Hello Zoho, I urgently need a feature to rename record templates that I send via the sendmail feature. The program I created sends emails to clients with an invoices that have been created in the invoice database. If the user selects 3 invoice numbers,

    • Alt Text On Multiple Images

      I'm using Zoho Social to post to a charity website. Often the posts have multiple images, but it seems there is only one field for Alt Text. Does that mean I can only include it for the first image? Or is there a way to add alt text for all the imag

    • [SOLVED] Getting 401 when trying to download ticket attachment via API

      I'm able to use the API just fine to access ticket content. But I cannot download ticket attachment, keep getting 401 Client Error. Example: https://desk.zoho.com.au/api/v1/tickets/{ticket_id}/attachments/{attachment_id}/content?orgId={org_id} For headers,

    • Zoho project – Workdrive integration.

      Hello everyone, I was wondering something, we did the Zoho projects integration with Zoho Workdrive but nowhere during this integration we could set the location of the folders that would be automatically created in Workdrive. As I understand it, it creates

    • Scriptを埋め込みたくてOn User Inputを探しているのですが・・・

      編集モードで、Scriptを埋め込みたい項目を選択し、「項目のプロパティ」パネルで、その下のほうに「フィールドアクション（Field Actions）」という見出しがあると聞いたのですが、そもそも、それが見つかりません。そのために、On User Inputなどのイベントが選べません。 画面の英語を日本語に訳しているためにわけわからん状態になっているのかも知れませんが、わかる方、いらっしゃいますか？

    • Problem with cloud query exceeded

      When making a call I get this error, It is associated with a function in node that calls external APIs This is the code //tokenConsultar = thisapp.ObtenerToken(); //input.token = tokenConsultar.get("output").toMap().get("token");

    • Next Page