Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • Proposal for Creating a Unique "Address" Entity in Zoho FSM

      The "Address" entity is one of the most critical components for a service-oriented company. While homeowners may change and servicing companies may vary, the address itself remains constant. This constancy is essential for subsequent services, as it provides

    • I cannot find my older documents from 2024 and 2023

      I cannot find my older documents from 2024 and 2023.

    • System default SLA descriptions can't be modified

      The system default SLAs have identical descriptions for all SLA levels, but their settings differ. However, I am facing an issue where I cannot modify these descriptions and save the changes. The content of the description box can be edited but the changes

    • Customising Help Center

      Hi I don't think it is possible to add custom pages to help center? We'd like to use this as a customer portal with support tickets, FAQ/Guides, Billing and contracts. Is there any plans to add a feature like this or an alternative way to do it other

    • Replies sometimes creating separate ticket

      Sometimes when a customer responds to an email coming from Zoho Desk, instead of adding a reply to the original ticket, a separate ticket is created. This happens even though the response subject line contained the ticket number, and the person responding

    • Ticket Approvals - External Users

      The ticket approval option - we need to be able to select external users (Contacts) for approvals. Sometimes we are working with an end user and their boss needs to approve a purchase. For example, working with a cashier on a broken cash register and

    • Force Users to Ask Answer Bot a question... First

      End users will always skip talking to a bot. It would be nice if Zoho adopted the standard and forced users to first ask a question to answer bot (or zia in some fashion) and then pass to the agent if it wasn't answered. Options to force the user to speak

    • View Answer Bot conversations?

      We are trialing Zia and are experimenting with Answer Bot on our knowledge base. So far so good! Management asks me if it is possible to view Answer Bot conversations, the purpose being to look over its shoulder and confirm that it is working as des

    • Mass Email an Account

      It would be nice to mass email an account stating there is an outage at their business or something specific to an account is needed to be mass communicated. Even if it makes a ticket for every Email Out to every contact in the Account. Think: the customer

    • Multiple Topics assigned to a single Campaign

      Hello, is it possible to assign multiple Topics to a single Campaign? We frequently write a content to our subscribers that spans multiple Topics and we would like to send it to all Contacts that are subscribed to at least one of the Topics. But it looks

    • Zoho Projects Fonts and Accessibility missing

      I cannot find any more the tab where I can change the font in Zoho Project. I also checked the knowledgebase and there they have accessibility tab which I am completely missing. Is there some setup I am missing or is it a problem with our account?

    • Zoho Books Custom Widgets Deprecation Error

      I created a simple sample widget with zet and published it using sigma Both in the Sandbox and Production the Widgets are showing this error

    • Problems with PDF files in notebook

      I'm evaluating Zoho Notebook as an alternative to Evernote and imported my Evernote account to Zoho Notebook. First issue is that notes in Evernote that comprise a PDF are turned into a 'group' with a single note page (that has the text from the Evernote

    • API (v3) Tasks sorting issue

      We are using the v3 API for Projects. When we gat all tasks, per page of 100 tasks, we get the task info alright. But when we try to sort based on DESC(last_modified_time) we don't get the correct sort order. It is roughly sorted by the last_modified_time,

    • Assemblies make my stock go negative

      I am sure this is just the way that we are using this feature, but we use assemblies, a lot. The issue for us is the way that the relive inventory and the fact that it makes our composite item stock go negative. I have added flows to auto assemble and

    • User Activity Reports

      I'd like to get data related to user activity.  For example, Login and logout times, emails sent/received, new records created , etc. Is that currently available. I just can't seem to find anything . Thanks, Dave

    • Help: Populate “Contact Owner” details into Customer custom fields (for email templates) in Zoho Books

      We want to send invoices on behalf of our sales agents, and include the agent’s name, email, and phone in the email body using placeholders. Plan is to copy the Customer Owner details into three Customer custom fields, so they can be used as placeholders

    • Undocumented Books API error message - 1000 - The requested action could not be completed. Please try again. | Unexpected error

      This code sometimes throws this error 1000 - The requested action could not be completed. Please try again. | Unexpected error What does it mean? result = zoho.books.updateRecord("salesorders",organization.get("organization_id"),salesorder_id,sales_

    • Partial payments for retainer invoices

      When a customer does not pay the entire retainer invoice there is no way to apply a partial payment. PLEASE add this function.

    • Making Tags Mandatory

      When creating an expense, is it possible to make the Tags field mandatory?  I see the option in settings to make other fields mandatory, like Merchant, Description, Customer, etc, but nothing about Tags. Thanks! Kevin

    • Mass Update not trigger workflows

      Hi, I have performed a mass update of all records in a custom module using a custom view. I have a dummy checkbox on my module that I turn on or off - hoping to trigger all the new workflows that have been created for that module. However, no workflows

    • Multi-Select lookup field has reached its maximum??

      Hi there, I want to create a multi-select lookup field in a module but I can't select the model I want the relationship to be with from the list. From the help page on this I see that you can only create a max of 2 relationships per module? Is that true?

    • Generating CRM reports based on date moved in staged history

      Hi everyone, I'm trying to generate CRM reports of jobs (I think these are called usually deals) based on when they were moved to a particular stage, ie all jobs that were moved to Proposal/Quote in the previous financial year. I can see from other similar

    • Modules for missed calls, emails, texts etc

      Hi there. Is there a way to create a module that would automatically show a list of all inbound calls that were missed by our users, as well as any inbound SMS's, emails & WhatsApp messages. That way, a user who is available, can work through that list

    • Subforms and automation

      If a user updates a field how do we create an automation etc. We have a field for returned parts and i want to get an email when that field is ticked. How please as Zoho tells me no automation on subforms. The Reason- Why having waited for ever for FSM

    • Conditional layouts - support for multi-select picklists

      Hi, The documentation for conditional layouts says the following: "Layout Rules cannot be used on the following field types: Auto Number Lookup Multi Select Lookup User Lookup Formula File Upload Multi Line" I have a custom module with a multi-pick list

    • Ability to Set a General Email Signature for the Organization in Zoho Recruit

      Dear Zoho Recruit Team, I hope you're doing well. We would like to request a feature that would allow us to set a general email signature at the organizational level within Zoho Recruit. Currently it is only possible for individual users to create their

    • Custom Buttons & Links Now Supported in Portals

      We’ve just made portals in Zoho Recruit more powerful and customizable than ever! You can now bring the power of Custom Buttons and Links to your Candidate, Client, Vendor, and Custom Portals — enabling portal users to take direct action without recruiter

    • Is there a way to show contact emails in the Account?

      I know I can see the emails I have sent and received on a Contact detail view, but I want to be able to see all the emails that have been sent and received between all an Accounts Contacts on the Account Detail view. That way when I see the Account detail

    • Live webinar: Transform data into impactful visuals with Zoho Show

      Data is frequently a core part of a presentation, whether it’s a sales pitch, project update, research report, or performance review. But when it’s just numbers on a page, the message you’re trying to convey can get buried. The real magic happens when

    • File attachments not working - web version

      Since Notebook announced better file attachments, I have experienced file attachments failing. When uploaded from my pc to the web version, notebook just spins. Acting like the attachment is very large. When it's 30k. If I cancel out of this, the notecard

    • API - Barebones Data for Send Doc For Signiture

      I am learning how to use with Zoho Sign API. I am wondering if someone can give me a bare bones JSON data sample for "Send Document For Signature". Below is the blank data that is pre-populated in postman. Seems like there is more here than the bare minimum

    • Revenue Management: #2 How to compute revenue?

      In our previous post, we discussed the revenue recognition standards under IFRS 15 and ASC 606 and the five-step model. Now, let's examine three distinct business scenarios to see how the standards and model work in practice. This will help you understand

    • External lookup in CRM (as in Books)

      Hello Context: We have a CRM module, similar to Deals, but for Purchasing. Once the PO is created, a link to this PO should be updated in that "deal". In Books, you can add a custom 'external' field which can look up into CRM modules. I'm asked to have

    • "Undo Send" Feature

      Would love it if TeamInbox had an "Undo Send" feature, that gives you 10 seconds or so to "undo" the sending of an email. Many other email clients already have this feature, and my clients really miss it, as it has saved them many times in the past when

    • Zoho Sites

      Does anyone have experience building Zoho sites or know how I can find someone who does?

    • Zoho Crm Lagging

      Hi Zoho Support Team, Starting from today, my Zoho CRM has been extremely slow and laggy when accessing any pages or modules. This is affecting my work and overall productivity. Could you please help to check if there are any ongoing issues or if there’s

    • Is there a way to reference/attach mails to deals/contacts when the mails haven't come through their contacts normal email and the mail comes through software / app who use their mail system

      There are often system mails that come through systems or other software which use their email addresses since they use their own mail servers to mail. This causes an issue as it does not record the mail in the history of the CRM since the email is not

    • Recommendation

      I give up on Zoho. It's never going to be an all in one solution, their own apps don't even connect. Can any one recommend an alternative at least for the crm / people.

    • CRM - Site/Location

      Hi Zoho, One massive oversight of the Zoho CRM is the inability to add multiple sites or locations to one account. Many organisations have several (in some cases, hundreds) of sites across the country or even the globe. The workaround to this, of course,

    • Next Page