Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Recent Topics

    • How do I move Notes around within a Group?

      It says here: " You can now sort notes by title (alphabetically), or by date modified and date created. You can even organize your notes by dragging and dropping them into a particular order. To sort your notes, simply go to Settings and tap “Sort By.” Please note: all sort settings will be saved and synced across devices, except for custom sorting. Custom sorting will be device specific."However, I am unable to 'custom sort' in either Notebook for Mac or on the Web. In addition, I can't find the

    • javax.mail.authenticationfailedexception 535 authentication failed

      Hi, I am facing 535 authentication failed error when trying to send email from zoho desktop as well as in webmail. Can you suggest to fix this issue,. Regards, Rekha

    • Pocket from Mozilla is closing shop. Don’t lose your favorites . Move them to Zoho Mail Bookmarks now! 📥🔖

      The end of Pocket shouldn't mean the end of your important links and content. Easily import them into Zoho Mail's Bookmarks and continue right where you left off. You can bring over your entire Saves, Collections, and tags just the way they are. Bookmarks

    • Zoho Sign: need to leave document pending for up to a year, or maybe there's a better way?

      I have zoho one, maybe there's a better way to do this with another service than sending a zoho sign template from zoho crm. At the end of the day this requirement is due to regulations, no matter how dumb it may seem. I'm just looking for a way of getting

    • 'Add Tax To Amount' not reflected in Invoice

      Hi Zoho Support, I'm experiencing an issue with tax calculation display in my invoice template. Despite having "Add tax to amount" box checked in the template settings, the Amount column is not showing the tax-inclusive total for line items. Current behaviour:

    • To Do: shareable task links without login

      Hi! I’m using Zoho Mail and ToDo in my daily work, and I’ve run into one limitation that’s a real blocker for me. Right now, to share tasks with managers or directors, they need to have a Zoho account and be added to a group. In practice, many of them

    • Separate Items & Services

      Hi, please separate items and services into different categories. Thank you

    • Unable to edit or delete email address

      I signed up for free Zoho today. I usually am pretty good at understanding and configuring things like this, but your interface baffles me, and your online help is cryptic to say the least. I have spent hours just trying to set up a couple of email accounts in Zoho before pointing my domain MX records to Zoho. I solved some other issues on my own, but I can't figure out this latest problem: I have created two email addresses in Zoho. Let's call the first one myname@mydomain.com and the second one

    • Mastering Zia Match Scores | Let's Talk Recruit

      Feeling overwhelmed by hundreds of resumes for every job? You’re not alone! Welcome back to Let’s Talk Recruit, where we break down Zoho Recruit’s features and hiring best practices into simple, actionable insights for recruiters. Imagine having an assistant

    • We are unable to process your request now. Please try again after sometime or contact support@zohoaccounts.com

      I cannot sign up and return the error of we are unable to process your request now. Please try again after sometime or contact support@zohoaccounts.com

    • Multi-currency - What's cooking ?

      Hi,       We have been doing this feature for sometime and we would like to give you some glimpses of it.  Working with Multi Currency :        Multicurrency support gives you the ability to handle business transactions in multiple currencies. You can define a base currency for your organization and add more currencies with exchange rates based on the base currency.  Setup :        From the setup page, you can manage all the currencies supported by your organization.       Currencies page        

    • Integrating Chatbot with Zoho Creator Application

      Is it possible to integrate a chatbot with a Zoho Creator application?

    • How to reduce programmatically the image uploaded by user?

      I need a function that will automatically reduce the pixel dimension to 800 x 600 pixels / 180 resolution or (approx. 1.37MB) of image uploaded by user from digital camera, for example, 2271 x 1704 pixels /180 resolution or approx. 11.1MB. After the user selected the image, the function will able to detect if pixels is above 800x600, process the photo (crop/ reduce) and resume upload. Need help...  

    • Dark mode for Zoho Creator / Zoho CRM Code editor

      Hi Team, Is there any plans for Dark mode in Zoho creator / Zoho Crm code editor and development pages in pipeline?

    • Is there a way to make a button scroll down?

      Looking to have a button on a landing page scroll down to another section on the page. Any recomendations outside of coding?

    • Collective-booking event not added to all staff calendars

      We assign two staff to certain events. When the client books this event, it adds it to one staff calendar (the 'organiser') but not the other. How can I ensure all staff assigned to a collective booking get the event in their calendar? (A side note: it

    • ZOHO Android Client

      Hi, I installed the Android app, but it had an issue, so I reinstalled it. I was able to add multiple accounts, but now when I add the next account, it just duplicates the one I already have and will not even allow me to enter the info for another account.

    • I'd like to suggest a feature enhancement for SalesIQ that would greatly improve the user experience across different channels.

      Hello Zoho Team, Current Limitation: When I enable the pre-chat form under Brands > Flow Controls to collect the visitor’s name and email, it gets applied globally across all channels, including WhatsApp, Messenger, and Instagram. This doesn't quite align

    • Enhance Barcode/QR Code scanner with bulk scanning or continuously scanning

      Dear Zoho Creator, As we all know, after each scan, the scanning frame closes. Imagine having 100 items; we would need to tap 100 times and wait roughly 1 second each time for the scanning frame to reopen on mobile web. It's not just about wasting time;

    • Managing Rental Sales in Zoho Inventory

      I am aware that Zoho Inventory is not yet set up to handle rental sales and invoicing. Is anyone using it for rentals anyway? I'd like to hear about how others have found work arounds to manage inventory of rental equipment, rental payments, etc. Th

    • Megamenu

      Finally! Megamenu's are now available in Zoho-Sites, after waiting for it and requesting it for years! BUT ... why am I asked to upgrade in order to use a megamenu? First: Zoho promised to always provide premium versions and options for all included Zoho-applications

    • Zoho Flow to Creator 3001 Respoonse

      I have updated my Flows with the new V2 connection to Zoho Creator, but now some Flows do not work. They take in data from a Webhook and are supposed to create a record in Creator, however creator returns a 3001 message along with a failure, but I cannot

    • File Upload to Work Drive While Adding Records in Zoho Creator Application

      Hi I am trying to set a file attachment field in zoho creator form, to enable the user to upload a scanned document from their local pc. The file should be uploaded to zoho workdrive and not to the default zoho creator storage. The file link should be

    • Why not possible to generate?

      Using this https://desk.zoho.com/DeskAPIDocument#TicketCount#TicketCount_Getticketcountbyfield on my ZML script url :"https://desk.zoho.com/api/v1/ticketsCountByFieldValues?departmentId=XXXXXXXXXXX&accountId!=XXXXXXXXX&customField1=cf_country_1:XXXXXX&field=overDue"

    • email

      Hi My crm email is not working, can you check, I have zoho one account.

    • Need option to see Mass Emails & Cadences in Gmail Outbox OR a dedicated Zoho Outbox

      Hi everyone, Right now, when we send 1:1 emails from gmail (with gmail API connected to Zoho CRM), those emails appear both in gmail's sent folder and in Zoho CRM. That works well. But when we send Mass Emails or Cadence emails form Zoho CRM, they are

    • I can't found API for Sales Receipts

      Hello May you please help me to find an API document for Sales Receipts to get data and retrive a custom fields like Invoice and credit notes Regards

    • Kaizen #205 - Answering Your Questions | Managing Picklists and Enabling History Tracking via Zoho CRM APIs

      Hello everyone! Welcome back to another post in our Kaizen series. In this post, we will look at how you can manage picklist fields in Zoho CRM using APIs. This topic was raised as feedback to Kaizen #200, so we are taking it up here with more details.

    • Multiple Vendor SKUs

      One of the big concerns we have with ZOHO Inventory is lack of Vendor Skus like many other inventory software packages offer. Being able to have multiple vendor skus for the same product would be HUGE! It would populate the appropriate vendor Sku for

    • Internally created tickets

      Hi there When tickets are created internally on-behalf of customers - there is nothing to show that the ticket was created by an internal agent. This means, that it's easy for our agents to confuse tickets which were created by internal team members and

    • Automatically change website passwords

      Hi everyone, We just switched to a Professional package to also use the "Automatically change website passwords" function. But I cannot find anything about it, how to use it, anywhere. Does anyone know how I can use this function? Best, Caspar

    • Change Invoice Prices for an Effective Date

      Hi, It would be a really good feature to be able to change the prices on invoices/recurring invoices from an effective date in the event of price increases. For instance, I am in the process of increasing prices that will be effective from a specific

    • "Other Current Asset" accounts as "Paid Through" accounts in Expense

      It would be incredibly useful to be able to assign accounts of type Other Current Asset as Paid Through accounts in Expense. Currently, Other Current Liability are permitted as Paid Through Accounts. This makes sense, as Credit Cards are current liabilities.

    • Multi column open text questions that allows respondents to add rows for additional information

      I need to create a question that has 2 columns with open text, but I also need to allow respondents to click a "+" button, or something similar, so that they can add additional information if they choose to. I've tried using the Multiple Textboxes type

    • Bot Filtering & Apple Mail Privacy Protection Compliance in Zoho Campaigns

      Dear Campaigns Users, The wait is over! We’re excited to announce that the enhanced bot filtering feature is now live in Zoho Campaigns. This update brings greater accuracy to your email campaign reports by distinguishing real user engagement from automated

    • Découvrons les détails qui simplifient vos journées de travail avec Trident

      Nous nous installons dans des routines efficaces et rodées avec le temps. Chaque matin, nous ouvrons nos e-mails, passons aux messages, consultons notre agenda, puis attaquons nos tâches. Ce processus nous semble maîtrisé, mais est-il réellement optimisé

    • Issue with Purchase Rate Showing as “0” After Importing Items List

      Dear Zoho Books Support Team, Good day. I’m reaching out regarding an issue I’m facing while importing my items list into Zoho Books. Despite mapping all fields correctly and including the purchase price for each product in my Excel file, the Purchase

    • API for Task Entity in Zoho Books

      I’m working on automating task creation in Zoho Books via a custom button in the Bills Module. The goal is to create a task in the Tasks Module and assign it to the Finance Team, so they can track progress efficiently. While reviewing Zoho Books documentation,

    • create invoice in zoho books from the zoho forms

      Is there a native way to have create invoice in zoho books, when zoho form is completed?

    • Email undelivered

      GOod Day I am always receiving an uncategorized-bounce to my email. I am not sure why this is happening.

    • Next Page