Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho Campaigns Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Bug - OTP (email) and No Duplicates
Scenario: Form with an email field, Validation: "No Duplicates" (because I want to ensure 1 entry per email). Embedded form into website (JS option). Enabled email based OTP. 1st test (via my website) - entered my email address - sent OTP - entered pin,Personal Facebook page posting instead of Business Page
I have a Facebook page that is associated with my Personal Profile and I am the Admin of that Page. I would like to schedule and Post to my Personal Page not the Business Page. Each time I try to connect to the "Page" it takes me to the Business Page. Is there a way of connecting to my personal page?Fixed asset management
I want to know if there is any individual module for fixed assets managementRecording depreciation of fixed assets as a percentage of residual value
In India, fixed assets are depreciated as a percentage of their residual value at the beginning of each fiscal year. I went through the documentation for creating recurring journal entries, but could only find ways to depreciate by a fixed rupee amountFunction #28: Automatically calculate Customer Loyalty points
Hello everyone, and welcome back to our series! Today, we're excited to share a workflow designed to streamline the management of loyalty points. Many businesses offer incentives or rewards in the form of loyalty points to their customers as a way toFunction #6: Calculate Commissions for paid invoices
Zoho Books helps you automate the process of calculating and recording commissions paid to sales persons using custom functions. We've written a script that computes the commission amount based on the percentage of commission you enter and creates anHow to Add Product SKU in Invoice?
How to Add Product SKU in Invoice?Tracking movement between departments
I've been developing a reporting system in Zoho and one of the groups I want to develop a report on primarily moves tickets from department to another. Is there a way to set up the reporting on Zoho (or Zoho Reports) that can tell me the number of ticketsZoho CRM Calendar View
Hello Zoho team, We need desperately a calendar view next to list, kandan and other views. I think it should be easy to implement as you already have the logic from Projects and also from Kanban View in CRM. In calendar view when we set it up - we chooseCall transcrition working for ringcentral?
I don't see anything about what telephony providers can be used. The Zoho support person A said that RingCentral isn't supported. Zoho support person B said that it works, just make sure the call recording link works. Excellent instructions here: CallWhat is syntax to call creator function (or trigger a creator workflow) from CRM deluge?
What is syntax to call creator function (or trigger a creator workflow) from CRM deluge?WhatsApp and Zoho Creator Integration
How we have integrate WhatsApp App with Zoho Creator without using external application ?Improve Creator Calendar Report
Please can you improve the Creator Calendar Report General There is no way to highlight certain days, for example weekends or public holidays. There is no way to hide certain days, for example weekends. There is no way to modify the day header, it justImportant updates to Zoho CRM's email deliverability
Last modified on: Jul 24, 2024 These enhancements are released for all users across all data centers. Modified on: Oct 30, 2023 Organisations that are in the Enterprise and above editions of Zoho CRM, and have not authenticated their email-sending domainsCustom modules not showing in developer console
I'm trying to create a custom summing function for a custom module I made in my CRM. When I go to create the function, my module isnt showing up. Do I need to share the custom moldule with my developer console or something of the like?Chart with Filtered Data vs Unfiltered Data
I am looking to create a chart view that displays the full data set vs a subset of the data filtered by user filter. However I do not seem to find any method by which to exclude a plot from the applied filter or any other method by which to display theMeetups Gratuitos Junio 2024 - Profundiza en las funcionalidades de tu Zoho CRM
Este junio, aprende a sacar el máximo provecho de tu Zoho CRM en la segunda edición de los Zoho Meetups 2024. Los días 18 a 21 de junio, Zoho organiza los Meetups gratuitos para usuarios de Zoho CRM en Valencia, Barcelona, Madrid y Sevilla, donde expertosHow to get the Dashboard page to be the first page when you open the app
So when it opens on a tablet or phone it opens on the welcome page, thanks.Integration between Zoho CRM and Zoho WorkDrive
I'd like to search Zoho for an invoice I've added as an attachment (pdf) to an account. The name of the invoice is 1388-advertiserx-July.pdf - but I can't find it using the search function for any of these terms: 1388 1388-advertiserx 1388-advertiserx-July.pdfTip #17: How to mandate partial payment for your appointments
When you require partial payments during the booking process, customers can only schedule with you after paying a certain amount in advance. This deposit acts as a commitment between both parties. Apart from that, it has many more advantages. BenefitsWhy option for 'include form submission in the body of the email' check box is missing
Hi In all our forms we have configured an <Email Notification> (<Rules> <Form Rules>) In some forms, there is an option to <include form submission in the body of the email> in the <Additional Options> section, however, this option is not available inMozilla Vault extension will not Unlock even once.
I have been using the Vault extension in Chrome based browsers for years, yet after installing the Mozilla Extension in Firefox, it will not unlock. Initially it did redirect me to log into my account, and then enter the master password. However, it didUnable to Download CRM Contact Data: WorkDrive Integration Issues
## Problem Description We need to let users download contact information from CRM as CSV files to their local computers. Since we couldn't implement a direct download option, we're trying to use WorkDrive as a workaround - but we're encountering issuesSort mail by name and subject
I don't see sort function on columns FROM and Subject. I see only sort functio by date. Could add it ?Zoho Creator monthly roundup - September 2024
Hello all, We're back with an exciting set of new features and enhancements that will elevate your Creator experience even further. In case you missed it, we’ve recently revamped our Product Roadmap page, now with a refreshed design and showcasing allKiosk Studio Session #1: View paid customers in the same industry
Update | 15 Oct 2024: Session #2 is now available here! Hello everyone! We're excited to launch our new series of posts on Kiosk Studio today. Called Kiosk Studio Sessions , these posts will be packed with actionable ideas to help you get the most outIssues hosting Zoho Desk Web Form on SharePoint and/or Power BI
Zoho Desk onboarding support has no experience with embedding their web form in either SharePoint or Power BI. Microsoft states that SharePoint and Power BI only support iframe HTML. And unfortunately, the web form embed code that Zoho generates is not"Send with Zoho Sign" broken
Our company uses hyphens in our file name conventions. Our users have been sending the files from other modules with the "Send with Zoho Sign" shortcut in the upper right buttons. Since around June 10, 2024, this stopped working. Our users can send theNot able to change colors help center
Hi. How can I change the orange color in the help center? You can change everything besides this font color And how can I remove the part on the bottom?Transform Numeric Values using st, nd, rd, th or Convert Numerals to Ordinal Form - Deluge
Please Use this sample Code This_Day_Date = zoho.currentdate.toString("dd"); value1 = "th"; if(This_Day_Date.right(1) ="1" && This_Day_Date != "11") { This_Day_Date = This_Day_Date+" "+"st"; } else if ( This_Day_Date.right(1) = "2" && This_Day_Date !=Kaizen #166 - Handling Query Variables in Zoho CRM
Hello, Code Enthusiasts! Welcome to another week of Kaizen! This week, we'll dive into handling variables in Zoho CRM Queries and see how they can be deployed in Kiosk to dynamically retrieve data. This technique is especially useful for integrating dataAutomate User Invitations on Zoho Desk with API
Automate User Invitations on Zoho Desk with API Hello Team, We are excited to announce that you can now automatically invite users to the Zoho Desk portal using the API! ### How It Works For example, when a contact is created in Zoho Desk and you enableIs there a way to add clients who don't have organisation in Zoho Books/Payroll/Expense ?
The Zoho Practice software is only allowing a total of 15 such clients who are not organisations in Zoho Books/Payroll/Expense. i.e. 5 organisation in each of the software by creating a new organization for them and adding the accountant as the adminEmail with attachments saving attachments into Zoho CRM from Zoho Mail
Hi, I get a lot of emails from prospective clients asking if we would bid their project. Those projects usually have many documents associated with them that I link to. I would like to have those documents be saved as an attachment in my Potential or Contact or Account. I don't see a way to do that that isn't multi-step. As of now I do the following: 1.) Open email 2.) If email sender isn't in my Zoho CRM database I enter them creating a Potential 3.) I download the attachment and save it to a differentFixed assets recording
Hello there, I recorded a bill for a vendor contain (Computer) so the PC is a fixed assets, do I need to do a manual journal to include this PC under the fixed assets category (furniture & equipment)? If yes, please take me through the manual journalMulti-Select lookup field has reached its maximum??
Hi there, I want to create a multi-select lookup field in a module but I can't select the model I want the relationship to be with from the list. From the help page on this I see that you can only create a max of 2 relationships per module? Is that true?Zoho Sheet-Pulling in Data
I have a module where we track POs, Bills etc (its easier for us this way). In this PO Module, we have a task related to a Job/Deal and various fields for costs. A Supplier on a Job might have several POs in this module for the job. I want to generateCan't get ZFS file ID of a field via specific module record
I have a `fileupload` field on a custom module that I need to replace the file for but I don't know how to get the ZFS ID from the module record. # Getting the current fileupload GET https://www.zohoapis.eu/crm/v7/<MODULE>/<ID> # Response { data: [{ //Contemplating moving my site from WordPress to Zoho Sites
Hi Everyone, We currently find ourselves in a situation where we ant to review and update our current sites content. We are small business owners, not developers. We currently use a wide range of Zoho products. We sometimes think about the possibility of either moving or just starting from scratch on Zoho Sites. I would like to know if anyone has done this and of course the things that need to be considered. We have spent quite a bit of time getting our current site positioned organically and I guessMultiple Salesperson against an invoice
Hello, Against a particular invoice, we have multiple sales people working. The reason we combine the invoice is becuase we are an exporter and often consolidate cargo for our customer to save them freight costs. How do I capture the contribution of eachNext Page