Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Kaizen #216 - Actions APIs : Email Notifications
Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes areKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achievedKaizen #142: How to Navigate to Another Page in Zoho CRM using Client Script
Hello everyone! Welcome back to another exciting Kaizen post. In this post, let us see how you can you navigate to different Pages using Client Script. In this Kaizen post, Need to Navigate to different Pages Client Script ZDKs related to navigation A.Kaizen #210 - Answering your Questions | Event Management System using ZDK CLI
Hello Everyone, Welcome back to yet another post in the Kaizen Series! As you already may know, for the Kaizen #200 milestone, we asked for your feedback and many of you suggested topics for us to discuss. We have been writing on these topics over theKaizen #197: Frequently Asked Questions on GraphQL APIs
🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Live webinar: Craft and deliver impactful slides with Show’s desktop apps
If you love the comfort of working on your desktop and want a setup that keeps you focused and uninterrupted, this session is made for you. We’re excited to invite you to our upcoming live webinar: "Craft and deliver impactful slides with Show's desktopEnhancements to Bigin's forms
Greetings, I hope all of you are doing well. We're happy to announce a few recent enhancements we've made to Bigin's forms. We'll go over each one in detail. Merge field support in auto-filled forms The auto-fill option in Bigin's forms lets you predefineTable dimensions
I try changing the dimensions of the table on my computer but it doesn't change. Do I have to be a premium member or does it only work on the app?Direct link to Record Summary
Hi everyone, In one of my reports, I have built a Record Summary template to display the details of one record. I would like to be able to link directly to this Record Summary once I submit a new record, without having to go to the list of records first and click on View. Is there a possibility to do so ? Should I use the URL by passing some parameters ? Thank you very much for your help ! GuillaumeAmendment effective date
Hi everyone, I noticed that the amendment effective date mentionned in my amendment is not right. Indeed, when a contract is amended several times, it states the previous amendment and their effective date. However, the effective date stated is alwaysSTOCK history in zohosheets
is it possible to get historical stock value using stock function in zoho sheets? i could not see from and to period in the helper document.Auto sync Photo storage
Hello I am new to Zoho Workdrive and was wondering if the is a way of automatically syncing photos on my Android phone to my workdrive as want to move away from Google? ThanksZoho Desk Training
Hello, We've had Zoho desk for a while now, but we run into issues occasionally, and I was wondering if there was a customer who currently uses it and really enjoys the functionality, that would be wiling to chat with us?Introducing parent-child ticketing in Zoho Desk [Early access]
Hello Zoho Desk users! We have introduced the parent-child ticketing system to help customer service teams ensure efficient resolution of issues involving multiple, related tickets. You can now combine repetitive and interconnected tickets into parent-childText/SMS With Zoho Desk
Hi Guys- Considering using SMS to get faster responses from customers that we are helping. Have a bunch of questions; 1) Which provider is better ClickaTell or Screen Magic. Screen Magic seems easier to setup, but appears to be 2x as expensive for United States. I cannot find the sender id for Clickatell to even complete the configuration. 2) Can customer's reply to text messages? If so are responses linked back to the zoho ticket? If not, how are you handling this, a simple "DO NOT REPLY" asAgent password reset
Hi Zoho support, I would like to ask if there is a way the admin can reset a password of an agent? RegardsCan receive but not send messages in Zoho Mail
Hello! I was able to configure my email client successfully in that I can receive messages just fine. However, when I send messages out, they seem to go out fine (I don't receive any errors or anything), but the recipient does NOT receive those messages.Mail is sent twice!
Been using Zoho for a while now. Installed Zoho for someone else and some weird things are happening. Mails are being sent twice. He is using Thunderbird as an email client. I already read about email being duplicated in the sent folder. But in my caseCan't login IMAP suddenly
Since this evening I'm getting the error: You are yet to enable IMAP for your account. Please contact your administrator... IMAP always been enabled in my account and was workign fine for the past 7 years. Already tried turning IMAP off and on again.Sending of username did not succeed: Mail server pop.zoho.com responded: User already specified
I am having issues receiving emails from Zoho in Thunderbird. I am getting the above error. The first error tells me Authentication failed, and prompts me to enter in my password. Then I get the above error. I can receive emails when I log in online toBug tracking
Hi, does anyone know how to track errors during picking or packing? This way I can keep track and see how to improve and prevent errors in this area.Exact match in name when searching workdrive
Hello, I am wondering how to search workdrive files/folders with an exact match in the name. For example, when I search across folder with the url param search[name]=someName, I get multiple results such as "someName", "someNameAndMore", or "someNameFlow - Fetch info from drop down in another module
I am running into a road block which I thought would be a simple task. My goal - The account is assigned to a "route" which can be selected from a drop down menu and adds a tag to the account accordingly (easy enough). Now when I create a task for thisMigration of corporate mail environment from Yandex 360 to Zoho mail
I have to migrate a corporate mail environment with an existing domain from Yandex 360 to Zoho mail. It is vital to migrate all users with all the data. I have read the article on this topic using MacMister Email Backup Software just now and have someI'm unable to send mail pthrough Zoho SMTP programmatically
This has been working for years, but today it's been offline all day long. I see nothing anywhere on your site about this. I'm not the only one experiencing this. Downdetector has a spike of reports todayCan no longer send email via Django site
This was working fine as of 11/7/25. Now I am unable to send user verification emails from a Django site on a AWS lightsail sever. When a user attempts to register the following error occurs. I have also attempted to send a test email via the shell andunable to send email but able to receive email
my email address is info@securityforceservices.caCustom Field for Subscription
Hi, I can't find a way to add a custom field (to contain a license key generated from our software) against a subscription? Is the only place to add this information in the Invoice module (as custom field for invoice)? When a customer views his subscription via the customer portal, there appears no way to display a license key for them? The invoice is not the natural place to store a license key for a particular subscription, so where else can this be stored and displayed?Login to server failing
When trying to retrieve my mail, I am getting this error message -- Login to server pop.zoho.com with username (my email address) failed. It gives me the option to retry, enter password, or cancel. Then I get this message -- Sending of username did notConfiguration failed: 200 response not received for POST request.
Hello, I am trying to set up a webhook to connect with an Salesforce but I receive the following error from Zoho: Configuration failed: 200 response not received for POST request I have tried testing it on webhook.site as well and receive the same errorZoho Migration Assistant not working
Hello, I am trying to use you Migration assistant to migrate emails from Rediff to Zoho. I am stuck in the first step. After downloading the migration tool, I copied the link to verify user credentials, however, after pasting the link in the browser,Paid Support Plans with Automated Billing
We (like many others, I'm sure) are designing or have paid support plans. Our design involves a given number of support hours in each plan. Here are my questions: 1) Are there any plans to add time-based plans in the Zoho Desk Support Plans feature? TheScheduled Reports - Do not send empty report
Hello, We are intensively using reports in the CRM, especially for sales managers. When data is empty, they still receive an email. Can you add an option to avoid sending the report when data is empty?Contacts Missing — PeopleSync/Zoho Mail
English: In our company we use ManageEngine Mobile Device Manager (MDM), Free edition, to manage corporate mobile devices. Our usage policy does not allow personal Google accounts on these devices; therefore, Google account sync is blocked through MDM.Best way to integrate Zoho with mobile app for managing customer requests with real-time notifications?
Hello, I'm building a solution for a travel company where customers submit requests through a website, and the sales team manages these requests through a mobile app. The Requirement: Customers fill a form on the website (name, email, number of children,Kaizen #57 - Mass Update API in Zoho CRM
Hello everyone! Welcome back to yet another post in the Kaizen series. This week, we will discuss the Mass Update API in Zoho CRM. In this post, we will cover the following: 1. Introduction 2. Mass Update Records API 3. Schedule Update and Get StatusGetting Attachments in Zoho Desk via API
Is there a way to get attachments into Zoho Desk via an API? We have a process by which a zoho survey gets sent to the user as a link in a notification. The survey has several upload fields where they can upload pdf documents. I've createdMultiple currencies - doesn’t seem to work for site visitors / customers
I am trying to understand how the multiple currency feature works from the perspective of the website visitor who is shopping on my Zoho Commerce site. My site’s base currency is US Dollars (USD) but my store is for customers in Costa Rica and I wouldPincode based Product Restriction
we have different types of products. 1) Very bulky items like plywood. 2) Too delicate items like glass These type of products we want to sell to local customers. Other products we want to supply all over India. There should be an option to restrict productsCan multiple agents be assigned to one ticket on purpose?
Is it possible to assign one ticket to two or more agents at a time? I would like the option to have multiple people working on one ticket so that the same ticket is viewable for those agents on their list of pending tickets. Is something like this currentlyRelated Lists filter
I have Contacts showing in our Accounts module. I customized the Contacts module with an Employment Status field, with the following picklist options: "Primary Contact", "Secondary Contact", "Active Staff(not a main contact)", and "No longer employed".Standalone custom function not generating logs
Why dont't standalone custom functions generate logs when the're called from another function? I have some functions (workflow, buttons and blueprint) that have common parts, so I put that part in a standalone function which is called from the others.Automated log-out/session end
I'm concerned about security of our data. Is it possible to set an automatic time-out for user sessions on Zoho CRM, after a certain period of inactivity or when the session reaches a certain duration (12 hours perhaps)?Add "Reset MFA" Option for Zoho Creator Client Portal Users
Hello Zoho Creator Team, We hope you are doing well. We would like to request an important enhancement related to Multi-Factor Authentication (MFA) for client portal users in Zoho Creator. Currently, Creator allows us to enforce MFA for portal users,Support Bots and Automations in External Channels
Hello Zoho Cliq Team, How are you? We actively use Zoho Cliq for collaboration, including with our external developers. For this purpose, external channels are a key tool since they work seamlessly within the same interface as all of our other channelsNext Page