Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
New to Zoho Recruit?
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Issue with ticket replies via Slack: '+' symbols replacing spaces in emails
Hello, support team! We're experiencing an issue when replying to tickets directly through Slack. When the reply is sent to the email, spaces are being replaced by '+' symbols. This makes the message harder to read and understand. Is there any solutionAllow 2 logos for Branding, one for Light Mode and one for Dark Mode?
Our logo has a lot of black text on it. If we leave the background transparent, per recommendation of Zoho, when a user is viewing a file and turns on dark mode, our logo is not really visible and looks really weird. It would be really great if we couldZoho Creator Populate radio field with values with all the created rows subfor
I have Main Form where i have a lookup field where i get brewery names and the number of tanks as a multiline text field with a list of beer names Based Brewery selected and bbt_tanks number i create rows in the subform and now i want to populate listCurrency column showing $ symbol
Hello, I'm importing data from Zoho Projects to Zoho Analytics and I was wondering why "Budget amount" column is set in dollars even if the "Currency" column = EUR: Is there a way to get the budget amount as "EUR" + nnnnnn? Thank youAndroid notifications not working
I've set push notifications to 'on' in ZohoMail for android settings but nothing doing. Can anyone help? I do use a VPN.my clients are not receiving mails
Hi, My clients are not receiving my mails sent . may we know the reason My dns server and imap settings are perfectMúltiple Deals when converting a Lead
Hello!!! I hope someone can help me figure out the best way to handle this scenario. I have a multi-select field named “Service” in the Leads module that captures either Service A, Service B, or both. When converting a lead, Zoho CRM currently createszoho mail and crm is very slow
I have recently employed Zoho in our organisation. Even after taking high speed internet, mail and CRM takes many minutes to even load. Its really slow and faces lot of downtime.How to use if_case with expressions other than equals
I'm trying to define a formula column that implements logic like this case statement would: case when numfld1 is null then null when numfld2 > 0 then 100*numfld2 when numfld2 < 0 then numfld2 else 0.0 end In formula columns, the docs say you need to useZoho CRM's V8 APIs are here!
Hello everyone!!! We hope you are all doing well. Announcing Zoho CRM's V8 APIs! Packed with powerful new features to supercharge your developer experience. Let us take a look at what's new in V8 APIs: Get Related Records Count of a Record API: Ever wonderedCreate global project dashboard for all users
Would like to be able to create a custom dashboard for projects with certain widgets that are default for all new projects. right now, I have to modify each project dashboard per project per user. This is not practical.What's New in Zoho Inventory | January - March 2025
Hello users, We are back with exciting new enhancements in Zoho Inventory to make managing your inventory smoother than ever! Check out the latest features for the first quarter of 2025. Watch out for this space for even more updates. Email Insights forInline images are not shown on iPhone
When I add an image inline it gets displayed on a Zoho's computer software or web browser, but not on Zoho's iPhone app - the image appears to be broken and cannot be copied neither saved. What's the problem with displaying images inline when readingKaizen #186 : Client Script Support for Subforms
Hello everyone! Welcome back to another exciting Kaizen post on Client Script! In this edition, we’re taking a closer look at Client Script Support for Subforms with the help of the following scenario. " Zylker, a manufacturing company, uses the "Orders"Viewing Live data
Where can I see the live data that is sent from the device?Canvas templates can now be shared with different CRM organizations
----------------------------------------Moderated on 14th February, 2023------------------------------------------- Dear all, This feature is now open for all users in all DCs. To learn more about importing and exporting canvas templates, read our helpFormatting Mailing Labels
I want to use the "Print Mailing Labels" function on the drop down list, but I am not seeing a way to change the formatting on the mailing labels. At the moment, the information that appears on the mailing labels ARE NOT mailing addresses, but random information. I would also like to change be able to change the size of the labels. At the very least I would like to know what type of labels I can get that would be the correct size.CRM to Writer Mail Merge Preview not working
When performing a mail merge from CRM to writer the preview function does not work. I get the following error. I am a Zoho one user on a ChromeOS. I have been successfully using mail merge from CRM to Writer about 4 years. This error seemed to coincideBest practice : when to convert lead to Deal
Hello, I'm new to Zoho and run my own business. To make sure I'm using Zoho correctly, when do I press convert, from Lead to Deal, at what stage in the conversion funnel/conversation. I want to make sure I can a) monitor status of all pending lead orShow Call History During a Blueprint Transition in Leads Module
Hi all, I have a Blueprint set up in the Leads module with a transition to Reattempt Call, which updates the lead status to Attempted Contact. I’d like to know if there’s a way to show the call history or at least a summary of how many call attempts haveHow do I filter contacts by account parameters?
Need to filter a contact view according to account parameter, eg account type. Without this filter users are overwhelmed with irrelevant contacts. Workaround is to create a custom 'Contact Type' field but this unbearable duplicity as the information alreadyHow to delete more than 100 leads at a time.
We are a call center and we need to upload fresh leads daily. Is there any way to delete all leads only at once. Currently we are deleting 100 at a time. Please anyone who can help. Thank you.The Next Chapter for CRM for Everyone: Moving from Early Access to Phased Rollout for Customers
#CRM25Q1 Hello Everyone, Until now, CRM for Everyone has been available in early access mode exclusively for users who opted to try the new version. We are now transitioning to a phased release, starting with the basic edition. We are thrilled to announceCanvas for related lists
Hey, we would like to customize our related lists. For us, it would make more sense to present the data from an assigned record vertical instead of horizontal. Can we get a related list Canvas view?Standalone custom function not generating logs
Why dont't standalone custom functions generate logs when the're called from another function? I have some functions (workflow, buttons and blueprint) that have common parts, so I put that part in a standalone function which is called from the others.Show Zoho Books Retainer Invoice in Zoho CRM
Hi Support, How can I get Retainer Invoices created in Zoho Books to show in Zoho CRM? If a sales person needs to collect an upfront deposit, they should be able to see that the retainer invoice has been created and paid. Thanks, AshleyFeature Request - Zoho Books - Add Retainer Invoices to CRM/Books integration
Hi Books Team, My feature request is to include Retainer Invoices in the finance suite integration with Zoho CRM. This way we will be able to see if retainer invoices have been issued and paid. I have also noticed that when the generate retainer invoiceACR Phone mobile app for logging phone calls into Zoho CRM
ACR Phone is an Android app recording voice calls with additional features such as blacklist, cloud upload, call log and more. Use the ACR Phone extension for Zoho CRM to generate Leads and Contacts with the voice recording attached right after your phoneNew in Zoho Sign: Allow recipients to review and change associated recipients
Greetings! We are happy to announce the addition of a new recipient role, Manages recipients, in our signature workflows. This role enables a recipient to review, modify, or add details for any recipients associated with them in the workflow. This ensuresHow can I throw an error / terminate the flow from within a custom function?
As the subject says. I would like to be able to terminate a flow from within a custom function if certain conditions are not met. I know I could hook a decision box to the output of my custom function and check return variable, but hoping there is a moreMultiple Zoho Attendees in a Customer meeting
We are having constraints with having to log duplicate meetings when we have 2 Zoho users attending a customer meeting. What are the options to resolve this? You can add participants, but you cannot report on them. What can be done to avoid creating soiOS 18 is here! Discover the enhanced Bigin app with iOS 18, iPadOS 18 and macOS Sequoia.
Hello, everyone! We are excited to be back with new features and enhancements for the Bigin app. Let us take a look at the new iOS 18 and iPadOS 18 features. The following is the list of features covered in this update: Control widgets. New app icons.Multi-Select lookup field has reached its maximum??
Hi there, I want to create a multi-select lookup field in a module but I can't select the model I want the relationship to be with from the list. From the help page on this I see that you can only create a max of 2 relationships per module? Is that true?Password change restriction
Can the administrator restrict users from changing their email password?NOW Zoho Creator still cannot bulk download Image or File Upload Field
The filedownloader has been deprecated for 5 years. Until now, we still cannot have a replacement tool. How can we bulk download the file that we uploaded to Zoho Creator. Previously, it was so simple to bulk download all those files. But now failed toCRM API Search Record for Last Name equals "."
When using the CRM API to look for all contacts with a lastname = "." The API returns an Invalid Query Reponse I have tried (Last_Name:equals:.) (Last_Name:equals:%5C.) (Last_Name:equals:\.) We have a scenario where the Last Name may not be known forMS PowerApps Custom Connector
Has anyone successfully connected to the API from MS PowerApps? The OAuth2 seems impossible to use from a platform like this.Zdk-cli
As i have tried to login to zdk cli as it returns this error ✔ Success! Logged in as JAYANTHAN ✖ Error during initialization of zdk api supported only in sandbox environmentSe puede mover un Cliente de un modulo a otro personalizado
Hola, tengo una duda que me gustaría resolver: Actualmente trabajo con dos módulos: Seguimiento de Venta y Cierre de Ventas . Mi objetivo es que, cuando desde el módulo de Seguimiento se marca una venta como "Venta Efectiva" , el cliente sea movido alIts 2022, can our customers log into CRM on their mobiles? Zoho Response: Maybe Later
I am a long time Zoho CRM user. I have just started using the client portal feature. On the plus side I have found it very fast and very easy (for someone used to the CRM config) to set up a subset of module views that make a potentially extremely usefulNext Page