Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho Campaigns Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Zoho Flow or Schedules
I have a process where we text our leads 7 times over a 14 day with different content for each text. I created one flow in Zoho Flow to do this, but wondering if there is a more efficient way to accomplish this via Schedules. It goes on for 6 more timesFree webinar: Zoho Sign 2024 wrap-up - Everything that is new and has changed
Hello, Are you looking up to catch up on all the updates made to Zoho Sign in 2024? Or are you still figuring out how you can use Zoho Sign better to get business paperwork done more efficiently? If so, we invite you to join us this Thursday, DecemberHow to Customize Task Creation to Send a Custom Alert Using JavaScript in Zoho CRM?
Hello Zoho CRM Community, I’m looking to customize Zoho CRM to send a custom alert whenever a task is created. I understand that Zoho CRM supports client scripts using JavaScript, and I would like to leverage this feature to implement the alert functionality.Workflow - Execute Based on Date
Hello, I have trouble understanding the documentation for Execute Based on Date or Date Time Field's Value. I want to send an email every time I have a Case opened for more than three days with its status unchanged. I set : This rule will be executed 3 days after [date]. Condition : Status is [New]. Instant Action : Send an email notification. However, I'm not sure I follow this part of the documentation: "For all the records matching the rule criteria, rule will be triggered either monthly or yearlyCan we set a BCC address as default to show while sending emails?
Two things inside ZohoCRM are annoying me because it's a repeated work. First one is that I always need to click manually to add the BCC field while sending an email to a lead. Can we set a default address so when I click to send a new email the BCC addressMake collecting payments from your customers in Bigin easier with payment links
Greetings, Efficient payment collection is crucial for business success. Bigin already helps your businesses manage and sell products effectively, but we can further enhance this by making payment collection easier. This integrated payment feature letsSend email is not authenticated
Hi, I’m getting an error in Gmail, when receiving an email from my account in zoho, my email is already authenticated in my domain, and I don't know why I keep receiving this message... also testing in outlook, the message goes directly to "junk".Inbox Preference - Saved replies based on message repsonse with specific word
Hi There, Can one create a workflow where the Save Reply will be generic for all messages that contains a specific word, various channels,. The saved reply should contain a link for download etc. that is a response to the word entered in the message containedBasic Price List Functionality Still Missing
I am having a problem with the most simple imaginable pricing scenario - you buy cheap, add profit, then sell high. Or in less simplistic terms: business buys a product at a given cost, then adds predetermined percentage markup, and finally sells thatHow do I hire employees????
Hi! I own a bookkeeping company, where a few of my clients use Zoho Books as their accounting platform . I started utilizing Zoho Practice to work on the books of my Zoho clients, some have Zoho One and some have Zoho Books plans. I just hired an employee,Automate data upload process like reports
I'll start with the end in mind. I want to basically keep certain creator tables updated with data that are in a sql database/tables in our office (employees, active jobs, employee positions) so I can reference that data and not have to duplicate it by hand every time someone adds a new job or employee in the office desktop software. Here are some thoughts I had about how to do this, but am unsure as to whether any of them are actually possible and how to go about it from there: Is there any wayGreylisted, try again after some time
Can you check my ip, i send to duyna@vietlinkjsc.vn but have an error; my ip is 112.213.94.12 Here is log: 2018-01-09 09:40:29 H=mx.zoho.com [204.141.32.121] SMTP error from remote mail server after RCPT TO:<duyna@vietlinkjsc.vn>: 451 4.7.1 Greylisted, try again after some time 2018-01-09 09:40:32 H=mx2.zoho.com [204.141.33.55] SMTP error from remote mail server after RCPT TO:<duyna@vietlinkjsc.vn>: 451 4.7.1 Greylisted, try again after some time 2018-01-09 09:40:32 duyna@vietlinkjsc.vn R=lookuphostEmails linked to Deal
Hello everyone, I’d like to ask a question to see if someone can help me out. We are requesting availability from suppliers by sending emails directly from the Opportunity. These emails we send are logged within the Opportunity; however, when we receiveHow to transfer all my mails from Zoho to Gmail or Office 365
is there any option to move my emails from zoho to gmail or office 365. i would like to export more than 25k emails from zoho to office 365 or gmail. can anyone help me to guide properly. this will help me to access my emails easily i have both account and can easily do it with office 365 or gmail. i want two options. direct from zoho to office 365 or exported eml files from zoho to gmail. please suggest me both if possibleInquiry Regarding Image Display Issue in Campaign Duplication
We are currently using Zoho Campaigns for email distribution to our clients. I would like to inquire about an issue we encountered. When duplicating a previously created and sent campaign from the "All Campaigns" section, the images used in the headerNew integrations for Bigin: Zoho Sign, SalesIQ, and Marketing Automation
Greetings, We're excited to share new integrations that make Bigin more powerful and useful for your business! Zoho Sign for Bigin Zoho Sign now integrates seamlessly with Bigin, enabling you to sign, send, and manage contracts or agreements without leavingAdd multiple users to a task
When I´m assigning a task it is almost always related to more than one person. Practical situation: When a client request some improvement the related department opens the task with the situation and people related to it as the client itself, the salesmanWhat is Attendee Status 0 and 1?
Hi there, I recently stumbled upon the API to get the attendee list and in the return value, there is a parameter called "status", and 0 supposed to mean not_attending, and 1 means attending. I cannot find this representation anywhere in the attendeeZOHO BackStage
How to get list of events, using ZOHO BackStage APIs. Is it possible OR not?How to Customize Task Creation to Send a Custom Alert Using JavaScript in Zoho CRM?
Hello Zoho CRM Community, I’m looking to customize Zoho CRM to send a custom alert whenever a task is created. I understand that Zoho CRM supports client scripts using JavaScript, and I would like to leverage this feature to implement the alert functionality.Zoho Meeting iOS app update - Join breakout rooms, access polls, paste links and join sessions, in session host controls
Hello, everyone! In the latest iOS version(v1.7) of the Zoho Meeting app, we have brought in support for the following features: Polls in meeting session Join Breakout rooms Paste link in join meeting screen Foreign time zone in the meeting details screen.Calculate hours between 2 date/time fields
Hi, Does anyone know if it is possible to get the number of hours between 2 date/time fields in a zoho crm custom function? Thanks, MichaelExternal ID validation.
I added an external ID field as below in one of my custom modules: When creating records via the API using some value (eg: 762115b2-097e-43b2-bdba-f3924a5371a6) for this field, it works without any problem. I can create and even see the records on theCRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive
Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage filesRemove Address from credit card payment
I would like to remove the need to add address when paying by credit card. I only want the customer to have to add their credit card details.Create a button that executes a customized function
Hello, I have created a summary view in which I combine the data from my items table and suppliers table. I would like to know if there is the possibility of adding a button somewhere in the view to be able to execute a function when clicked on it. ICustom field doesn't fill when converting sales order to invoice
Hi, When I convert a Sales Order to an Invoice one of the custom fields on a product line names "Subsidie" does not seem to fill in automatically. I manually have to select the product again by clicking on the product name in the order line en re-selectCan I use ZOHO calendar to schedule a Youtube video that is already in my youtube account, but listed as private or unlisted?
I am creating Youtube videos and shorts and then uploading them to our Channel so others can view and approve. Once approved I would like to just schedule them over the next few days within Zoho. So far it looks like I have to re-upload the video to ZohoHow to Display a Logo Image on a Public Form?
I would like to display a logo image in the header of a form. To achieve this, I added an Add Notes field to the form. The code below works perfectly for Zoho users accessing the form. However, when the form is made public, the image does not load properly:Advice for my first project in Zoho
Hello, how can I design and implement a customized ERP and CRM system using Zoho to automate and manage core business functions, including customer relationship management, property inventory, sales tracking, and financial processes. This is one of myAssociate Email API Internal Error
I am trying to associate an already existing email within a function using the Related Emails API. To provide more context, I also have admin permissions and have ensured that the fields are correct and that I have admin permissions when associating theProfit on Sales order
Hi, would it be possible to implement a column at the Sales order overview of Purchase amount? So a field with the amount of all purchase related to this Sales order? This is very usefull so you will see the profit you made on this deal. I tried to getSubform Data in v2 REST API
What is the mechanism for adding subform data in the Creator v2 REST APIs? There is nothing documented in the Data APIs documentation (https://www.zoho.com/creator/help/api/v2/). I was able to determine how to GET the subform data by adding it to theIs the filename of an attachment restricted to only certain languages?
Hi, I tried to create a file card in a notebook, but I encountered an issue: when the filename of an attachment is in languages other than the main ones (such as English, Japanese, or Simplified Chinese......), syncing stops or the filenames become garbledHow to create comparison time periods like these examples
In a Pivot Report I would like to be able to select any date range and show a set of metrics for that date range e.g. revenue, orders, units sold. I would then like to be able to compare to the previous period based on the amount of days on the selectedOverwrite Option for custom modules
Hi Team, I noticed that the overwrite option is unavailable in Zoho Books when importing data for custom modules. This limitation makes it challenging to bulk update old data, as the only option is the 'bulk update' feature, which is restricted to 25Zoho Creator - Zoho Analytics
I am facing an issue in Zoho Analytics where I am still seeing deleted data from the Zoho Creator form I created. Could you please look into this and let me know what needs to be done?Unable to send emails
I have this email parth@mrcolumbus.in, but I couldnt send outgoing email. Can you please help?Notifications push : Encourager le réengagement et renforcer la fidélité des utilisateurs efficacement
Vous avez déjà souhaité engager et communiquer de manière proactive avec les utilisateurs, y compris lorsqu'ils utilisent votre application de manière peu active ? Zoho Apptics vous offre déjà des fonctionnalités qui vous permettent d'évaluer la performance,Easy way to delete attachments
I've reached my data limit and would like to run a view/report, and mass delete attachments. Is there an easy, fast way to do this? Moderation Update: Post Summary: There are two features the post discusses a) Easy way to remove Email attachments WillNext Page