Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho Campaigns Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Blueprint: multi-select lookup field not available in the criteria option
I read this old forum post which stated that multi-select lookup fields are now selectable as an option in a Blueprint transition configuration: https://help.zoho.com/portal/en/community/topic/blueprint-multi-select-lookup-field-not-available-within-blueprint-transitionApply Credit Note Automatically
We need the ability to apply open Credit Notes toward invoices generated by recurring invoice as the first, priority payment. This should be an option that we can enable/disable in the recurring profile and/or the Credit Note. Other invoicing systems can do this. I'm not sure why Zoho Invoice doesn't have it. Here's an example for a Recurring Invoice... If a customer has open Credit Notes, and a saved credit card set for auto-pay, once an invoice goes out, the credit balance gets automatically appliedWorkflow for "Expenses" module?
Hi there, over the last 2 years, Zoho Expense has seen tremendous growth and we are happy with it. But, sometimes it is frustrating to see things are being implemented halfheartedly, or so it seems. For example, There is the possibility to create workflowsRecord Template - Conditionally printing sections
Is there a way to conditionally print a section of a Record Template? More specifically I am printing records from a Form "Invoice". That Form has 3 SubForms. I'd like to print the content of those SubForms using a Record Template but only if they have at least one line item. If they have no line items, I'd like to hide the headers for that section on the printout.User can choose the PDF report
Hi. I would like to find out if a user (Creator or customer portal) to choose from the different PDF customised reports that have been built?Query table pull last 12 months
I am tying to pull the following criteria and the date is always what causes me the issue. I want to pull people (pco_id) who have entries of "event_id" being these 2 events and whos "kind" is Regular or Guest and where the event_starts_at (date column)PLEASE FIX Search options and consider a Global seach option
A recent update has removed the ability to search for addresses and phone numbers under contacts. We cannot find where this moved to (If it is still available). Please put these options back as we cannot locate specific projects anymore. Also please consider allowing for a Global search. This would really improve the search engine. For example: If I search for "Sally Jones" then all invoices , estimates, vendors etc.. would populate.. Please let me know if you need any more information. Thank You....Customize your calendar based on personal preferences
Greetings, We're happy to introduce a few new capabilities to the Activities module's Calendar View! Now you can tailor your calendar's appearance and notification settings to suit your needs. In the past, the Calendar View lacked customization optionsBUG ALERT: Client Script + Commands -> $Page contextual data is not updated
When using the new Client Script Commands feature, there is an issue with the Client Script $Page contextual data not accurately being updated each time a Command is run. Assuming a Client Script Command called "Client Script Command Bug" with the followingShow iFrame of related List inside of Blueprint Transition
Hey, is it possible to show an iFrame of a related list like this inside of a Blueprint transition?Lookup Fields not Converting
I manage holiday properties. I have a lookup to the Accounts (Properties) in the Leads module. The lookup is connected to the property address field. When I convert it the lookup field does not update in Deals, although the property address does. There2024: A Year of Transformation with Zoho Forms
As we close the curtain on another exciting year, it’s time to reflect on the strides Zoho Forms has taken in 2024. From empowering businesses with advanced tools to simplifying workflows and enhancing user experiences, our updates this year were allStop selling out of stock Items.
Hi I have been using Zohobooks for a around 8 month now. I am not involved in selling process but my staff cant stop selling product which they do not hold in stock, this is a big headache for me as physical count never matches what is shown on the books.Bigin API Token Request ("invalid_client")
Hi people, I tried to connect to the API without success, I've read all of the documentation multiple time and tried just about everything. I tried to do it with Python Request module and with Postman, passing the information through both the URL parameterCustomer Happiness not clickable when using API
Is there a way to automatically add the Customer Feedback links when generating email drafts via the API? Currently, the feedback links are only added when generating an email draft using the UI. I tried using the endpoint described in https://desk.zoho.com/DeskAPIDocument#CustomerFeedback#CustomerFeedback_GetthecustomerfeedbackplaceholderlinkSend To Zoho Sign not Showing
The button send to Zoho sign is not showing on my Zoho CRM . Is there additional steps I need to take after installing Zoho Sign to CRM ?How to Get An Image's URL once it's uploaded to library?
I manage to find URLs to the images I uploaded to my library, but after a day, it seems the links stop working like its only temporary. Where can I find the ACTUAL solid URL for my images that I upload to my Library so I can use them for my custom template / HTML coded template? Thanks, Mac【Zoho CRM】インポート機能のアップデート：既存データへのタグ追加が可能に！
ユーザーの皆さま、こんにちは。コミュニティチームの藤澤です。 今回は「Zoho CRM アップデート情報」の中のインポート機能のアップデートをご紹介します。 Zoho CRMのタグは、データを効率的に分類、認識するためのラベルです。 タグ付けは次の3つの方法で行えます： 個別タグ付け：少数のデータを手動でタグ付け 自動化：特定のタイミングで繰り返しタグ付け 一括更新：インポート機能でタグを追加または更新 今回のアップデートでは、インポート時に既存のタグを残したまま、新しいタグの追加、既存タグを置き換えできるようになりました。Link to Desk tickets
Hello, We are using Analytics to analyze data in Desk. Is there a way to embed a link to a ticket in reports? We'd love to be able to see the drill down data, and click a value in a result row that would launch the Desk Ticket in another window/tab. Thanks in advance for any ideas!Bluerprints: How to connect the created record back?
I've a blueprint which creates another record as part of an 'After' transition. But the two records don't seem to be linked together. If it's not automatic - how do I get the created record ID to link it to the original record? Thanks!Populate a Related List Item based on a Stage
I would like to know if I can populate a CLOSED DEAL section in a contact that populates only when a Deal (something in the Deal Module) is listed as Closed - Won. I'd like another section that is just called deals, which shows me all other deals thatBeing able to draw inside a module ?
I was wondering if anyone know of a solution for this request. We would like to be able to draw directly from one module in the CRM and have it attached to that record. Here is an example. Paul would go to the customer once he as done the measuring, he2 serial numbers for 1 item (Mac address and Serial number)
There is a way to track 2 serial number type for 1 Item. Ex: Some electronic devices have a MAC address and a serial number. I need to track those 2 numbersHolidays
Hi; For defining Holidays, you need to add logic to handle the year as well as the month & day. We need to be able to enter Holidays for the next year. I need to add a holiday for January 2, 2017, but I can't until January 1st, which is a Sunday and weDelete a department or category
How do I delete a Department? Also, how do I delete a Category? This is pretty basic stuff here and it's impossible to find.Remove or hide default views
I'm looking to only have the views pertinent to my organization. Is there a way to show only my custom views (or separate them to a different area or something)? If not, this should be a feature as switching from Zendesk we had this option...Analytics Module: Can you move items from one dashboard to another?
Is there a way to move items from one dashboard to another? I want to rearrange my dashboard now that I know what i'm doing but i don't want to remake my various widgets? Edit: Hey Zoho, This would be a good feature: to be able to move/copy widgets toCopy Widget to another Dashboard
I can see the option to clone a widget to the same dashboard but is it possible to copy it to another dashboard?Is there a way to print the dashboard?
I would like the capability of printing the dashboard - is that possible?Custom Deal Name in Lead Conversion Mapping
I know there are ways to change the name of a Deal after conversion using a custom function, so no need to repost that information. I would like to see the CRM Improved with Deal Name Customisation and I think the Lead Conversion Mapping page would beIntroducing Record Summary: smarter insights at your fingertips
Hello everyone, Building on the recent launch of Zoho's in-house Zia Large Language Model (Zia LLM)—a major milestone in Zoho CRM’s AI capabilities—we’re excited to introduce the Record Summary feature. This powerful addition makes use of Zia LLM to simplifySuppress "spreadsheet will not be saved" message on published sheet
I have published a sheet and have one column on that sheet that the user can edit (a dropdown picklist where the user can select the status for each line). Is there a way to suppress the Zoho Sheet message "Any changes made to this published spreadsheetMissing "Email Authentication" tab
Backstage is alerting users about "Unauthenticated Domains" with a large prompt in the backend. It adds a link to learn more. On this tutorial page, it shows where to find the "Email Authentication" tab in this screenshot. However, in our Backstage, thatShow item Cost value on Item screen
The Item screen shows Accounting Stock and Physical Stock. It would be very helpful if value information could be displayed here as well, for instance Cost Price. Currently, to find the Cost Price (as used for inventory valuations) from inside the itemZoho Analytics to Zoho Sheets - automatic update?
Hi all, If I create a zoho sheet from an Analytics Report or Analytics Data, is there a way for the zoho sheet to automatically update as the Report / Data in analytics updates?Integration with Moodle
Greetings, I hope find all doing well and safe. I've recently returned to using Zoho Flow after a break and was hoping to connect my WooCommerce store with Moodle, the world's most widely used learning management system. My goal was to automatically enrollHow to Record Loan with interest
I have received loans from friend he give me like 2 loans so far one is one year repayment and one short, how to properly record his payment, and repayment and give him statement for each loan he give meTask status - completed - other options
I have a dumb question I know i can make custom statuses for the tasks - but is there anyway to make additional "completed" statuses like for instance if i have a task "call back customer" and i leave a vm for them to call back marking it "completed -Task module and related-to field
In modules other than the Task Module I can add several lookup fields to provide a variety of relationships. In the Task module lookup fields are not available. There is only one "related to" field which I want to use for Company. But I want to relateAdd Lookup Field in Tasks Module
Hello, I have a need to add a Lookup field in addition to the ones that are already there in the Tasks module. I've seen this thread and so understand that the reason lookup fields may not be part of it is that there are already links to the tables (https://help.zoho.com/portal/en/community/topic/custom-fields-on-task-module).Next Page