Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho Campaigns Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Subform Time field to string.
Good afternoon All. I have a Subform 'Delivery_Receiving_Hours' that captures Day (Dropdown), Time_Open (Time), and Time_Close (Time). I need to capture this data and send it to a multiline field in the CRM. The code, posted below, below will captureworkflow for bounced email gets triggered, but email is status = opened
Hello, I have a workflow that sends me an email if outgoing email are bounced. Now I got some kind of this emails, but the corrosponding contacts have status = open at the email. Why this bounce-workflow is triggered? Reports > Email Reports > BounceData export
I need to export our customer's data and projects' data for our purpose but am unable to export full data i only get around 3160 projects and around 2k customer can you please help me to get full data, pleaseAdjusting Physical Inventory
Not getting very far with support on this one, they say they are going to fix it but nothings happened since November. Please give this a thumbs up if you would like to see this feature or comment if you have some insight. Use Case: Inventory set to beZoho Marketing Plus : Un outil tout-en-un pour la création de pages, la collaboration et la gestion du calendrier marketing
Nous sommes ravis de vous présenter trois nouvelles fonctionnalités puissantes de Zoho Marketing Plus s’enrichit désormais d’un page web (l'éditeur de pages), qui vous permet de créer des pages attrayantes et à fort taux de conversion pour vos campagnesGrouping payments to match deposits
Is there a way to group multiple invoice payments together so they match credit card batches and grouped deposits in the bank account? Basically, we are creating invoices for each of our transactions, and applying a payment to each of the invoices. Our payments are either credit cards or checks. We want to be able to group payments together so when our bank account reflects a credit card batch made up of many transactions, or the deposit we took to the bank that has multiple checks from differentEmployees can not add some expenses suddenly
Zoho expense was working fine and whenever there was a new merchant, it would automatically add and also the same auto added in Zoho Books (due to merchant-vendor sync) untill now. From today, it is having problems in searching the existing vendors andZoho email setup in office365
When i am trying to setup zoho mail setup using my domain in office365 and it is not working and it says that we couldn't log on to the incoming (IMAP) server and please check your email address and password and try again. I was able to login using myiOS 10: Caller ID new feature?
Hi, in the update history of the iOS App (for iOS10) - v.3.2 - i found the point "caller identification" has this feature been deactivated again? i cannot find anything on my iphone on how to activate this feature. or does it just work from the beginning?Recommendations to store meeting notes for easy access from Contacts, Accounts & Deals module records?
I would like your advice on how to achieve this use case for my organization. It’s related to where/how best to store meeting notes from a conversation with Contact(s) working at an Account (Company) in the context of a Deal. The ideal solution (fromBank reconciliation. Match Transaction -filter
When matching an imported bank statement file we only get a match if it is an excact match on both amount and date. Then a suggestions comes up with a very broad selection regarding amount, and no default "between" dates. I can then go an manually adjust the filter, and have to put in from-to amounts and dates. How do I set a default from-to date? As an example, I would like the date to be +- 3 days, Thanks.Added new staff but does not appear in other organization list
Hi, I added the new staff under Sales Manager in the contacts, but it does not appear in the other organization list where I need to create a contact, and I can't select the newly added Sales ManagerIntegrating Calendly with Zoho Calendar in Zoho Mail
I moved my office into a business incubator space that uses Calendly for meeting management and events. Calendly doesn't have a integration with Zoho Calendar and vice versa. I was directed to Zapier for integration but it doesn't have an integrationMap fields from module X to a lookup field in subform in module Y
Hi there In the 1st screenshot attached, you can see a subform in myLeads module. You can see that there is a number already filled there - that is the 'Property ID' and it is a single line field. It is the 'Property ID' of an entry I have in another🎄 Jingle, Mingle, and Automate: Spread Christmas Cheer with Zoho Desk Auto-Replies! 🎄
Hello Everyone! Welcome to this week's episode of the Community Learning Series. Christmas is in the air, and I’m sure we can all feel the jingle and the mingle of the season! The folks at Zylker Techfix are no exception—they’re busy with holiday planshow to create a new line in string in Client Script?
I want to show an alert using client script, I need to add a new line in String, I assume I can use \n\n inside a string, but unfortunately it doesnt work ZDK.Client.showAlert("First Line \n\nI expect this is in second line");Surely it's time Inline editing from views
I think the first request I found for in-line editing from grids was approximately 12 years ago - that post was locked because it was suggested Zoho sheetview solved the problem. However, it's now 2024, and in-line editing from grids is just a basic expectation.Multi branding issue with sender addresses
Hello, I'm currently working on a project involving two (seperate) brands. Named 'Windeck' and 'Prolance'. They've chosen CRM Plus and I'm currently working on CRM, SalesIQ, Social and Marketing Automation. So far, I'm able to make enough separationsHow to Replace an Assessment in a Job Opening on Zoho Recruit
Hi everyone, I’m currently using Zoho Recruit and would like to replace the assessment linked to a specific job opening. I want to remove the existing assessment and add a new one. What is the best way to do this without losing any important data or affectingIs there API Doc for Zoho Survey?
Hi everyone, Is there API doc for Zoho Survey? Currently evaluating a solution - use case to automate survey administration especially for internal use. But after a brief search, I couldn't find API doc for this. So I thought I should ask here. ThanEmail Campaigns overview page is missing SENT DATE and # people sent to!
I would like to see the date the email campaign was sent, so I can understand and track when each email campaign was sent. Right now, unless you go to a contact who received a campaign, you cannot see when the campaign was sent (!!!!!!). So, if my bossSEO recommendation of H1 tag for website tittle
The exact words are “ It is good practice to place the page title inside the H1tag.” Now I already have one H1 tag on my website but it is not website tittle. In the SEO recommendation that is clear too that I have h1 tag on my page. Now I don’t knowHow to choose other payment methodes than creditcards
We have connected stripe as a payment provider in zoho books, booking, commerce and checkout. In stripe we selected al major payment methodes for Belgium (mainly bancontact). However, at checkout customers seems to have only the possibility to pay withIntroducing Zia LLM: Zoho’s in-house Generative AI solution for CRM's AI capabilities
Hello everyone, We're excited to announce the launch of our in-house Large Language Model (LLM) by Zia to power our AI offerings. What is LLM? LLM stands for Large Language Model, a powerful AI technology that processes and generates human-like text basedHow to call a Creator function which is in a different Creator application?
How to call a Creator function which is in a different Creator application?Can the code in my "Successful form submission" WF be invoked from a function?
Can "Successful form submission" be invoked from a function? Data gets into a form manually and programatically. My code in "successful form submission" is good and I want to reuse it/call it, from another function which does Insert Into How to achieveKaizen #169 - Serialization and Schema Management in Queries
Hello everyone! Welcome back to another post in the Kaizen series! In Kaizen #166, we discussed handling Variables in Queries and associating the query in Kiosk. This week, we will discuss Serialization and Schema management in Queries. Business ScenarioIntroducing Keyboard Shortcuts for Zoho CRM
Dear Customers, We're happy to introduce keyboard shortcuts for Zoho CRM features! Until now, you might have been navigating to modules manually using the mouse, and at times, it could be tedious, especially when you had to search for specific modulesFeature Request: Notebooks within notebooks (Tree-like structure)
Dear Zoho! I already migrated all my stuff from Google Keep, Im really fond of Zoho Notebook so far. One thing that could make the service much more powerful is multi-level notebooks (or tree like structure). For example, entering into Notebook namedCan't get authorization for Sandbox environment using the self client
Hello, After creating a self client, and following the client-credentials method (as it's not optimat to manually generate a code for every 10 minutes), after inputting the sandbox org id for SOID parameter, im getting the error: "error": "no_org". ForCreate landing pages from Zoho Marketing Plus
Hey everyone, Over the last few months, we've introduced various features and enhancements to bolster the marketing capabilities of Zoho Marketing Plus and make it simpler for everyone. To that end, we're excited to announce that Zoho LandingPage is nowCustom service report or Zoho forms integration
Hello, So far the experience with Zoho FSM and the integration with Books has been good, however there are limitations with service reports. As with my business, many organisations send technicians to different types of jobs that call for a differentEmail tracking subdomain
The Email Tracking configuration screen of the ZeptoMail asks for a subdomain. I have gone through the documentation but could not find more information about how that subdomain is used by ZeptoMail to track the emails. Can someone throw some light aboutChart View group X-axis values above a value
I have a data set with X values ranging from 0 up to 300-400, the Y values are an AVG of the values for the given X. I am interested in the values at the low end of the scale, say 0-10 and want the X values 10 and greater to be grouped into a single categoryHow do I get the Text Account name instead of the Reference number?
Good Morning everyone! I am very new to zoho analytics. I'm trying to create some pivot tables and when I add the Account or Division name it comes up as a reference number. From my research, I need to use a lookup. I'm having a really hard time understandingCustom API - Need to create a string return value, not only MAP
@Support: When creating a Custom API it only allows a return from a function of MAP type. The service I'm using requires a string return, how can this be achieved?missing video-urgent
hi..I have a problem regarding zoho meeting. I already record almost 2hrs for my interview session. After end my session, i'm stop the recording. Its happened when i didn't received any meeting recording at my email.But i received recording 7minutes afterDuplicate New Ticket Emails
I am getting two Ticket emails when a ticket is created. The first one goes to me as the Admin helpdesk@ And because I am an Agent as well it also send to helpdesk@, support1@ and support2@ Is there any way to stop this?Ticket Response Email Template
Is there a way to have an email template automatically selected when I click on Reply for a ticket?Unpublishing Job Post for LinkedIn
Hi, We have experienced issue with the above. We did unpublish a job in Recruit but it still appear under Job section in our LinkedIn page. It did not remove from LinkedIn as it should be. Please check and advise. Regards, SnyderNext Page