Data Synchronization using Bulk Read and Notification APIs
Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.
Welcome back to yet another insightful post in our Kaizen series.
Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.
In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.
Bulk Read API
This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.
Notification API
This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.
Flow Diagram
The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.
Step 1: Sync Start
This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.
Step 2: Subscribe to the channel via Notification API
First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.
Reason:
During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.
To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.
Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST
Sample Input
{ "watch": [ { "channel_id": "10000", "events": [ "Leads.create", "Leads.edit", "Leads.delete" ], "channel_expiry": "2024-01-21T00:13:59-08:00", "token": "leads.all.notif", "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915" } ] } |
Sample Response
{ "watch": [ { "code": "SUCCESS", "details": { "events": [ { "channel_expiry": "2024-01-21T00:13:59-08:00", "resource_uri": "https://www.zohoapis.com/crm/v6/Leads", "resource_id": "5725767000000002175", "resource_name": "Leads", "channel_id": "10000" } ] }, "message": "Successfully subscribed for actions-watch of the given module", "status": "success" } ] } |
The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.
Step 2.1: On each notification, update that record in DB along with the modified time
If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.
Sample JSON Notification Response
Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.
Step 3: Bulk Read Initialize
Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API. As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API.
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.
Sample Request for Initial Bulk Read
Request URL: {{api-domain}}/crm/bulk/v6/read
Request Method: POST
Sample Input
{ "callback": { "url": "https://www.xyz.com/callback", "method": "post" }, "query": { "module": { "api_name": "Leads", "page" : 1 } } } |
Sample Response
{ "data": [ { "status": "success", "code": "ADDED_SUCCESSFULLY", "message": "Added successfully", "details": { "id": "5725767000002002008", //job_id "operation": "read", "state": "ADDED", "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-20T04:01:48-08:00" } } ], "info": {} } |
Get the Status of Scheduled Bulk Read Job
Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET
Sample Response
{ "data": [ { "id": "5725767000002025007", "operation": "read", "state": "COMPLETED", "result": { "page": 1, "per_page": 200000, "count": 142, "download_url": "/crm/bulk/v6/read/5725767000002025007/result", "more_records": false }, "query": { "module": { "id": "5725767000000002175", "api_name": "Leads" }, "page": 1 }, "created_by": { "id": "5725767000000411001", "name": "Patricia Boyle" }, "created_time": "2024-01-24T21:35:11-08:00", "file_type": "csv" } ] } |
The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.
Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time
Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.
If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.
Channel Resubscription
The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.
Troubleshooting
There are some scenarios where data synchronization may fail.
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.
We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------
Related Links
- API Help Documentations :
- Kaizen Posts :
- Kaizen Series
Access your files securely from anywhere
Zoho Developer Community
New to Zoho LandingPage?
Zoho LandingPage Resources
New to Zoho Survey?
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Bigin?
Topic Participants
Subramanian K
Sticky Posts
Client Script | Update - Introducing Commands in Client Script!
Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! CommandsKaizen #165 : How to call Zoho CRM APIs using Client Script
Welcome back to another exciting Client Script post! In this post, we will discuss one of the most frequently asked questions: How do you call Zoho CRM APIs from Client Scripts? In this kaizen post, 1. Ways to make calls to Zoho CRM APIs using ClientKaizen#162 : Add Tags and Open Pre-filled Email Drafts via Client Scripts
Hello everyone! Welcome to another informative Kaizen post. In this post, let us see how to accomplish the following using Client Script. 1. How to auto-tag a record based on field update? 2. How to Open a pre-filled email draft This post will provideExtension Pointers - JS SDK series #4: Managing data from a widget using ZOHO.CRM.HTTP
Handling and managing the data between two applications is a major factor for an efficient business. There are a variety of ways to handle data between Zoho CRM and other third-party applications. You can use deluge CRM tasks, create connectors, use APIKaizen #152 - Client Script Support for the new Canvas Record Forms
Hello everyone! Have you ever wanted to trigger actions on click of a canvas button, icon, or text mandatory forms in Create/Edit and Clone Pages? Have you ever wanted to control how elements behave on the new Canvas Record Forms? This can be achieved
New to Zoho TeamInbox?
Zoho TeamInbox Resources
Zoho DataPrep Resources
Zoho CRM Plus Resources
Zoho Books Resources
Zoho Subscriptions Resources
Zoho Projects Resources
Zoho Sprints Resources
Qntrl Resources
Zoho Creator Resources
Zoho Campaigns Resources
Zoho CRM Resources
Design. Discuss. Deliver.
Zoho Show Resources
Get Started. Write Away!
Writer is a powerful online word processor, designed for collaborative work.
Zoho CRM コンテンツ
-
オンラインヘルプ
-
Webセミナー
-
機能活用動画
-
よくある質問
-
Ebook
-
-
Zoho Campaigns
- Zoho サービスのWebセミナー
その他のサービス コンテンツ
Nederlandse Hulpbronnen
ご検討中の方
Recent Topics
Meetups Gratuitos Junio 2024 - Profundiza en las funcionalidades de tu Zoho CRM
Este junio, aprende a sacar el máximo provecho de tu Zoho CRM en la segunda edición de los Zoho Meetups 2024. Los días 18 a 21 de junio, Zoho organiza los Meetups gratuitos para usuarios de Zoho CRM en Valencia, Barcelona, Madrid y Sevilla, donde expertosHow to get the Dashboard page to be the first page when you open the app
So when it opens on a tablet or phone it opens on the welcome page, thanks.Integration between Zoho CRM and Zoho WorkDrive
I'd like to search Zoho for an invoice I've added as an attachment (pdf) to an account. The name of the invoice is 1388-advertiserx-July.pdf - but I can't find it using the search function for any of these terms: 1388 1388-advertiserx 1388-advertiserx-July.pdfTip #17: How to mandate partial payment for your appointments
When you require partial payments during the booking process, customers can only schedule with you after paying a certain amount in advance. This deposit acts as a commitment between both parties. Apart from that, it has many more advantages. BenefitsWhy option for 'include form submission in the body of the email' check box is missing
Hi In all our forms we have configured an <Email Notification> (<Rules> <Form Rules>) In some forms, there is an option to <include form submission in the body of the email> in the <Additional Options> section, however, this option is not available inno puedo recibir correos
Hola, en la cuenta de la empresa desde hace unos tres días dejamos de recibir correos y no hemos realizado ningun cambio de configuracion, requerimos asistencia urgente por favorNO PUEDO RECIBIR CORREOS EN MI CUENTA ZOHO MAIL
LO QUE SI PUEDO HACER ES ENVIAR CORREOS PERO NO PUEDO RECIBIR CORREOS. Y LA CONFIGUIRACION DE MI DNS YA ESTA CORRECTA Y APUNTAN A ZOHO PERO AUN ASI NO PUEDO RECIBIR NECESITO AYUDA URGENTE YA QUE EN MEXICO NO HAY FORMA DE COMUNICARNOS VIA TELEFONIA CONMozilla Vault extension will not Unlock even once.
I have been using the Vault extension in Chrome based browsers for years, yet after installing the Mozilla Extension in Firefox, it will not unlock. Initially it did redirect me to log into my account, and then enter the master password. However, it didUnable to Download CRM Contact Data: WorkDrive Integration Issues
## Problem Description We need to let users download contact information from CRM as CSV files to their local computers. Since we couldn't implement a direct download option, we're trying to use WorkDrive as a workaround - but we're encountering issuesSort mail by name and subject
I don't see sort function on columns FROM and Subject. I see only sort functio by date. Could add it ?Zoho Creator monthly roundup - September 2024
Hello all, We're back with an exciting set of new features and enhancements that will elevate your Creator experience even further. In case you missed it, we’ve recently revamped our Product Roadmap page, now with a refreshed design and showcasing allKiosk Studio Session #1: View paid customers in the same industry
Update | 15 Oct 2024: Session #2 is now available here! Hello everyone! We're excited to launch our new series of posts on Kiosk Studio today. Called Kiosk Studio Sessions , these posts will be packed with actionable ideas to help you get the most outIssues hosting Zoho Desk Web Form on SharePoint and/or Power BI
Zoho Desk onboarding support has no experience with embedding their web form in either SharePoint or Power BI. Microsoft states that SharePoint and Power BI only support iframe HTML. And unfortunately, the web form embed code that Zoho generates is not"Send with Zoho Sign" broken
Our company uses hyphens in our file name conventions. Our users have been sending the files from other modules with the "Send with Zoho Sign" shortcut in the upper right buttons. Since around June 10, 2024, this stopped working. Our users can send theNot able to change colors help center
Hi. How can I change the orange color in the help center? You can change everything besides this font color And how can I remove the part on the bottom?Transform Numeric Values using st, nd, rd, th or Convert Numerals to Ordinal Form - Deluge
Please Use this sample Code This_Day_Date = zoho.currentdate.toString("dd"); value1 = "th"; if(This_Day_Date.right(1) ="1" && This_Day_Date != "11") { This_Day_Date = This_Day_Date+" "+"st"; } else if ( This_Day_Date.right(1) = "2" && This_Day_Date !=Kaizen #166 - Handling Query Variables in Zoho CRM
Hello, Code Enthusiasts! Welcome to another week of Kaizen! This week, we'll dive into handling variables in Zoho CRM Queries and see how they can be deployed in Kiosk to dynamically retrieve data. This technique is especially useful for integrating dataAutomate User Invitations on Zoho Desk with API
Automate User Invitations on Zoho Desk with API Hello Team, We are excited to announce that you can now automatically invite users to the Zoho Desk portal using the API! ### How It Works For example, when a contact is created in Zoho Desk and you enableIs there a way to add clients who don't have organisation in Zoho Books/Payroll/Expense ?
The Zoho Practice software is only allowing a total of 15 such clients who are not organisations in Zoho Books/Payroll/Expense. i.e. 5 organisation in each of the software by creating a new organization for them and adding the accountant as the adminEmail with attachments saving attachments into Zoho CRM from Zoho Mail
Hi, I get a lot of emails from prospective clients asking if we would bid their project. Those projects usually have many documents associated with them that I link to. I would like to have those documents be saved as an attachment in my Potential or Contact or Account. I don't see a way to do that that isn't multi-step. As of now I do the following: 1.) Open email 2.) If email sender isn't in my Zoho CRM database I enter them creating a Potential 3.) I download the attachment and save it to a differentFixed assets recording
Hello there, I recorded a bill for a vendor contain (Computer) so the PC is a fixed assets, do I need to do a manual journal to include this PC under the fixed assets category (furniture & equipment)? If yes, please take me through the manual journalMulti-Select lookup field has reached its maximum??
Hi there, I want to create a multi-select lookup field in a module but I can't select the model I want the relationship to be with from the list. From the help page on this I see that you can only create a max of 2 relationships per module? Is that true?Zoho Sheet-Pulling in Data
I have a module where we track POs, Bills etc (its easier for us this way). In this PO Module, we have a task related to a Job/Deal and various fields for costs. A Supplier on a Job might have several POs in this module for the job. I want to generateCan't get ZFS file ID of a field via specific module record
I have a `fileupload` field on a custom module that I need to replace the file for but I don't know how to get the ZFS ID from the module record. # Getting the current fileupload GET https://www.zohoapis.eu/crm/v7/<MODULE>/<ID> # Response { data: [{ //Contemplating moving my site from WordPress to Zoho Sites
Hi Everyone, We currently find ourselves in a situation where we ant to review and update our current sites content. We are small business owners, not developers. We currently use a wide range of Zoho products. We sometimes think about the possibility of either moving or just starting from scratch on Zoho Sites. I would like to know if anyone has done this and of course the things that need to be considered. We have spent quite a bit of time getting our current site positioned organically and I guessMultiple Salesperson against an invoice
Hello, Against a particular invoice, we have multiple sales people working. The reason we combine the invoice is becuase we are an exporter and often consolidate cargo for our customer to save them freight costs. How do I capture the contribution of eachAllow a brand to connect multiple facebook pages at the same time
Our company have multiple facebook pages to represent, every branch, right now we can only link one facebook page, so we can only recive messages from one branch. It would be nice to be able to connect all pages.Whatsapp reply not reflected under Tickets for offline agent
Hi, We are encountering a situation where WA response from customer is not reflected under tickets (if the agent handling the case previously is not active due to off day/ MC). The ticket will remain in the offline agent's queue until agent is back toSelect CRM Custom Module in Zoho Creator
I have a custom module added in Zoho CRM that I would like to link in Zoho creator. When I add the Zoho CRM field it does not show the new module. Is this possible? Do i need to change something in CRM to make it accesible in Creator?Send Whatsapp with API including custom placeholders
Is is possible to initiate a session on whatsapp IM channel with a template that includes params (placeholders) that are passed on the API call? This is very usefull to send a Utility message for a transactional notification including an order numberIncrease Round Robin Scheduler Frequency in Zoho Desk
Dear Zoho Desk Team, We hope this message finds you well. We would like to request an enhancement to the Round Robin Scheduler in Zoho Desk to better address ticket assignment efficiency. Current Behavior At present, the Round Robin Scheduler operatesSeparate One-Time Mentions from Ongoing Ticket Following in Zoho Desk
Dear Zoho Desk Support Team, We are writing to request an enhancement to the mention functionality within Zoho Desk. Currently, when an agent is mentioned in a ticket, they receive notifications for all subsequent activity on that ticket, similar to beingMass update Accounts in Zoho Desk
We used the integration feature to bring in all of our accounts from Zoho CRM. While all of the names and links are in place, on the Zoho Desk side the email, phone, website, and address fields are blank. By going into each account I can select "UpdateVivaldi will not open the Zoho Mail app when I click a email link in the browser
I'm running Vivaldi on Arch with the Plasma desktop and have under default applications Zoho Mail set as my default email client. Whenever I click a email link in Vivaldi I get the below when it should be launching Zoho. Any ideas on rectifying this issue?Digital publisher
Kinetic Digital Publisher provides comprehensive digital publishing services, including eBook creation, formatting, cover design, and marketing. We help authors publish and promote their work across online platforms with ease.Time Zone Correction with Daylight Savings Time
Hey, I'm writing a series of reports where the source data is synced from an external source which stores the date/time information in GMT, but I want the reports to be in local time. Now, I understand there is the CONVERT_TZ function which allows meLoad form in iframe without header
I am trying to load a form into an iframe without the header, but I am not having any luck. I am using openUrl() to load the iframe with the form URL and zc_Header set to false, e.g. #Form:Add_Case?zc_Header=false but it is still loading the header. AnyFilter embedded report
How to filter embedded report in a page, below code is not working. dateField => startDate & dateField=< endDate The report should print on page containing records from startDate to endDate. params='zc_Header=true&Service_Date__gte=<%=startDate%>&Service_Date__lte=<%=endDate%>'Creator Simplified #5: Set file upload restrictions in Zoho Creator
Hey Creators, Welcome to the next post in the Creator Simplified series. Today, we’ll explore how to implement file upload restrictions to limit user submissions to specific file types. By implementing an allowed list for file uploads, you can optimizeField Type: Address, Change District/State to Dropdown with picklist??
Using the Address Field type, is there a way to make the District/State field be a dropdown with a picklist so the users can select rather than type the state name every time? I know this can be done if I use a drowdown field for the State (or entire address information), but that isn't as tidy as using the address field type. I apologize if this is a duplicate. I posted this question the other day, or so I though. Can't find that post.Next Page