Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.

To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch

Request Method: POST

Sample Input

{     "watch": [         {             "channel_id": "10000",             "events": [                 "Leads.create",                 "Leads.edit",                 "Leads.delete"             ],             "channel_expiry": "2024-01-21T00:13:59-08:00",             "token": "leads.all.notif",             "notify_url": "https://webhook.site/6c3d08c2-2c21-4d16-9134-712acb513915"         }     ] }

Sample Response

{     "watch": [         {             "code": "SUCCESS",             "details": {                 "events": [                     {                         "channel_expiry": "2024-01-21T00:13:59-08:00",                         "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",                         "resource_id": "5725767000000002175",                         "resource_name": "Leads",                         "channel_id": "10000"                     }                 ]             },             "message": "Successfully subscribed for actions-watch of the given module",             "status": "success"         }     ] }

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 

Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read

Request Method:  POST

Sample Input

{     "callback": {         "url": "https://www.xyz.com/callback",         "method": "post"     },     "query": {         "module": {             "api_name": "Leads",             "page" : 1         }     } }

Sample Response

{     "data": [         {             "status": "success",             "code": "ADDED_SUCCESSFULLY",             "message": "Added successfully",             "details": {                 "id": "5725767000002002008", //job_id                 "operation": "read",                 "state": "ADDED",                 "created_by": {                     "id": "5725767000000411001",                     "name": "Patricia Boyle"                 },                 "created_time": "2024-01-20T04:01:48-08:00"             }         }     ],     "info": {} }

For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.

Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}

Request Method: GET

Sample Response

{     "data": [         {             "id": "5725767000002025007",             "operation": "read",             "state": "COMPLETED",             "result": {                 "page": 1,                 "per_page": 200000,                 "count": 142,                 "download_url": "/crm/bulk/v6/read/5725767000002025007/result",                 "more_records": false             },             "query": {                 "module": {                     "id": "5725767000000002175",                     "api_name": "Leads"                 },                 "page": 1             },             "created_by": {                 "id": "5725767000000411001",                 "name": "Patricia Boyle"             },             "created_time": "2024-01-24T21:35:11-08:00",             "file_type": "csv"         }     ] }

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.

When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 

For instance,

1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.

2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.

3. The code logic in the Webhook URL may break due to unforeseen reasons.

To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com

Stay tuned for more insights in our upcoming Kaizen posts!

------------------------------------------------------------------------------------------------------------------------------

Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script

-------------------------------------------------------------------------------------------------------------------------------

Related Links

• API Help Documentations : 
  
• Bulk Read API
  
• Notification API
  
• Kaizen Posts :
  
• #12 Bulk Read API 
  
• #14 Notification API
  
• Kaizen Series

• Topic Participants

• Subramanian K

  

• Sticky Posts

• Kaizen #197: Frequently Asked Questions on GraphQL APIs

  🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

  Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

• Kaizen #193: Creating different fields in Zoho CRM through API

  🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Client Script | Update - Introducing Commands in Client Script!

  Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

• Recent Topics

• Always display images from this sender – Is this feature available?

  In Zoho mail, I had my "Load external images" setting set to "Ask me", and that's fine. That's the setting I prefer. What's not fine though is I always need to tick "Display now" for each email I get, regardless if I've done that multiple times from several

• Function #9: Copy attachments of Sales Order to Purchase Order on conversion

  This week, we have written a custom function that automatically copies the attachments uploaded for a sales order to the corresponding purchase order after you convert it. Here's how to configure it in your Zoho Books organization. Custom Function: Hit

• Free webinar: Security that works: Building resilience for the AI-powered workforce

  Hello there, Did you know that more than 51% of organizations worldwide have experienced one or more security breaches, each costing over $1 million in losses or incident response? In today’s threat landscape, simply playing defense is no longer enough.

• "Subject" or "Narration"in Customer Statement

  Dear Sir, While creating invoice, we are giving in "Subject" the purpose of invoice. For Example - "GST for the month of Aug 23", IT return FY 22-23", "Consultancy", Internal Audit for May 23". But this subject is not coming in Customer Statement. Only

• Apply Vendor Credit Automatically

  Hello!!! Is there a way where in we can apply vendor credits automatically on the FIRST OUTSTANDING BILL of the vendor?? We have lots of VENDOR CREDITS ISSUES mostly!!! Applying it manually is a pain for us. Would be great if we have a way to apply the

• Zoho Notebook Sync problem

  I'm facing a problem with syncing of notebook on android app. It's not syncing. Sometimes it syncs after a day or two.  I created some notes on web notebook but it's not syncing on mobile app. Please help!!!!

• Apply Vendor Credits Automatically

  We are bulk importing Vendor credits in Zoho Books!!! Is there a way to apply vendor credits automatically to the first UNPAID bill of the Vendor?

• Cant Save Gauge on Creator Page

  How to Save Gauge on Creator Page see movie https://vimeo.com/1116410860?share=copy#t=0

• Apply Advance option not shown in report

  We are facing an issue in Zoho Expenses. While approving an Expense Report, the "Apply Advance" option is not appearing under the three dots (More Options). Details: Module: Expense Reports Issue: "Apply Advance" option not visible Status of Report: Awaiting

• Auto Capitalize First Letter of Words

  Hi I am completely new to ZOHO and am trying to build a database. How can i make it when a address is entered into a form field like this: main st it automatically changes is to show: Main St Thank You

• Follow-up emails via Workflow Automation not staying in the same thread

  Dear Zoho Support Team, I am currently using Workflow Automation in Zoho Campaigns to send follow-up emails. In my test case, I noticed the following behavior: All emails in the automation have the same subject line. If the follow-up email is sent within

• Client Script refuses to set an initial value in Subform field

  I tried a very simple, 1 line client script to set a default value in a custom subform field when the "Add Row" button is clicked and the user is entering data. It does not work - can someone tell me why? ZDK documentation suggests this should be doable.

• Emails Are Not Being Delivered to My Inbox

  Hello Zoho Support Team, I am experiencing an issue with my Zoho Mail account. The most important problem is that emails are not being delivered to my inbox. Details: My Zoho Mail address: info@coreforcelife.com What happens: I am not receiving any incoming

• Help Center IFrame Issue

  I have had a working Help Center on my website using an iframe for a while. But now for some reason the sign in page gets a refused to connect error. Can someone please help. If I go to the url manually it works correclty

• Comment Templates

  Is it possible to add a template option for comments? We have some agents in the process who's responses require a pre-formatted layout. It would be incredibly handy to have a template for them where they can insert the template and then add their responses

• [ZohoDesk] Improve Status View with a new editeble kanban view

  A kanban view with more information about the ticket and the contact who created the ticket would be valueble. I would like to edit the fields with the ones i like to see at one glance. Like in CRM where you can edit the canvas view, i would like to edit

• Adding Markdown text using Zoho Desk API into the Knowledge Base

  Hi Zoho Community members, We currently maintain the documentation of out company in its website. This documentation is written in markdown text format and we would like to add it in Zoho Knowledge Base. Do you know if there is REST API functionality

• An Exclusive Session for Zoho Desk Users: AI in Zoho Desk

  A Zoho Community Learning Initiative Hello everyone! This is an announcement for Zoho Desk users and anyone exploring Zoho Desk. With every nook and corner buzzing, "AI's here, AI's there," it's the right time for us to take a closer look at how the AI

• Shared values: From classroom lessons to teaching moments in customer service

  While the world observes Teachers’ Day on October 5, in India, we celebrate a month earlier, on September 5, to mark the birth anniversary of Dr. Sarvepalli Radhakrishnan, a great teacher, renowned scholar, educationist, and advocate for empowerment.

• Create a list of customers who participated in specific Zoho Backstage events and send them an email via Zoho CRM

  How to create a list of customers who participated in specific Zoho Backstage events and send them an email via Zoho CRM? I was able to do a view in CRM based on customer that registered to an event, but I don't seems to be able to include the filter

• Zoho Desk blank page

  1. Click Access zoho desk on https://www.zoho.com/desk/ 2. It redirects to https://desk.zoho.com/agent?action=CreatePortal and the page is blank. Edge browser Version 131.0.2903.112 (Official build) (arm64) on MacOS

• I hate the new user UI with the bar on the left

  How can I reverse this?

• Question regarding import of previous deals...

  Good afternoon, I'm working on importing some older deal records from an external sheet into the CRM; however, when I manually click "Add New Deal" and enter the pertinent information, the deal isn't appearing when I look at the "Deals" bar on the account's

• Client Script also planned for Zoho Desk?

  Hello there, I modified something in Zoho CRM the other day and was amazed at the possibilities offered by the "Client Script" feature in conjunction with the ZDK. You can lock any fields on the screen, edit them, you can react to various events (field

• One person/cell phone to manage multiple accounts

  Hi. I have a personal Free account to keep my own domain/emails. Now I need to create a Business account to my company's own domain, but I have only one mobile phone number I use to everything. How do I do to manage this? Can I manage a Free domain and

• Tracking KPIs, Goals etc in People

  How are Zoho People users tracking employee targets in People? For example, my marketing assistant has a target of "Collect 10 new customer testimonials every month". I want to record attainment for this target on a monthly basis, then add it to their

• Zoho Desk: Ticket Owner Agents vs Teams

  Hi Zoho, We would like to explore the possibility of hiding the ‘Agents’ section within the Ticket Owner dropdown, so that we can fully utilise the ‘Teams’ dropdown when assigning tickets. This request comes from the fact that only certain agents and

• Can not Use Attachment Button on Android Widget

  this always pops up when I touch the attach button on android widget. going to settings, there is no storage permission to be enabled. if I open the app, and access the attach feature there, I can access my storage and upload normally.

• Announcing new features in Trident for Mac (1.24.0)

  Hello everyone! Trident for macOS (v.1.24.0) is here with interesting features and thoughtful enhancements to redefine the way you plan and manage your calendar events. Here's a quick look at what's new. Create calendar events from emails. In addition

• Vendor Master Enhancements for Faster Purchase Entry

  I’d like to suggest a few features that will improve accuracy and speed during purchase voucher entry: Automated Item Tax Preference in Vendor Master Add an option to define item tax preference in the vendor master. Once set, this preference should automatically

• Mass Mail Statistics - Number of unsent emails

  How do I find out which emails were not sent?

• Est-il possible d'annuler l'envoi d'un mail automatique ?

  Bonjour, Lorsque je refuse un candidat, il reçois un mail dans les 24h pour l'informer que sa candidature n'est pas retenue. J'ai rejeté un candidat par erreur. Savez-vous s'il possible d'annuler l'envoi de ce mail ? Merci d'avance pour votre aide.

• embed a form in an email

  Hello, how to embed a form in an email that populates Zoho CRM cases? I would like to send emails to a selected audience offering something. In the same email the recipients - if interested - instead of replying to can fill in a Zoho CRM form that creates

• Zoho Bookings - Reserve with Google

  Does Zoho Bookings plan to to integrate with Reserve with Google?

• How to add Zoho demo site page designs to my Zoho Sites website

  Hi, I would like to add the design from the following demo URLs into my current Zoho website. I have already created two new pages on my site, named “Menu2” and “Menu3.” For the “Menu2” page, I want to use the design from this demo: https://naturestjuice-demo.zohosites.com/menu

• Digest Août - Un résumé de ce qui s'est passé le mois dernier sur Community

  Bonjour chère communauté ! Voici le résumé tant attendu de tout ce qui a marqué Zoho le mois dernier : contenus utiles, échanges inspirants et moments forts. 🎉 Découvrez Zoho Backstage 3.0 : une version repensée pour offrir encore plus de flexibilité,

• Text snippet

  There is a nice feature in Zoho Desk called Text Snippet. It allows you to insert a bit of text anywhere in a reply that you are typing. That would be nice to have that option in Zoho CRM as well when we compose an email. Moderation Update: We agree that

• Kaizen #206 - Answering your Questions | Displaying Related Purchase Orders from Zoho Books in CRM Deals using Queries

  Hello everyone! We're back with another post in the Kaizen series. We're grateful for the feedback we received from all of you! One of the questions we received was "I would like to see the list of Purchase Orders in Zoho Books for a Deal in CRM." We

• Add Analytics function for Title case (capitalising each word in a string)

  At present, you can only capitalise each word in a string in Analytics during data import. It would be really useful to be able to do this with a formula column, but there is no Title Case function.

• How to conditionally embed an own internal widget with parameters in an html snippet?

  Hello everyone, I'm trying to create a dynamic view in a page using an HTML snippet. The goal is to display different content based on a URL parameter (input.step). I have successfully managed to conditionally display different forms using the following

• Next Page