Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Kaizen #122 - Data Synchronization using Bulk Read and Notification APIs

Hello everyone!
Welcome back to yet another insightful post in our Kaizen series.

Consider that your organization is involved in selling laptops. Your sales team uses Zoho CRM for their daily activities, managing Leads, Deals, and inventory modules, whereas your Accounts team relies on a legacy application. It is important to maintain data synchronization from Zoho CRM to the legacy system to ensure that the Accounts team always works with the latest information.
To ensure that this synchronization happens in near real-time, the legacy application uses Zoho CRM's Bulk Read API and Notification API.

In this article, we'll discuss how one-way data sync will be achieved for the "Leads" module.

Bulk Read API 

This asynchronous API facilitates the export of up to 200,000 records from a module in a single job. If there are more than 200,000 records, additional API calls are required to fetch all the records.

Notification API

This API allows you to subscribe to events such as creating, updating, and deleting a record in a module through a webhook URL. Each of these actions triggers a corresponding HTTP request to your webhook.

Flow Diagram

The above diagram represents the initial sync on how the legacy application syncs data from Zoho CRM.

Step 1: Sync Start

This is the starting point of the code for the data sync from Zoho CRM to the legacy application and keeping the data updated in near real-time.

Step 2: Subscribe to the channel via Notification API

First, subscribe to events (create, update, and delete) occurring in the Leads module through a channel created via the Notification API, and then trigger the Bulk Read API (Step 3) to synchronize the entire data in Zoho CRM.

Reason:

During the asynchronous execution of the Bulk Read API job, all records from the specific module are saved to a CSV file. Let's consider a scenario where a record that has already undergone processing by the Bulk Read API is updated through the web UI or any other source. This means that the data in the Bulk Read CSV may become outdated by the time the job concludes.

To address this issue, you can leverage the Notification API. You subscribe to a channel, actively monitoring any actions performed on records in that module. When an action occurs, you will receive a notification through the configured URL. Subsequently, upon receiving this notification, you can store the updated data in the legacy database, including the modified time. This ensures that only the most recent data is stored in the database.

Sample Request URL: {{api-domain}}/crm/v6/actions/watch
Request Method: POST

Sample Input
{
    "watch": [
        {
            "channel_id": "10000",
            "events": [
                "Leads.create",
                "Leads.edit",
                "Leads.delete"
            ],
            "channel_expiry": "2024-01-21T00:13:59-08:00",
            "token": "leads.all.notif",
        }
    ]
}

Sample Response
{
    "watch": [
        {
            "code": "SUCCESS",
            "details": {
                "events": [
                    {
                        "channel_expiry": "2024-01-21T00:13:59-08:00",
                        "resource_uri": "https://www.zohoapis.com/crm/v6/Leads",
                        "resource_id": "5725767000000002175",
                        "resource_name": "Leads",
                        "channel_id": "10000"
                    }
                ]
            },
            "message": "Successfully subscribed for actions-watch of the given module",
            "status": "success"
        }
    ]
}

The response confirms a successful subscription for actions-watch on the "Leads" module. For more information about the Notification API, refer to this Kaizen post on the Notification API.

Step 2.1: On each notification, update that record in DB along with the modified time

If a record gets updated or a new record is created, or a record is deleted in the Leads module, you are notified about the change in data via the subscribed notification channel and stored in the legacy database along with the modified time.

Sample JSON Notification Response 

Note: The primary purpose of subscribing to the channel via notification is not to miss any data that has been modified or created during the data backup.

Step 3: Bulk Read Initialize

Initiate the data backup by using the Bulk Read API after subscribing to the channel via Notification API.  As the API is an asynchronous one, a bulk read job is scheduled. After the job is completed on the Zoho CRM end, a notification will be sent via the callback URL. Also, the application can periodically check the job status using the Get the Status of the Bulk Read Job API. 
Note: For the Bulk Read API, the records will be sorted based on the id field in ascending order.

Sample Request for Initial Bulk Read

Request URL:  {{api-domain}}/crm/bulk/v6/read
Request Method:  POST

Sample Input
{
    "callback": {
        "url": "https://www.xyz.com/callback",
        "method": "post"
    },
    "query": {
        "module": {
            "api_name": "Leads",
            "page" : 1
        }
    }
}

Sample Response
{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully",
            "details": {
                "id": "5725767000002002008", //job_id
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "5725767000000411001",
                    "name": "Patricia Boyle"
                },
                "created_time": "2024-01-20T04:01:48-08:00"
            }
        }
    ],
    "info": {}
}
For more details and examples, refer to the Create Bulk Read Job API help document. 

Get the Status of Scheduled Bulk Read Job

Check the status of the scheduled bulk read job using the job_id you received.
Request URL: {{api-domain}}/crm/bulk/v6/read/{job_id}
Request Method: GET

Sample Response
{
    "data": [
        {
            "id": "5725767000002025007",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "per_page": 200000,
                "count": 142,
                "download_url": "/crm/bulk/v6/read/5725767000002025007/result",
                "more_records": false
            },
            "query": {
                "module": {
                    "id": "5725767000000002175",
                    "api_name": "Leads"
                },
                "page": 1
            },
            "created_by": {
                "id": "5725767000000411001",
                "name": "Patricia Boyle"
            },
            "created_time": "2024-01-24T21:35:11-08:00",
            "file_type": "csv"
        }
    ]
}

The above response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url. The download_url in the callback response from which you can download the zip file containing the CSV file.

Step 4: Sync the Bulk Read data with the legacy DB, if it has a more recent modified time

Once the bulk-read data CSV is available, the data will be updated in the database. If any record's modified time in the CSV file is less than the modified time already present in the database, then it's an outdated record and need not be updated in the legacy's database, as it has already been handled via the Notification API's subscribed channel.

If the CSV contains exactly 200,000 records, there is a possibility of having more records. In the response of the Status of the Bulk Read Job API, if the more_records key is true, it indicates additional records to be exported. To retrieve them, simply update the value of the page key in the POST request and schedule another bulk read job to fetch the next set of records. By default, you can fetch up to 200,000 records in a single API call. Repeat this process until all the records are fetched from the CRM side. For more details on how to download, refer to this Kaizen post about the Bulk Read API.

Channel Resubscription

The channel subscription will remain active for a maximum of one day. After this period, it needs to be re-subscribed using the Enable Notification API to continue receiving notifications for create/update/delete actions in the Leads module. It is recommended to perform the re-subscription every 23 hours and 55 minutes, just short of 24 hours. Note that if the expiry time is not specified during the subscription, it will expire within one hour.

Troubleshooting

There are some scenarios where data synchronization may fail. 
For instance,
1. Unreachable Webhook URL: If the webhook URL is down, Zoho CRM cannot notify data updates through it.
2. Notification Expiry : If the notification expires before resubscribing to the channel, there is a risk of losing newly registered leads in between.
3. The code logic in the Webhook URL may break due to unforeseen reasons.
To handle such scenarios, store the last successful data sync time. Use this stored time as the modified time criteria in the bulk-read API to fetch any missed data and update the database.

We trust that this post meets your needs and is helpful. Let us know your thoughts in the comment section or reach out to us at support@zohocrm.com
Stay tuned for more insights in our upcoming Kaizen posts!
------------------------------------------------------------------------------------------------------------------------------
Previous Kaizen Post : Kaizen #121 - Customize List Views using Client Script
-------------------------------------------------------------------------------------------------------------------------------

Related Links





    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • Show Attachments in the customer portal

      Hi, is it possible to show the Attachments list in the portal for the particular module? Bests.

    • Zoho CRM Formula - Current Time minus Date/Time field

      Hello, I am trying to prevent duplicate emails going to clients when more than 1 deal is being updated. To do this, I would like to create a formula to identify if a date/time field is >= 2 hours ago. Can someone please help me write this formula? Example:

    • Does Zoho Docs have a Line Number function ?

      Hi, when collaborating with coding tasks, I need an online real time share document that shows line numbers. Does Zoho's docs offer this feature ? If yes, how can I show them ? Regards, Frank

    • Feature Request - Insert URL Links in Folders

      I would love to see the ability to create simple URL links with titles in WorkDrive. or perhaps a WorkDrive extension to allow it. Example use case: A team is working on a project and there is project folder in WordDrive. The team uses LucidChart to create

    • Organization Emails in Email History

      How can I make received Org Emails to show up here?

    • How to sync from Zoho Projects into an existing Sprint in Zoho Sprints?

      Hi I have managed to integrate Zoho Projects with Zoho Sprints and I can see that the integration works as a project was created in Zoho Sprints. But, what I would like to do is to sync into an existing Zoho Sprints project. Is there a way to make that

    • how to differentiate if whatsapp comes from certain landing page?

      I create a Zobot in SalesIQ to create a Whatsapp bot to capture the lead. I have 2 landing pages, one is SEO optimized and the other want is optimized for leads comes from Google Ads. I want to know from which landing page this lead came through WhatsApp

    • How to record company set up fees?

      Hi all, We are starting out our company in Australia and would appreciate any help with setting up Books accounts. We paid an accountant to do company registration, TFN, company constitution, etc. I heard these all can be recorded as Incorporation Costs, which is an intangible asset account, and amortised over 5 years. Is this the correct way to do it under the current Australian tax regulations? How and when exactly should I record the initial entry and each year's amortasation in Books? Generally

    • How to create a drop down menu in Zoho Sheets

      I am trying to find out, how do I create a drop down option in Zoho sheet. I tried Data--> Data Validation --> Criteria --> Text  --> Contains. But that is not working, is there any other way to do it.  Thanks in Advance.

    • Show Payment terms in Estimates

      Hi,  we are trying to set up that estimates automatically relates payment terms for the payment terms we introduced on Edit contact (Field Payment terms).  How can it be done? Our aim is to avoid problems on payment terms introduced and do not need to introduce it manually on each client (for the moment we are introducing this information on Terms and Conditions.  Kind Regards, 

    • How can I calculate the physical stock available for sale?

      Hey Zoho Team,  I've tried to calculate the physical stock on hand in various ways - but always receive a mismatch between what's displayed in Zoho Inventory & analytics.  Can you please let me know how the physical stock available for sale is calculated?

    • When dispatched to crew, assigning lead missing

      Hello, For the past two or three weeks, whenever an officer assigns Service Appointment to a team, the lead person is missing from the assigned service list. Therefore, we have to reschedule the SA and then the lead person becomes visible in the assigned

    • open word file in zoho writer desktop version

      "How can I open a Microsoft Word (.doc or .docx) file in Zoho Writer if I only have the file saved on my computer and Zoho Writer doesn't appear as an option when I try 'Open with'? Is there a way to directly open the .doc file in Zoho Writer?"

    • I want to transfer the project created in this account to another account

      Dear Sir I want to transfer the project created in one account to another account

    • Inactive User Auto Response

      We use Zoho One, and we have a couple employees that are no longer with us, but people are still attempting to email them. I'd like an autoresponder to let them no the person is no longer here, and how they can reach us going forward. I saw a similar

    • Weekly Tips : Customize your Compose for a smoother workflow

      You are someone who sends a lot of emails, but half the sections in the composer just get in your way — like fields you never use or sections that clutter the space. You find yourself always hunting for the same few formatting tools, and the layout just

    • Zoho Slowness - Workarounds

      Hi all, We've been having intermittent slowness and Zoho just asks for same stuff each time but never fix it. It usually just goes away on it's own after a couple weeks. Given that speed is a very important thing for companies to be able to keep up with

    • Custom Bulk Select Button

      Zoho CRM offers the ability to select multiple records and invoke a Custom Button This functionality is missing from Recruit Currently we can only add buttons in the detail page and list But we cannot select Multiple Records and invoke a function with

    • Zoho CRM still doesn't let you manage timezones (yearly reminder)

      This is something I have asked repeatedly. I'll ask once again. Suppose that you work in France. Next month you have a trip to Guatemala. You call a contact there, close a meeting, record that meeting in CRM. On the phone, your contact said: "meet me

    • Power of Automation :: Smart Ticket Management Between Zoho Desk and Projects

      Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate

    • First day of trying FSM in the field.

      What we found. 1. with out a network connection we were unable to start a service call? 2. if you go to an appointment and then want to add an asset it does not seem possible. 3. disappointed not to be able to actually take a payment from within the app

    • BUG - Google Business Buttons - Add a button to GBP Post

      I am experiencing an issue with the "Add a button" feature when creating posts for my Google Business Profile (GBP) through Zoho Social. When I schedule or publish a GBP post and include a call-to-action button with a specific URL, the post itself publishes

    • Rich text Merge field - Not using font specified in HTML

      I have a rich text merge field in a writer template which is creating a table. I have chosen to use this method instead of a repeat region because I need to specify specific cell background colours which change every time the document is created. The

    • Support for Custom Fonts in Zoho Recruit Career Site and Candidate Portal

      Dear Zoho Recruit Team, I hope you're doing well. We would like to request the ability to use custom fonts in the Zoho Recruit Career Site and Candidate Portal. Currently only the default fonts (Roboto, Lato, and Montserrat) are available. While these

    • CC and/or BCC users in email templates

      I would like the ability to automatically assign a CC and BCC "User (company employee)" into email templates. Specifically, I would like to be able to add the "User who owns the client" as a CC automatically on any interview scheduled or candidate submitted

    • Trying to export a report to Excel via a deluge script

      I have this code from other posts but it gives me an error of improper statement, due to missing ; at end of line or incomplete expression. Tried lots of variations to no avail. openUrl(https://creatorapp.zoho.com/<username>/<app name>/XLSX/#Report:<reportname>,"same

    • Need help to create a attach file api

      https://www.zoho.com/crm/developer/docs/api/v8/upload-attachment.html Please help me to create it... It's not working for while. Do you have some example?

    • Export view via deluge.

      Hi, Is it possible to export a view (as a spreadsheet) via deluge? I would like to be able to export a view as a spreadsheet when a user clicks a button. Thanks     

    • how to add subform over sigma in the CRM

      my new module don't have any subform available any way to add this from sigma or from the crm

    • Outdated state in mexico

      Hello Zoho team, the drop down to add the state for customers, when they introduce their state in mexico has a city named “Distrito Federal” that name changed many years ago to “ciudad de mexico”. could you please update this so my clients can find the

    • Support new line in CRM Multiline text field display in Zoho Deluge

      Hi brainstrust, We have a Zoho CRM field which is a Muti Line (Small) field. It has data in it that has a carriage return after each line: When I pull that data in via Deluge, it displays as: I'm hoping a way I can change it from: Freehand : ENABLED Chenille

    • Possible to generate/download Quote PDF using REST API?

      See title. Is there any way after a quote has been created to export to a PDF using a specified template and then download it? Seems like something that should be doable. Is this not supported in the API v2.0?

    • Creating an invoice to be paid in two installments?

      Hi there, I own a small Photographic Services business and have not been able to find a way to fit my billing system into Zoho, or any other Accounting software. The way my payments work is: 1. Customer pays 50% of total price of service to secure their

    • Bug in allowing the user to buy out of stock items

      Hi i want to allow the user to buy out of stock items, according to the commerce documentation if i disable Restrict "Out of stock" purchases it will, but it doesnt work, so i want to know if it had any relation with zoho inventory, and if theres any

    • Zoho CRM Calendar | Custom Buttons

      I'm working with my sales team to make our scheduling process easier for our team. We primary rely on Zoho CRM calendar to organize our events for our sales team. I was wondering if there is a way to add custom button in the Calendar view on events/meeting

    • Replace Lookup fields ID value with their actual name and adding inormation from subforms

      Hi everyone,  I wanted to see if someone smarter than me has managed to find any solutions to two problems we have. I will explain both below.  To start we are syncing data from Zoho CRM to Zoho Analytics and I will use the Sales Order module when giving

    • Can a Zoho Sites page be embedded into another website (outside Zoho)

      Hi All, We have a request from a client - they'd like to take one of our information pages created in Zoho Sites and embed it into their own website? I was told through an email with Zoho that this was possible >>Thank you for your patience regarding

    • Bug in allowing the user to buy out of stock items

      Hi i want to allow the user to buy out of stock items, according to the commerce documentation if i disable Restrict "Out of stock" purchases it will, but it doesnt work, so i want to know if it had any relation with zoho inventory, and if theres any

    • Transition Criteria Appearing on Blueprint Transitions

      On Monday, Sept. 8th, the Transition criteria started appearing on our Blueprints when users hover over a Transition button. See image. We contacted Zoho support because it's confusing our users (there's really no reason for them to see it), but we haven't

    • Zoho CRM Sales Targets for Individual Salespeople

      Our organistion has salespeople that are allocated to different regions and have different annual sales targets as a result. I am building an CRM analytics dashboard for the sales team, which will display a target meter for the logged in salesperson.

    • Next Page