Kaizen 123- Data Synchronization from third party application

Kaizen 123- Data Synchronization from third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

          Zoho Developer Community




                                    Zoho Desk Resources

                                    • Desk Community Learning Series


                                    • Digest


                                    • Functions


                                    • Meetups


                                    • Kbase


                                    • Resources


                                    • Glossary


                                    • Desk Marketplace


                                    • MVP Corner


                                    • Word of the Day



                                        Zoho Marketing Automation


                                                Manage your brands on social media



                                                      Zoho TeamInbox Resources

                                                        Zoho DataPrep Resources



                                                          Zoho CRM Plus Resources

                                                            Zoho Books Resources


                                                              Zoho Subscriptions Resources

                                                                Zoho Projects Resources


                                                                  Zoho Sprints Resources


                                                                    Qntrl Resources


                                                                      Zoho Creator Resources



                                                                          Zoho Campaigns Resources


                                                                            Zoho CRM Resources

                                                                            • CRM Community Learning Series

                                                                              CRM Community Learning Series


                                                                            • Kaizen

                                                                              Kaizen

                                                                            • Functions

                                                                              Functions

                                                                            • Meetups

                                                                              Meetups

                                                                            • Kbase

                                                                              Kbase

                                                                            • Resources

                                                                              Resources

                                                                            • Digest

                                                                              Digest

                                                                            • CRM Marketplace

                                                                              CRM Marketplace

                                                                            • MVP Corner

                                                                              MVP Corner





                                                                                Design. Discuss. Deliver.

                                                                                Create visually engaging stories with Zoho Show.

                                                                                Get Started Now


                                                                                  Zoho Show Resources


                                                                                    Zoho Writer Writer

                                                                                    Get Started. Write Away!

                                                                                    Writer is a powerful online word processor, designed for collaborative work.

                                                                                      Zoho CRM コンテンツ






                                                                                        Nederlandse Hulpbronnen


                                                                                            ご検討中の方





                                                                                                  • Recent Topics

                                                                                                  • Show item Cost value on Item screen

                                                                                                    The Item screen shows Accounting Stock and Physical Stock. It would be very helpful if value information could be displayed here as well, for instance Cost Price. Currently, to find the Cost Price (as used for inventory valuations) from inside the item
                                                                                                  • Zoho Analytics to Zoho Sheets - automatic update?

                                                                                                    Hi all, If I create a zoho sheet from an Analytics Report or Analytics Data, is there a way for the zoho sheet to automatically update as the Report / Data in analytics updates?
                                                                                                  • Integration with Moodle

                                                                                                    Greetings, I hope find all doing well and safe. I've recently returned to using Zoho Flow after a break and was hoping to connect my WooCommerce store with Moodle, the world's most widely used learning management system. My goal was to automatically enroll
                                                                                                  • How to Record Loan with interest

                                                                                                    I have received loans from friend he give me like 2 loans so far one is one year repayment and one short, how to properly record his payment, and repayment and give him statement  for each loan he give me 
                                                                                                  • Task status - completed - other options

                                                                                                    I have a dumb question I know i can make custom statuses for the tasks - but is there anyway to make additional "completed" statuses like for instance if i have a task "call back customer" and i leave a vm for them to call back marking it "completed -
                                                                                                  • Task module and related-to field

                                                                                                    In modules other than the Task Module I can add several lookup fields to provide a variety of relationships. In the Task module lookup fields are not available. There is only one "related to" field which I want to use for Company. But I want to relate
                                                                                                  • Add Lookup Field in Tasks Module

                                                                                                    Hello, I have a need to add a Lookup field in addition to the ones that are already there in the Tasks module. I've seen this thread and so understand that the reason lookup fields may not be part of it is that there are already links to the tables (https://help.zoho.com/portal/en/community/topic/custom-fields-on-task-module).
                                                                                                  • migrating from Zoho Invoices (CRM) to Zoho Books

                                                                                                    Good day, I was wondering if there was a easy way to migrate all the quotes and invoices from Zoho Invoices CRM to Zoho Books. We plan to move to using Zoho Books in a few weeks and would like to have all the quotes and invoices from the past 3 years
                                                                                                  • Zoho MA and Custom Module

                                                                                                    I am trying to create a sync between Markting Automation and Zoho CRM. I am mapping a custom module from the CRM. The custom module has email field mobile phone field However I cannot finish the integration since the system keeps asking me for email or
                                                                                                  • When is partial reimbursement going to be launched?

                                                                                                    Hi there. I saw somewhere that the partial reimbursement feature is in the work. What is the update and ETA of that? Our clients and prospects have been asking us and we agree that that is an important feature to have.
                                                                                                  • Year-End Wrap: Declutter Your Inbox Using Email Filters

                                                                                                    Ping!—an email drops in. And another. And another! It's finally that time of the year when your inbox will be bursting with messages from team members, clients, and marketing agents, leaving you feeling overwhelmed and distracted. Sounds familiar? Now
                                                                                                  • Unified Notes View For Seamless Collaboration

                                                                                                    To facilitate better coordination among different departments and team members, the notes added to a record can now be accessed in all its associated records. With this, team members, from customer service representatives to field technicians, can easily
                                                                                                  • Q4 Europe In-person Zoho User Group Meetup: Streamlining Your Business Processes & Introduction to Zoho CRM for Everyone

                                                                                                    Hello Zoho Community, We are excited to announce our upcoming Zoho User Group meetup in Europe! This session is designed to help you streamline your business processes using Zoho CRM, with a special focus on enhancing customer interactions and leveraging
                                                                                                  • Formula fields - Request for dynamic recalculation / proper implementation

                                                                                                    Hi Guys, I have a big problem with Zoho formula fields. They don't recalculate each time the record is viewed - only when a record is edited. Formula fields should be updated dynamically each time a record is retrieved. As an example: I have a formula
                                                                                                  • Items attribute questions

                                                                                                    Many of my items have attributes, such as size and color. How can I add new fields to the "New Items" screen to capture that in my Purchase Orders, Items, and Sales Order pages? I only see these attribute fields when adding an Item Group. Also, on the
                                                                                                  • Organize and Track Phases with Phase Custom Views

                                                                                                    Phase Custom Views let you filter and display phases based on specific criteria. This helps you focus on what’s most relevant for you and your team. Filter phases using criteria such as budget, status, and more. Share views with specific users or teams
                                                                                                  • Record Asset Received as Payment

                                                                                                    How exactly would you account for this in books? For example, I receive a mini computer for a review and I get to keep it after the video is published. Would debit my normal asset account (e.g. Computers) and credit an income account (e.g. Other Income).
                                                                                                  • Invoice Line Item Report

                                                                                                    Is it possible to run an 'Invoice Line Item Report'? The 'Invoice Details Report' shows one row per invoice. I would like one row per Invoice Line Item.
                                                                                                  • Transform Your Customer Support with AI-Powered Chatbots in Zoho SalesIQ

                                                                                                    Ever wondered how some companies seem to have superhuman customer support? Let's uncover their secret! In the digital age, customer expectations are skyrocketing. Did you know that according to McKinsey, 75% of consumers expect a response within five
                                                                                                  • Progressive Invoicing

                                                                                                    Progressing invoicing is needed for many industries. It would be great to see it implemented into Zoho Books as well. Set up and send progress invoices in QuickBooks Desktop (intuit.com)
                                                                                                  • Client Script - mapping data from different module

                                                                                                    Dear ZOHO Team Firstly I need to describe the need - I need to have data from Contacts module based on lookup field - the 5 map limit is not enough for me because I have almost 20 fields to copy So I have decided to make a Customer Script - and from unknown
                                                                                                  • DORA compliance

                                                                                                    For DORA (Digital Operational Resilience Act) compliance, I’ll want to check if Zoho provides specific features or policies aligned with DORA requirements, particularly for managing ICT risk, incident reporting, and ensuring operational resilience in
                                                                                                  • Files Uploaded to Zoho WorkDrive Not Being Indexed by Search Engines

                                                                                                    Hello, I have noticed that the files I upload to Zoho WorkDrive are not being indexed by search engines, including Google. I’d like to understand why this might be happening and what steps I can take to resolve it. Here are the details of my issue: File
                                                                                                  • Zoho Creator Upcoming Updates - December 2024

                                                                                                    Hi all, We're excited to be back with the latest updates and developments on the Creator platform. Here's what we're going over this month: Deluge AI assistance Rapid error messages in Deluge editor QR code & barcode generator Expandable RTF and multi
                                                                                                  • Customer can't comment on SO or Invoice

                                                                                                    Hi I just saw that my customers are not able to submit a comment either on invoices or sales order. What happens if my customer hits submit is just nothing. only a red line appears on top of the page which probalby indicates an error. I'm not able to
                                                                                                  • Zoho Creator customer portal limitation | Zoho One

                                                                                                    I'm asking you all for any feedback as to the logic or reasoning behind drastically limiting portal users when Zoho already meters based on number of records. I'm a single-seat, Zoho One Enterprise license holder. If my portal users are going to add records, wouldn't that increase revenue for Zoho as that is how Creator is monetized? Why limit my customer portal to only THREE external users when more users would equate to more records being entered into the database?!? (See help ticket reply below.)
                                                                                                  • See Calendar When Creating Meetings On Record Page

                                                                                                    It would be a great user experience to see you calendar while you are creating a meeting on a record page. Here is how I imagine it could look:
                                                                                                  • Saved filters, layout rule support, related list quick navigation, and more

                                                                                                    Hello Everyone, We're excited to share some new features and enhancements in the Zoho CRM iOS and Android apps that will improve your mobile experience. These updates will make your CRM journey more efficient and user-friendly. Here's a look at what's
                                                                                                  • Power of Automation: Automatically send an email to all portal users with today's list of Open tasks.

                                                                                                    Hello Everyone, A Custom Function is a user-written set of code to achieve a specific requirement. Set the required conditions needed as to when to trigger using the Workflow rules (be it Tasks / Project) and associate the custom function to it. Requirement:-
                                                                                                  • Introduction of Robotics Process Automation in Zoho products

                                                                                                    It will be great if Zoho can start advancing from automation to robotics process automation. For a start, it can be started with smart document understanding. Provide OCR engines Google cloud, Microsoft Azure Computer vision OCR, Microsoft OCR, Omnipage
                                                                                                  • Lock a custom field on a deal record but keep all other fields editable?

                                                                                                    I have a custom field, which auto-populates a job number upon converting a lead to a deal but the automation breaks if someone accidentally edits that field. I want to lock that field but keep all other fields open. Is this possible? I've tried through
                                                                                                  • Add Feature To Hide Plugin Sections On Record View

                                                                                                    Hi team, I'm trying to help a client tidy up their CRM. When it comes to record view some sections and fields are visible no matter what Layout Rules are applies and they are not removeable from the layout editor. I would like to see an option to hide
                                                                                                  • Creator Simplified #10: Predefine Form Field Values and Make Them Read-Only for Users

                                                                                                    Hey Creators, Ready for this week's tip in the Creator Simplified series? Today, we will explore how to have read only fields in a form. Use Case: Assume a scenario where the default value for a Department field needs to be English Literature, but you
                                                                                                  • Problem configuring/customizing sales pipeline steps

                                                                                                    Hello, I have created several sales pipelines with different stages in them. Unfortunately I forgot to properly configure these steps (conversion probability, forecast category). How can I modify and customize all these steps? Thnak you by advance M
                                                                                                  • fetch records from analytics table from creator

                                                                                                    I have a creator workflow that I am working in that will compare data from within the app to a table in zoho analytics. Is there a way to fetch a record from Analytics? I have attempted a custom connector with analytics and tried to use it with invoke
                                                                                                  • Directly Edit, Filter, and Sort Subforms on the Details Page

                                                                                                    Hello everyone, As you know, subforms allow you to associate multiple line items with a single record, greatly enhancing your data organization. For example, a sales order subform neatly lists all products, their quantities, amounts, and other relevant
                                                                                                  • Ability to Change Custom View After Cadence Creation

                                                                                                    Dear Zoho Team, I hope you are well. We would like to request an enhancement to the Cadence feature in Zoho CRM. Currently, during the creation of a Cadence, we can select a Custom View under the "Who is this for?" section. However, once the Cadence is
                                                                                                  • Zoho Creator Integration with QuickBooks: A Step-by-Step Guide

                                                                                                    Introduction: Integrating Zoho Creator with QuickBooks allows you to sync your business data between the two platforms, providing a seamless experience for managing accounting, invoicing, and financial data. This integration helps automate workflows and
                                                                                                  • Note not being pulled for other modules in email template

                                                                                                    Hi there, Currently i am creating an email template that is able to pull the data from notes field in estimate module and email it to procurement team where they will be able to receive the email with the contents of the note, i am unable to replicate
                                                                                                  • No Sales Returns on SO's with Dropped Shipped items + Inventory Items

                                                                                                    We have encountered an issue in Zoho related to sales orders that include both dropshipped items and inventory items. Specifically, it is currently not possible to create sales returns for the company’s own inventory items from these sales orders. This
                                                                                                  • Next Page