Kaizen 123- Data Synchronization from third party application

Kaizen 123- Data Synchronization from third party application

Welcome back to the Kaizen series!

In last week's post in the Kaizen series, we discussed one-way data synchronization from Zoho CRM to a legacy application, utilizing Zoho CRM's Bulk Read API and Notification API. This week, we will discuss data synchronization from a third party app to Zoho CRM.

The APIs available in Zoho CRM to insert, update or upsert records are :

1) Insert, Update, and Upsert Records APIs

These are synchronous APIs that are used to perform createupdate or upsert operations for up to 100 records in a Zoho CRM module. While the Insert Records API and Update Records API is used to create and update a record respectively, the  Upsert Records API is used to upsert (insert or update) a record depending on the availability of the record in your org . In Upsert Records API, the system checks for duplicate records based on the values in the duplicate_check_fields json array in the input json. If the record is already present, it gets updated. If not, the record is inserted. You will get the status of these operations immediately. Read more about the Upsert Records API in this Kaizen post.

2) Bulk Write API

 The Bulk Write API is an asynchronous API that is used to perform create/update/upsert operations for up to 25K records in a Zoho CRM module. A background job will be scheduled in Zoho CRM. You can check the status of the scheduled job using the Bulk Write Job Details API. Alternatively, you can configure a callback URL when you schedule the job. The details of the Bulk Write Job result are posted on the callback URL upon successful completion or failure of the job. This bulk write API involves the following steps:

To learn more about these steps in the Bulk Write API and about the Bulk Write API in general, refer to the Kaizen post on the Bulk Write API.


  Insert, Update, Upsert Records


Bulk Write API


Insert, update, or upsert up to 100 records per API call.

Insert, update, or upsert up to 25000 records per API call.
The response is available instantly.


The response to the Bulk Write API request will not be available immediately. You can check the status of the job by polling for it, or you can wait for the status to be available in the callback URL after the job has completed.

These APIs consume one credit per 10 records. 

Bulk Write API consumes 500 credits per API call. 
Input should be in JSON format

The input file should be in CSV format.




Let us now examine four different scenarios for record insertion in a module, specifically with varying quantities: 100, 2000, 5000, and 25000 records, and discover how these APIs perform in terms of number of API calls and credits consumed.


Number of Records

Insert,Update or Upsert Records API

Bulk Write API
100
1 API Call
10 API Credits
1 API Call
500 API Credits
2000
20 API Calls
200 API Credits
1 API Call
500 API Credits
5000
50 API Calls
500 API Credits
1 API Call
500 API Credits
25000
250 API Calls
2500 API Credits
1 API Call
500 API Credits

Clearly, if you have more than 5000 records, it is advisable to use the bulk write API, given that you are ok with the operation being run asynchronously. If you want instant results, you should use insert/update/upsert APIs. However, more API credits will be consumed in this case.

Data Sync

Consider an e-commerce company where the Order management team uses a legacy system and the Accounts team uses Zoho CRM. To ensure that the Accounts team works with near real-time data in Zoho CRM, data synchronization from the legacy application is essential. Assume that in Zoho CRM, there is a custom module named "Orders" that contains the details from the legacy system. Imagine that you are in charge of the data synchronization between Zoho CRM and the legacy system. How should you call these APIs to achieve this synchronization? The synchronization process involves two steps. 
Step 1: Initial Data Sync : Initial data sync from the legacy system to Zoho CRM.
Step 2: Subsequent Data Sync : Push any new orders or updated orders to Zoho CRM.

The solution we propose here is one of the many ways in which the data sync can be achieved.

This solution is built on the below three key points: 

1) Unique field in Order Module: Maintain a unique field "parent_id" in the "Orders"  module. The unique identifier of the orders in your legacy system should be mapped to this parent_id field. In create bulk write job, this can be achieved by the section field_mappings. For ensuring correct working of upsert operations, you can make use of "find_by" field inside resource JSON Array in the Create Bulk Write Job API Request.  You should provide the value of "find_by" as the unique field "parent_id" .  To read more about find_by field refer to Create Bulk Write Job page.

2) Sync Flag in the Orders table in legacy system: For the sync, maintain a flag in your legacy system orders table for indicating whether the record is synced with Zoho CRM or not. Once the sync is done, update the flag to  true. Whenever a record is modified or sync fails, the flag value should be updated to false. This ensures that in case a record that is synced gets updated during the initial sync process, this flag column helps us re-sync this record again.

3) Grouping of data to batches:  For initial sync, the data volume in the table will be huge. So they have to be batched in a group of 25k records. This grouping can be done based on sorting the table with a particular column & filtering records based on the sync column with criteria sync=false.

Step 1: Initial Data Sync 

Since you have multiple batches of 25k records, you have to iterate them one by one and send them to the Bulk Write API.

Initial Data Sync
For each batch of data, prepare them as a CSV file and initiate a bulk write Job. Next, check the status of the Bulk Write job, and once complete, look for any records that have been skipped during the process. If there are any, fix them, update the sync flag as false, and include them in the next round of data sync. Continue this loop until all records, including any that were missed, are processed. Take the next batch and repeat till all the batches are over.

Step 2: Subsequent Data Sync

The Order Management team continues to use the legacy application, leading to continuous creation or update of records in the Orders module.
It is not recommended to invoke Upsert Records API whenever an operation happens in Orders in the legacy system. Subsequent data syncs can be done in two ways: either by  (a)near real-time sync or (b) Interval based sync.


(a) Near Real-time Sync

 As we want near real-time data in Zoho CRM, it is ok for the records that are updated in the legacy system from last synchronization to be unavailable in Zoho CRM for 10 minutes.

Near Real-time Sync

Instead of making API calls to Zoho CRM on every data operation, store the data in a temporary data store like a DB table or Redis. Once 100 records are accumulated in that data-store, the Upsert Records API can be promptly invoked with all these record details and followed by emptying of the data store. This approach will lead to lower consumption of API credits.
There are chances that in a period of 10 mins, 100 record operations may not happen. Scheduler jobs or CRON jobs can be created to handle the records in the data store to update records to Zoho CRM. Once these records are handled, the data store should be emptied.

(b) Interval based synchronization

For interval based synchronization, schedule bulk write API calls at suitable intervals to upsert the Orders module in Zoho CRM. Whenever there is a record operation in the legacy table, update the sync flag to "false". At regular intervals, such as every 12 or 24 hours, the bulk write API or Upsert Records API can be employed depending on the volume of the data changes.

We hope you found this article useful. We will be back next week with another interesting topic. If you have any questions, write to us at support@zohocrm.com or let us know in the comment section. 


Related Links




    Access your files securely from anywhere

          Zoho Developer Community




                                    Zoho Desk Resources

                                    • Desk Community Learning Series


                                    • Digest


                                    • Functions


                                    • Meetups


                                    • Kbase


                                    • Resources


                                    • Glossary


                                    • Desk Marketplace


                                    • MVP Corner


                                    • Word of the Day



                                        Zoho Marketing Automation


                                                Manage your brands on social media



                                                      Zoho TeamInbox Resources

                                                        Zoho DataPrep Resources



                                                          Zoho CRM Plus Resources

                                                            Zoho Books Resources


                                                              Zoho Subscriptions Resources

                                                                Zoho Projects Resources


                                                                  Zoho Sprints Resources


                                                                    Qntrl Resources


                                                                      Zoho Creator Resources



                                                                          Zoho Campaigns Resources


                                                                            Zoho CRM Resources

                                                                            • CRM Community Learning Series

                                                                              CRM Community Learning Series


                                                                            • Kaizen

                                                                              Kaizen

                                                                            • Functions

                                                                              Functions

                                                                            • Meetups

                                                                              Meetups

                                                                            • Kbase

                                                                              Kbase

                                                                            • Resources

                                                                              Resources

                                                                            • Digest

                                                                              Digest

                                                                            • CRM Marketplace

                                                                              CRM Marketplace

                                                                            • MVP Corner

                                                                              MVP Corner





                                                                                Design. Discuss. Deliver.

                                                                                Create visually engaging stories with Zoho Show.

                                                                                Get Started Now


                                                                                  Zoho Show Resources


                                                                                    Zoho Writer Writer

                                                                                    Get Started. Write Away!

                                                                                    Writer is a powerful online word processor, designed for collaborative work.

                                                                                      Zoho CRM コンテンツ






                                                                                        Nederlandse Hulpbronnen


                                                                                            ご検討中の方





                                                                                                  • Recent Topics

                                                                                                  • AI read notes or explanation

                                                                                                    It would be such a great feature to have AI voices be able to read our Notes or Explanation If not having the AI speak the notes at run time, how about a feature where inside of Zoho Show you can have it look at all the notes of all the slides and have
                                                                                                  • Top Menu Disappeared from Blog Page

                                                                                                    Hi, Our top menu disappeared at Blog Posts page. However, it's still visible any other page on the website. I attached two screenshots, so it can be understood clearly. How can we bring back top menu? Thanks, K.
                                                                                                  • Managing Prepaid Hours for Consulting

                                                                                                    We are a consulting firm that bills clients a flat upfront annual fee plus an hourly rate and offer a discount for pre-paying a block of hours. Hours that surpass the pre-paid block are billed monthly at the normal rate. If there are any pre-paid hours remaining at the end of the project they are banked for future use. I'm not seeing a method of doing this in Projects/Books/CRM... thoughts?
                                                                                                  • ZOHO Widget SDK not loading in html

                                                                                                    I have this code below, I have imported the widgetsdk however I get the error shown in the image, I have tried many different ways of importing and initiating the function ZOHO but nothing is working. can someone explain what I'm doing wrong, if I am
                                                                                                  • Enhancements to Zoho Corp Help Center "Team Requests" View

                                                                                                    Dear Zoho Team, I hope this message finds you well. The ability to view both my tickets and my team’s tickets in the Zoho Corp Help Center is a fantastic feature, especially as the focal point for Zoho in our organization. However, we’ve encountered a
                                                                                                  • Allow Multiple Scheduled Appointments with Zoho Support

                                                                                                    Dear Zoho Team, I hope you're doing well. First, thank you for introducing the option to schedule support calls via the Zoho CRM booking link. This has been a fantastic enhancement, eliminating the need for back-and-forth coordination when scheduling
                                                                                                  • Projectwise budget ---

                                                                                                    Can we have a Project wise subject in addition to the Monthly, and quarterly ACCOUNT LEVEL budget?
                                                                                                  • WorkDrive API Documentation

                                                                                                    WorkDrive provides users and developers an extensive set of APIs to help integrate functionalities of Zoho WorkDrive with other Zoho applications and third-party tools. We have published the official WorkDrive API Documentation page for all external users.
                                                                                                  • Error 403: Forbidden When Updating Email Signature via API

                                                                                                    Hi Zoho Desk team, First, congratulations again on the excellent Zoho API. But, I’m encountering an issue while attempting to update an email signature via the API. Whenever I make a request to update the signature, the response returns an HTTP 403 Forbidden
                                                                                                  • Who can see draft replies on tickets?

                                                                                                    We have noticed that we are able to see draft replies made by other agents. Which settings can limit this visibility? It makes sense to me that admins and the agent who created the draft would be able to see the draft, but no one else. How can we make
                                                                                                  • Serious question: Are there actually "solo-preneurs"/small business owners who made Zoho-one work well for them?

                                                                                                    L.S. After already many years of continued struggle with Zoho-One, I am seriously wondering if there are actually solo-preneurs (one person small business owners - without a large, dedicated IT dept.) who got it (Zoho-One) to work well for their businesses.
                                                                                                  • Major iOS issues when accessing forms via the browser

                                                                                                    Hi, We have been using forms for some time, while the office staff are accessing the forms via the app on Android mobiles, we have a fleet of sub contractors that we would not like them having access to the main app as some of the forms are confidential
                                                                                                  • All notes disappeared

                                                                                                    I've been using the notebook app for over five years on my phone without being logged into an account. A few days ago I opened the app and all my notes had disappeared. Since then I tried restarting my phone, updating the app and logging into my account,
                                                                                                  • How to Iterate a Function in Zoho Desk Workflow with Delay Between Calls?

                                                                                                    Hi everyone, I’m working on a function in Zoho Desk that searches for a specific ticket record. If the ticket is not found, I need to retry the search multiple times with a delay between each attempt until the ticket is located or a maximum number of
                                                                                                  • How to Iterate a Function in Zoho Desk Workflow with Delay Between Calls?

                                                                                                    Hi everyone, I’m working on a function in Zoho Desk that searches for a specific ticket record. If the ticket is not found, I need to retry the search multiple times with a delay between each attempt until the ticket is located or a maximum number of
                                                                                                  • Work Orders / Bundle Requests

                                                                                                    Zoho Inventory needs a work order / bundle request system. This record would be analogous to a purchase order in the purchasing workflow or a sales order in the sales cycle. It would be non-journaling, but it would reserve the appropriate inventory of
                                                                                                  • Zoho Books API Limit Is RIDICULOUS!!!!!!!!!!!!!

                                                                                                    The 2,500 API call limit in Zoho Books is about as useful as AOL dialup.  Seriously Zoho, not only can I use up 2,500 API calls in no time with my own app but YOUR OWN STUPID IPAD APP blows through them super fast too, so if any one of my clients wants
                                                                                                  • QR codes in templates

                                                                                                    I'm excited about the new QR code generator. I have included a QR code that contains the record ID setting "${ID}" as input data. In the report detail it works perfectly but when printing it in a template the code is not shown.
                                                                                                  • Button Display Conditions

                                                                                                    Hi Guys, Is it at all possible to have extra button conditions? Context: We have data in our deals module which has a custom button which converts the deal into contacts + set up relationships between them. At the end of the conversion we set a field
                                                                                                  • Knowledge base: The nitty-gritty of SEO tags

                                                                                                    A well-optimized knowledge base with great SEO can benefit your company by allowing customers to find help articles and support resources using search engines. This enables customers to quickly and efficiently find the information they need without direct
                                                                                                  • Social Media Simplified with Zoho Social: Make the best out of the publishing calendar

                                                                                                    Are you a marketer who likes visualizing your plan of action before you start social media posting? Are you part of a team that works on social media on a rotational basis, so the most important task is to collaborate to avoid overlap and confusion? Or
                                                                                                  • Custom function daily limit and procedural programming

                                                                                                    Dearest Zoho Today, support confirmed that if I call a custom function from another custom function then I will use up two, with regards to my daily limit. A few times, we have blown our daily limit and that means that ordinary business processes don't run for the rest of the day. I have to mop these up the following day and there is no guarantee that I will get it right. Therefore, I can't afford to waste any. Procedural programming has been around for over 50 years now and it greatly simplifies
                                                                                                  • Unified customer portal login

                                                                                                    As I'm a Zoho One subscriber I can provide my customers with portal access to many of the Zoho apps. However, the customer must have a separate login for each app, which may be difficult for them to manage and frustrating as all they understand is that
                                                                                                  • Welcome Link Expired

                                                                                                    Hi The links sent to the users didn't get clicked on in time and now all the links have expired. Is there a way to send a new link without deleting them and re-adding them>
                                                                                                  • New enhancements: Changing portal users' email addresses and new customization options for templates

                                                                                                    Dear All, Portals have enabled organizations to extend access to various CRM modules to their customers, vendors, partners, and end users, per their business requirements. When a portal is created, an invitation email is sent to portal users with a link
                                                                                                  • Granular Time Frame Settings for Message Deletion and Editing in Zoho Cliq

                                                                                                    Dear Zoho Team, I hope you're doing well. Currently, the settings for message deletion and editing in Zoho Cliq are configured globally under: Admin Panel > Organisation > Configurations > Conversations Delete messages: Time frame to allow message deletion
                                                                                                  • New Built In QR/Barcode Generator Print Settings

                                                                                                    I'm trying out the new QR/Barcode generator field in Creator. I would think most people will want to print these, like I do. I am not seeing any way to control the height or width of the barcode for printing (inside the print/pdf template builder). The
                                                                                                  • Zoho One. Client Script

                                                                                                    Hi, I would like to know if the Client Script feature is available in Zoho One. If it's, how can I enable it?
                                                                                                  • Calendar View for Zoho Tickets

                                                                                                    Is there a way to view your tickets with due dates on a calendar view? I can not find a way to merge my Zoho Calendar and Tickets. This would be extremely helpful to my team.
                                                                                                  • Delete / Modify Default Career Site - Zoho Recruit

                                                                                                    Hello, It would be very useful if we could delete a default career site or change which of our career site is the default. Our Career site was created when there were issues with Zoho Recruit creating English CTA buttons on French Career sites. The only
                                                                                                  • Workflows for Timesheet

                                                                                                    Good day, Any way to have timesheet as triggers? I looked into Zoho Flow and into Zoho Project automation but no where can I have timesheet as a trigger. Basically, I would like to trigger something upon timesheet approval. Right now, the only way to
                                                                                                  • Is it possible to hide Developer Space for all user in Zoho Projects

                                                                                                    Hello! I am Zoho admin in a company and we want to use Zoho Project to manage projects, but after a few days of testing we are not able to "hide" the Developer Space from all kind of users except the admin. To sum up, I want to hide this for all users.
                                                                                                  • Introducing automation and utility conversations in WhatsApp marketing

                                                                                                    We’re excited to announce the addition of two new features to our WhatsApp integration: Automation and Utility conversations. These enhancements will allow you to streamline your marketing efforts and engage with your customers more effectively by automating
                                                                                                  • Extracting data from cells in zoho sheets for zoho books

                                                                                                    I am currently uploading my bank statment in excel format to zoho workdrive. I would like flow to extract certain data and send it to zoho books. Would scripting in zoho flow be able to help me with this? By this I mean should I attempt this in zoho flow
                                                                                                  • Within the Basic KPI component in Analytics, it is impossible to set "next" day range as a filter

                                                                                                    Hi there, I am currently setting up a deal dashboard for the Sales team. While it is possible to filter deal records to show records that were created LAST X days only, it looks like a NEXT X days Closing date filter is not available. Would it be possible
                                                                                                  • Pulling Specific Products from Sales Orders in Books to a CRM Record

                                                                                                    We currently process orders directly through our website (woocommerce) as well as through manual sales orders in zoho books. When an order comes through the website, all of the individual products from that order show up in the CRM record of that customer.
                                                                                                  • Você já viu os cursos do Zoho Mind?

                                                                                                    Pessoal, Tem uma plataforma da Zoho chamada Zoho Mind, muito interessante os cursos e vídeos tutoriais que lá possui. Para a turma do Zoho Creator, tem uma dica de Buscar dados em Formulário, segue o link e clique em Zoho Creator. https://www.zohomind.com.br/#/videostutoriais
                                                                                                  • Como gerar gatilhos para pagamento de impostos no Zoho Books?

                                                                                                    Olá Pessoal, boa tarde! Gostaria de saber como vocês estão escriturando os impostos a pagar no Zoho Books. Vi que temos a opção de Bills, porém se eu escriturar nesta aba do Zoho Books para gerar lembretes de tempo de vencimento por exemplo vai refletir
                                                                                                  • Subform Time field to string.

                                                                                                    Good afternoon All. I have a Subform 'Delivery_Receiving_Hours' that captures Day (Dropdown), Time_Open (Time), and Time_Close (Time). I need to capture this data and send it to a multiline field in the CRM. The code, posted below, below will capture
                                                                                                  • workflow for bounced email gets triggered, but email is status = opened

                                                                                                    Hello, I have a workflow that sends me an email if outgoing email are bounced. Now I got some kind of this emails, but the corrosponding contacts have status = open at the email. Why this bounce-workflow is triggered? Reports > Email Reports > Bounce
                                                                                                  • Next Page