Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Unable to remove the “Automatically Assigned” territory from existing records

  Hello Zoho Community Team, We are currently using Territory Management in Zoho CRM and have encountered an issue with automatically assigned territories on Account records. Once any account is created the territory is assigned automatically, the Automatically

• Improved Functionality PO Bill SO Invoice

  Hello, I need to enter over 100 items, it's frustrating to scroll a few item rows and wait for more to load, then scroll again. It would be nice to have buttons that scroll to the top or bottom with one click. Furthermore, these items I'm adding are VAT

• Wishes for 2026

  Hello, and a happy new year 2026! Let's hope it's better for everyone. I'd like to share some thoughts on Zoho One and what could be useful in the short, medium, and long term. Some things are already there, but not applied to Zoho One. Others seem like

• How to Integrate Zoho Books with Xero (No Native Connection Available)

  Hi everyone, I’m currently facing an issue with integrating Zoho Books invoices with Xero, as I’ve noticed Zoho does not provide a native integration with Xero at this time. I would like to ask: What are the common or recommended solutions for syncing

• ZohoSalesIQ.Chat cannot send messages

  Chat cannot send messages. Our app implements the customer service chat window functionality by integrating the Mobilisten SDK. Recently, we encountered an issue: after successful SDK initialization and visitor registration, when the `startWithQuestion`

• Markup instead of discount??

  Is it possible to have the discount field on an invoice or estimate be a markup instead? It's the very same functionality that I am looking for - to be able to markup an individual item by either a $ or a % and have that cost added (instead of subtracted) to the item cost.  Currently, I use the tax feature to accomplish this, but it's very awkward to have "Markup 15%" listed after the subtotal. Plus, if I want to markup items at different rates, I end up with several "markup" lines in the totals

• Plan switch

  I thought I pressed the Free button, but after they set up the account, I see that I am on a Premium trial. Is this usual? How do I get onto the Free version?

• 'Tax registration date'?

  The second question in setting up is 'Tax registration date'. My first question: which tax? My second question: why do you need the registration date?

• What if we don't have an ABN?

  Australian business. Setting up. We have no ABN. It's not compulsory. Field not allowed to be blank. What to do?

• Keeping track of project expenses

  I have talked to a few support techs and it is very hard for me to believe that Zoho's project accounting software can't keep accounts for my projects. I must not understand what they're saying. We get a contract to build something. So the project revenue

• Linking Multi-UOM Barcodes to Products in Zoho Books

  Greetings, I'm using Zoho Books for retail shop and I'm running into a bit of a challenge with products that have multiple Units of Measurement (UOMs) and corresponding barcodes. For example, I sell cigarettes both as individual packets and in cartons

• How to change "from" email address when invoicing.

  HELP! I'm new to Zoho Books and when I first set it up used a different email address than I currently want to use. I've tried to go to settings and change the default email address but when I generate an invoice it still shows the old email address (although I can't find it anywhere else). I really need to get this changed before I can use the program. Any help would be appreciated.

• Critical Issue: Tickets Opened for Zoho Support via the Zoho Help Portal Were Not Processed

  Hi everyone, We want to bring to your attention a serious issue we’ve experienced with the Zoho support Help Portal. For more than a week, tickets submitted directly via the Help Portal were not being handled at all. At the same time no alert was posted

• Hide/Show Subform Fields On User Input

  Hello, Are there any future updates in Hide/Show Subform Fields "On User Input"?

• Cloning Item With Images Or The Option With Images

  Hello, when I clone an item, I expect the images to carry over to the cloned item, however this is not the case in Inventory. Please make it possible for the images to get cloned or at least can we get a pop up asking if we want to clone the images as

• Custom validation in CRM schema

  Validation rules in CRM layouts work nicely, good docs by @Kiran Karthik P https://help.zoho.com/portal/en/kb/crm/customize-crm-account/validation-rules/articles/create-validation-rules I'd prefer validating data input 'closer to the schema'

• Adding Default Module Image into mail merge field

  As with most people finding their way to these forums i have a specific requirement that doesn't seem to be supported by Zoho I have created 2 custom modules to suit my purpose 1 is an inventory type module that lists aluminium extrusions, and all relevant

• Is it possible to roll up all Contact emails to the Account view?

  Is there a way to track all emails associated with an Account in one single view? Currently, email history is visible when opening an individual Contact record. However, since multiple Contacts are often associated with a single Account, it would be beneficial

• Managing credit cards

  Hi, I'm hoping someone on this forum can help me out managing credit card accounts.  Transaction are entered as expenses using the credit card account and, as expected, the card balance shows as a negative since this is clearly money I owe them.  Once I make a payment, of the full amount owed, I woudl expect the balance to become zero. It doesn't, the negative balance just keeps on increasing even though the payment is reflected correctly in my main operating account.  No-one at Zoho seems to understand

• Free Training - Explore What’s New in Zoho One 2025

  Greetings! We hope you have all had a chance by now to get hands-on with the new features and updates released as part of ZO25. Yes, we understand that you may have questions and feedback. To ensure you gain a comprehensive understanding of these updates,

• Quick way to add a field in Chat Window

  I want to add Company Field in chat window to lessen the irrelevant users in sending chat and set them in mind that we are dealing with companies. I request that it will be as easy as possible like just ticking it then typing the label then connecting

• How to create a two way Sync with CRM Contacts Module?

  Newbie creator here (but not to Zoho CRM). I want to create an app that operates on a sub-set of CRM Contacts - only those with a specific tag. I want the app records to mirror the tagged contacts in CRM. I would like it to update when the Creator app

• How to import data from PDF into Zoho Sheet

  I am looking to import Consolidated Account Statement (https://www.camsonline.com/Investors/Statements/Consolidated-Account-Statement) into zoho sheet. Any help is appreciated. The pdf is received as attachment in the email, this document is password

• Trigger Workflow, Approval Process, Blueprint by Deluge in Zoho CRM

  #Tips & Trick If you intend to activate your Workflow, Approval Process, or Blueprint through either the deluge script or Zoho CRM API during the creation, updating, or deletion of a record Try this Script mp = Map(); trigger = List(); // Workflow trigger.add("workflow");

• Zoho Projects Android app: Integration with Microsoft Intune

  Hello everyone! We’re excited to announce that Zoho Projects now integrates with Microsoft Intune, enabling enhanced security and enterprise app management. We have now added support for Microsoft Intune Mobile Application Management (MAM) policies through

• Cant't update custom field when custom field is external lookup in Zoho Books

  Hello I use that : po = zoho.books.updateRecord("purchaseorders",XXXX,purchaseorder_id,updateCustomFieldseMap,"el_books_connection"); c_f_Map2 = Map(); c_f_Map2.put("label","EL ORDER ID"); c_f_Map2.put("value",el_order_id); c_f_List.add(c_f_Map2); updateCustomFieldseMap

• Set connection link name from variable in invokeurl

  Hi, guys. How to set in parameter "connection" a variable, instead of a string. connectionLinkName = manager.get('connectionLinkName').toString(); response = invokeurl [ url :"https://www.googleapis.com/calendar/v3/freeBusy" type :POST parameters:requestParams.toString()

• Possible to connect Zoho CRM's Sandbox with Zoho Creator's Sandbox?

  We are making some big changes on our CRM so we are testing it out in CRM's Sandbox. We also have a Zoho Creator app that we need to test. Is it possible to connect Zoho CRM's Sandbox to Zoho Creator's Sandbox so that I can perform those tests?

• I Need Help Verifying Ownership of My Zoho Help Desk on Google Search Console

  I added my Zoho desk portal to Google Search Console, but since i do not have access to the html code of my theme, i could not verify ownership of my portal on Google search console. I want you to help me place the html code given to me from Google search

• Timeline Tracker

  Hi Team, I am currently using Zoho Creator – Blueprint Workflows, and I would like to know if there is a way to track a timeline of the approval process within a Blueprint. Specifically, I am looking for details such as: Who submitted the record Who clicked

• Primary / Other Billing Contacts

  If you add an additional contact to a Zoho Billing Customer record, and then mark this new contact as the primary contact, will both the new primary and old primary still receive notifications? Can you stop notifications from going to the additional contacts

• Is it possible to enforce a single default task for all users in a Zoho Projects ?

  In Zoho Projects, the Tasks module provides multiple views, including List, Gantt, and Kanban. Additionally, users can create and switch to their own custom views. During project review meetings, this flexibility creates confusion because different users

• [Free Webinar] Zoho Creator webinars - Learning Table and Creator Tech Connect Series in 2026

  Hello everyone, Wishing you all a wonderful new year! May 2026 and the years ahead bring more opportunities, growth, and learning your way 🙂 We’re excited to kick off the 2026 edition of the Learning Table Series and Creator Tech Connect Series ! Learning

• Reply and react to comments

  Hi everyone! We're excited to bring to you a couple of new features that'll make your sprint process simpler. A cloud application brings with it an array of social media features that can be efficiently used in your organizational setup. As an agile scrum

• Module Customisation - Lookup function not available

  Good evening, Within my business, I can have multiple customers, who have multiple mobile assets. When I set these assets up, I enter information such as vehicle registration, Vehicle identification number (VIN), Unit number, YOM, in addition to others.

• zoho click, and nord VPN

  Unfortunately, we've been having problems with Zoho Click, where essentially the line cuts off after about a minute's worth of conversation every time we are on VPN. Is there a way we can change this within the settings so it does not cut the line off

• Zoho Calender

  a) does the clanender in zoho project allow you to see the name of the event in the celnder view, it currently says either "Task (1) or "Milestoen (1)" b) Alternatively does the calender in Zoho project integrate with zoho calender?

• Matching ZOHO Payments in Banking

  Our company has recently integrated ZOHO Payments into our system. This seemed really convenient at first because our customers could pay their account balance by clicking on a link imbedded in the emailed invoice. Unfortunately, we can't figure out how

• New Enhancements to Zoho CRM and Zoho Creator Integration

  Hello Everyone, We’ve rolled out enhancements to the Zoho Creator and Zoho CRM integration to align with recent updates made to the Zoho Creator platform. With enhancements to both the UI and functionality, This update also tightens access control by

• Work Type - Limitation

  Hello, I'm setting up work types and have noticed, a limitation on the parts area to 10 lines. Can this be increased to 20 or greater? In addition to this, when I attempt to add the work type to a work order, the correct labour hours doesn't flow through.

• Next Page