Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Live webinar: Craft and deliver impactful slides with Show’s desktop apps

  If you love the comfort of working on your desktop and want a setup that keeps you focused and uninterrupted, this session is made for you. We’re excited to invite you to our upcoming live webinar: "Craft and deliver impactful slides with Show's desktop

• Enhancements to Bigin's forms

  Greetings, I hope all of you are doing well. We're happy to announce a few recent enhancements we've made to Bigin's forms. We'll go over each one in detail. Merge field support in auto-filled forms The auto-fill option in Bigin's forms lets you predefine

• Table dimensions

  I try changing the dimensions of the table on my computer but it doesn't change. Do I have to be a premium member or does it only work on the app?

• Direct link to Record Summary

  Hi everyone, In one of my reports, I have built a Record Summary template to display the details of one record. I would like to be able to link directly to this Record Summary once I submit a new record, without having to go to the list of records first and click on View. Is there a possibility to do so ?  Should I use the URL by passing some parameters ? Thank you very much for your help ! Guillaume

• Amendment effective date

  Hi everyone, I noticed that the amendment effective date mentionned in my amendment is not right. Indeed, when a contract is amended several times, it states the previous amendment and their effective date. However, the effective date stated is always

• Show both Vendor and Customers in contact statement

  Dear Sir, some companies like us working with companies as Vendor and Customers too !!! it mean we send invoice and also receive bill from them , so we need our all amount in one place , but in contact statement , is separate it as Vendor and Customer, 

• STOCK history in zohosheets

  is it possible to get historical stock value using stock function in zoho sheets? i could not see from and to period in the helper document.

• Auto sync Photo storage

  Hello I am new to Zoho Workdrive and was wondering if the is a way of automatically syncing photos on my Android phone to my workdrive as want to move away from Google? Thanks

• Agent password reset

  Hi Zoho support, I would like to ask if there is a way the admin can reset a password of an agent? Regards

• Can receive but not send messages in Zoho Mail

  Hello! I was able to configure my email client successfully in that I can receive messages just fine. However, when I send messages out, they seem to go out fine (I don't receive any errors or anything), but the recipient does NOT receive those messages.

• Mail is sent twice!

  Been using Zoho for a while now. Installed Zoho for someone else and some weird things are happening. Mails are being sent twice. He is using Thunderbird as an email client. I already read about email being duplicated in the sent folder. But in my case

• Can't login IMAP suddenly

  Since this evening I'm getting the error: You are yet to enable IMAP for your account. Please contact your administrator... IMAP always been enabled in my account and was workign fine for the past 7 years. Already tried turning IMAP off and on again.

• Sending of username did not succeed: Mail server pop.zoho.com responded: User already specified

  I am having issues receiving emails from Zoho in Thunderbird. I am getting the above error. The first error tells me Authentication failed, and prompts me to enter in my password. Then I get the above error. I can receive emails when I log in online to

• Bug tracking

  Hi, does anyone know how to track errors during picking or packing? This way I can keep track and see how to improve and prevent errors in this area.

• Migration of corporate mail environment from Yandex 360 to Zoho mail

  I have to migrate a corporate mail environment with an existing domain from Yandex 360 to Zoho mail. It is vital to migrate all users with all the data. I have read the article on this topic using MacMister Email Backup Software just now and have some

• I'm unable to send mail pthrough Zoho SMTP programmatically

  This has been working for years, but today it's been offline all day long. I see nothing anywhere on your site about this. I'm not the only one experiencing this. Downdetector has a spike of reports today

• Can no longer send email via Django site

  This was working fine as of 11/7/25. Now I am unable to send user verification emails from a Django site on a AWS lightsail sever. When a user attempts to register the following error occurs. I have also attempted to send a test email via the shell and

• unable to send email but able to receive email

  my email address is info@securityforceservices.ca

• Login to server failing

  When trying to retrieve my mail, I am getting this error message -- Login to server pop.zoho.com with username (my email address) failed. It gives me the option to retry, enter password, or cancel. Then I get this message -- Sending of username did not

• Configuration failed: 200 response not received for POST request.

  Hello, I am trying to set up a webhook to connect with an Salesforce but I receive the following error from Zoho: Configuration failed: 200 response not received for POST request I have tried testing it on webhook.site as well and receive the same error

• Zoho Migration Assistant not working

  Hello, I am trying to use you Migration assistant to migrate emails from Rediff to Zoho. I am stuck in the first step. After downloading the migration tool, I copied the link to verify user credentials, however, after pasting the link in the browser,

• Contacts Missing — PeopleSync/Zoho Mail

  English: In our company we use ManageEngine Mobile Device Manager (MDM), Free edition, to manage corporate mobile devices. Our usage policy does not allow personal Google accounts on these devices; therefore, Google account sync is blocked through MDM.

• Best way to integrate Zoho with mobile app for managing customer requests with real-time notifications?

  Hello, I'm building a solution for a travel company where customers submit requests through a website, and the sales team manages these requests through a mobile app. The Requirement: Customers fill a form on the website (name, email, number of children,

• Kaizen #57 - Mass Update API in Zoho CRM

  Hello everyone! Welcome back to yet another post in the Kaizen series. This week, we will discuss the Mass Update API in Zoho CRM. In this post, we will cover the following: 1. Introduction 2. Mass Update Records API  3. Schedule Update and Get Status

• Getting Attachments in Zoho Desk via API

  Is there a way to get attachments into Zoho Desk via an API?      We have a process by which a zoho survey gets sent to the user as a link in a notification.    The survey has several upload fields where they can upload pdf documents.    I've created

• Pincode based Product Restriction

  we have different types of products. 1) Very bulky items like plywood. 2) Too delicate items like glass These type of products we want to sell to local customers. Other products we want to supply all over India. There should be an option to restrict products

• Related Lists filter

  I have Contacts showing in our Accounts module. I customized the Contacts module with an Employment Status field, with the following picklist options: "Primary Contact", "Secondary Contact", "Active Staff(not a main contact)", and "No longer employed".

• Standalone custom function not generating logs

  Why dont't standalone custom functions generate logs when the're called from another function? I have some functions (workflow, buttons and blueprint) that have common parts, so I put that part in a standalone function which is called from the others.

• Add "Reset MFA" Option for Zoho Creator Client Portal Users

  Hello Zoho Creator Team, We hope you are doing well. We would like to request an important enhancement related to Multi-Factor Authentication (MFA) for client portal users in Zoho Creator. Currently, Creator allows us to enforce MFA for portal users,

• Urgent Security Feature Request – Add MFA to Zoho Projects Client Portal Hello Zoho Projects Team,

  Hello Zoho Projects Team, We hope you are doing well. We would like to submit an urgent security enhancement request regarding the Zoho Projects Client Portal. At this time, as far as we are aware, there is no Multi-Factor Authentication (MFA) available

• Creator HTML page refresh

  Hi, I have added around 5 different html snippets in single creator page. I understand, I can refresh the entire page from page script using Navigational URLs https://help.zoho.com/portal/en/kb/creator/developer-guide/others/url-patterns/articles/navigational-urls

• Zoho Desk - Custom Module Related List Columns on Tickets

  I have a custom module in Zoho Desk called Asana Tasks, each task has a lookup to a Ticket. On the Ticket I want to see the columns of the Asana Tasks in the related list . Is there a way to do this? Right now it just has the name of the record and I

• What is your opinion of the new UI?

  Hi Everyone, I would like to see what everyone thinks of the new Zoho One Dashboard. I don't get it, but perhaps I'm missing something. What are your thoughts?

• What are the OAuth scopes needed to access the Zoho MCP server?

  I think I'm authenticating with the MCP server endpoint successfully using a bearer token but I can't successfully initialize an MCP session. I'm using the official MCP Python sdk (https://github.com/modelcontextprotocol/python-sdk). I get as far as here

• "Temporary Error" in Zoho CRM Data Source sync

  I edited my setup for the Zoho CRM to Zoho Analytics Data Source Import configuration to add some new fields. After I saved the changes, I clicked the (Sync Now) link. After a few minutes I got a Sync failure error. The only reason given is "A temporary

• Account blocked for IMAP use

  Hello, My email client (Evolution) can't sync mail anymore. It gives the error: "Your account is temporarily blocked for IMAP use. This may happen if you exceed the maximum number of simultaneous IMAP connections allowed. Kindly try again after some time."

• Is SMTP included in the free plan?

  My client has a Zoho Mail Free 14 day trial that ends tomorrow. She has set up a domain email address with SMTP on a third party app and is wondering if it will continue working once her plan downgrades to the free forever plan. Thanks

• Zoho mail issue.

  I have verified my domain ohhoexpress.online by adding necessary TXT and MX records. But when I am sending email to any external email id, it is showing as zohomail.com. Also while I am sending mail to internal mail id, it is saying user not found whereas

• My email is blocked from sending

  I get this message when I try to send any message .. > Unable to send message, Reason 554 5.1.8 Email Outgoing Blocked We have urgent emails to send and the issue continues to persist from yesterday. Please help us with this issue as soon as possibl

• [WEBINAR] Smooth year-end closure with Zoho Books (KENYA)

  Hello there, This webinar is for all Kenyan businesses looking to wrap up their financial year smoothly! Join our free session to learn how Zoho Books can simplify your year-end process. What to expect from this webinar: - All the latest updates in Zoho

• Next Page