Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Zoho DataPrep switching Date Format

  When using a pipeline that is importing Zoho Analytics data into Zoho DataPrep, the month and day of date fields are switched for some columns. For example, a Zoho Analytics record of "Nov. 8, 2025" will appear in Zoho DataPrep as "2025/08/11" in "yyyy/MM/dd"

• Using Another Field Value for Workflow Field Update

  I'm trying to setup a Workflow with a "Field Update" action on the Lead module, but I would like the new value to actually be taken from a DIFFERENT Field's on the Lead record (vs just defining some static value..) Is this possible? Could I simply use

• Tax information

  Hello, I need help/guidance on how to add my organization's Tax/VAT information. Thank you Pavly

• Build smarter Guided Conversations with Fork Blocks

  When your customers arrive on your support channel, they're not there to explore. They are usually confused and stuck while trying to fix something important. We understand how stressful that moment can feel and we want your bot to make things easier,

• Custom item field won't allow decimal.

  Hello, I have a custom item field that needs to be able to have a value with a decimal place such as 6.7 or 6.18. I have tried custom formatting the input format but can not get the correct syntax to allow this. Is this possible in Zoho?

• Free webinar: Zoho Sign unwrapped – 2025 in review

  Hey there! 2025 is coming to an end, and this year has been all about AI. Join our exclusive year-end webinar, where we'll walk you through the features that went live in 2025, provide answers to your questions, and give you a sneak peek on what to expect

• HubSpot CRM to Zoho Creator Integration

  I'm trying to create an integration between HubSpot CRM and Zoho Creator with HubSpot being the push source (if a contact is created or updated in HubSpot, it pushes the information to Zoho Creator). I have two questions: 1- Is it best to use Zoho Flow

• Gain control over record sharing with portal users through our new enhancement: criteria-based data exposure

  Dear Customers, We hope you're well! Portals is a self-service avenue through which your clients can access and manage their direct and related data in Zoho CRM. This empowers them to be more independent and enables them to experience a sense of transparency

• Best-practice setup in Zoho One for managing combined candidate pools and exporting anonymised CVs

  We are new users of the Zoho One bundle and operate a consulting and engineering company. Our workforce model includes a mix of permanent employees, active job applicants, and freelance/independent consultants. All three groups need to be searchable,

• Meet Canvas' Grid component: Your easiest way to build responsive record templates

  Visual design can be exciting—until you're knee-deep in the details. Whether it's aligning text boxes to prevent overlaps, fixing negative space, or simply making sure the right data stands out, just ironing out inconsistencies takes a lot of moving parts.

• hard-bounced email list

  Hi, Below pops up when I try to send an Email to some of my customer. Please guide me how to take it out from hard-bounced list? I am not sure how they were marked in hard-bounced list

• Register the 'Contact Role' addition and change as a Potential edition so it can trigger Workflows

  We are trying to use "Contact Roles" in Potentials. Contact Roles are special and different than the other Related lists, so, it may have a special behavior. Something to keep in mind is that you will never have 100 Contact Roles as you can have 100 Tasks, Calls, or any other Related list. In our case we will have 2 in average and up to 4 or 5 maximum. The problem is that we need to bring information from 3 key Contact Roles to the Potential and adding a Contact to the Contacts Roles area never trigger

• Can I change the format of the buttons in the email templates?

  Hi all! We have been working hard trying to brand our email templates, and have some way to go yet. One of the things we can't seem to edit is the green ${Cases.CUSTOMER_PORTAL_BUTTON} button and the font of the View Ticket text. Is there any way of doing

• Best practice to structure reporting to include events covering multiple months / quarters.

  Hi, I'm new to Zoho, have some experience of more "enterprise" tools, looking for some input from the community. I'm looking to create a report that includes events that cover a long period, each event has a start / end date and I'm struggling undertanding

• Marketing Tip #11: Turn features into benefits that sell

  We all love talking about our products, but here’s a secret: customers don’t just buy features, they buy benefits. Instead of just saying "Made from 100% organic cotton," try "Soft, breathable comfort that lasts all day." Benefits tell shoppers how your

• Synchronise item image between Zoho Commerce and Zoho Books/Inventory/CRM

  Here is a blindingly simple idea to tie several Zoho products together. Zoho - please include a method to synchronise the item image (or images) from one Zoho application to another. For example, if you upload an item image in Zoho Inventory, a user should

• How do I migrate from zoho mail to Office 365?

  The manual migration of Zoho Mail to Microsoft 365 typically requires using IMAP to move emails and configuring Microsoft Outlook to sync Zoho Mail. This approach can be error-prone, especially if there are multiple accounts or large email archives. Moreover,

• New Update: Convert External Users in Bulk in Zoho Directory

  Greetings to all Zoho Directory users out there! We’re excited to introduce a new update that makes user management in Zoho Directory even more efficient; you can now convert external users in bulk! Earlier, admins could convert only one external user

• New Update: Convert External Users in Bulk in Zoho One

  Greetings to all Zoho One users out there! We’re excited to introduce a new update that makes user management in Zoho One even more efficient; you can now convert external users in bulk! Earlier, admins could convert only one external user at a time.

• How to edit form layout for extension

  I am working on extension development. I have created all the fields. I want to rearrange the layout in Sigma platform. But there is no layout module in Sigma. How can I achieve this for extensions other than Zet CLI and putting the fields into widget

• Website not properly connecting with Zoho Creator app portal (embed & data sync issue)

  Hello Zoho Community, I’m currently facing an issue while trying to connect my external website with a Zoho Creator app portal. I have a tool-based website ( https://mygardencalculator.com/ ) where users interact with calculators and dynamic content.

• 日本語フォントの選択肢拡充についての要望

  日本語フォントの選択肢がとても少ないと感じたことはありませんか？ 多くのアプリ（たとえば Invoice）ではフォントが1種類しかなく、正直あまり使いやすい・見た目が良いとも言えません。 そろそろ、もっと多くの日本語フォントを追加してほしい、そしてすべてのアプリで同じフォント選択肢を使えるようにしてほしいと、私たちユーザーが声を上げる時期だと思います。 ご存じのとおり、現状ではアプリごとにフォント周りの仕様にほとんど一貫性がありません。 みなさん、一緒に要望を出していきましょう！

• Does anyone know how to setup Zoho Desk or Zoho CRM as a custom outgoing/incoming Call Centre?

  I need to setup a call center so I can setup agents to make phone calls across Canada to market our services.  I am trying to figure out the most reliable and cost efficient way to do this? I am currently paying for two phone services and neither seem

• Display actual mileage on an invoice

  My users are creating expenses in Zoho expense. For example, they expense 10 miles and get paid 7 dollars (10 miles * IRS rate of .70). If I look at the expenses in Zoho Books, it does show them at 10 miles at .70 cent When I add these expense to an invoice

• Prevent Unapproved Quotes from Exporting to Zoho CRM Finance Module

  Is it possible to prevent unapproved quotes in Zoho Books from being exported from Zoho Finance module inside Zoho CRM?

• Default font size for printing is too big

  A recent issue in printing e mails is that the default font size is huge. This happens in both Edge and Firefox. In order to get what I would call a "normal" printout of an e mail it is necessary to go into the printer preferences / options and set the scale to 75%, otherwise a short e mail with signature and logos etc printed at 100% can take 3 or 4 pages. The annoying thing is that it is necessary to do this each time a printout is to be made as a change in scale only applies to that particular

• Request for Auto PO - Min–Max based Automated Purchase Feature

  Dear Zoho POS Team, I’m writing to request a feature enhancement that would significantly streamline inventory management for businesses using Zoho POS — particularly supermarkets, FMCG retail, and multi-store operations like ours. Feature Requested:

• Export Invoices to XML file

  Namaste! ZOHO suite of Apps is awesome and we as Partner, would like to use and implement the app´s from the Financial suite like ZOHO Invoice, but, in Portugal, we can only use certified Invoice Software and for this reason, we need to develop/customize on top of ZOHO Invoice to create an XML file with specific information and after this, go to the government and certified the software. As soon as we have for example, ZOHO CRM integrated with ZOHO Invoice up and running, our business opportunities

• issue with deluge script

  i used chat gpt to build this script and I am getting 2 errors which I cannot figure out how to fix: void monthly_sales_order_generation() { try { // ---------------- CONFIG ------------------- analytics_url_1 = "https://analytics.zoho.com/api/<workspaceID>/report/<reportID1>/data";

• Introducing LinkedIn Apply Connect for Zoho Recruit

  Attract up to 3x more qualified candidates and hire more efficiently with LinkedIn Apply Connect. Let candidates fill-in job applications without any redirections, gain deeper insights with applicant highlights within Zoho Recruit, and keep candidates

• Recruit paid support?

  Hi all, Could anyone who has paid support package advise if it provides value for money with regards to support response times? Exploring the idea as unfortunately when we have faced issues with Recruit it has been a 7+ day timescale from reporting to

• Ticket Status email

  Good day, This was discussed in the past, but it would be helpful if we could have the system assign a custom response to a status. We have various statuses for tickets, e.g. "closed due to no response", or "Pending Status", it would be helpful for the

• Future Orders - Due Date

  Hi In my role, I can receive tickets where the work required is requested months in advance. Using a Future Orders option, which I believe was setup under the On Hold status type, hides the Due Date, in all views/ticket etc. Whilst I understand the reasoning

• Introducing Withdrawal Reasons for Offers

  We’re excited to introduce a new enhancement to the Offer module that brings more clarity and accountability to every withdrawn offer. The Withdrawal Reason update ensures that each withdrawal — manual or automatic — is backed by a clear explanation,

• Zoho Creator customer portal users

  Hi, I'm in a Zoho One subscription with our company. I'm running a project now that involves creating a Zoho Creater application and using the Zoho Creator Customer Portal.  At most we need 25 customer portal users. In our Zoho One plan we only get 3

• GPS tracking only or Check out facility

  Dear Team, Zoho CRM is quite robust CRM but till date i was waiting for the feature of having GPS tracking of Sales employees which is a common demand by all customers for thier field sales executives. We cover them by saying that CRM provides Checkin

• Appraisals on Employee Information Profile

  Is it possible to show completed appraisals on each employee's "Employee Information" page? I would presume at the bottom - similar to the "Related Lists" concept in Zoho CRM. Obviously view access would be limited to employee and appropriate other roles

• Kaizen #220: Actions API - Webhooks APIs - Part 2

  Hello all!! Welcome back to the follow-up Kaizen post of Kaizen #219: Actions API - Webhooks APIs - Part 1. In the previous week, we covered how to configure a basic Webhook and how to include Headers, Body, and URL Parameters using both the POST Webhook

• Standard Payment Term is not pulled from account to quotation

  Hey Team There seems to be something off. I do have "Net 30" as my default payment term in Zoho Books for my customers. If, from the customer overview or quote section, I create a new Quotation, the payment terms field stays blank and doesn't get the

• Manage your invoices from Bigin's mobile app (iOS)

  Hello everyone! We're happy to announce that users can now integrate Zoho Books with the latest version of Bigin's iOS app. Zoho Books can be integrated with Bigin only via the web application. Users can view the Zoho Books tab in the detail pages of

• Next Page