Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Special characters (like â, â, æ) breaking when input in a field (encoding issue)

        Hey everyone, We are currently dealing with a probably encoding issue when we populate a field (mostly but not exclusively, 'Last Name' for Leads and Contracts). If the user manually inputs special characters (like ä, â, á etc.) from Scandinavian languages,

      • Set Custom Icon for Custom Modules in new Zoho CRM UI

      • All new Address Field in Zoho CRM: maintain structured and accurate address inputs

        The address field will be available exclusively for IN DC users. We'll keep you updated on the DC-specific rollout soon. It's currently available for all new sign-ups and for existing Zoho CRM orgs which are in the Professional edition. Latest update

      • is there any way to change the "chat with us now" to custom message?

        is there any way to change the "chat with us now" to custom message? I want to change this text

      • Notes badge as a quick action in the list view

        Hello all, We are introducing the Notes badge in the list view of all modules as a quick action you can perform for each record, in addition to the existing Activity badge. With this enhancement, users will have quick visibility into the notes associated

      • Is Zoho Live Chat compatible with WordPress CMS?

        Hello, I have a website called www.jjrlab.com and I'm interested in using Zoho Chat on it. Does it support WordPress CMS? Thanks.

      • Introducing spam detection for webforms: An additional layer of protection to keep your Zoho CRM clean and secure

        Greetings all, One of the most highly anticipated feature launches—Spam Detection in webforms—has finally arrived! Webforms are a vital tool for record generation, but they're also vulnerable to submissions from unauthenticated or malicious sources, which

      • Bring your CRM and Desk app inside SalesIQ with widgets

        Have you ever been confused and frustrated with multiple open tabs and switching back and forth from SalesIQ to other apps to perform your business-specific operations? How effective would it be to have all the required tools and data of the apps you

      • Speak Your Customers' Language: SalesIQ's chatbots now support 30 languages 🤖

        We're unveiling some major upgrades to our chatbot that are set to revolutionize your experience! Now SalesIQ support 30 languages for both Zobot and Answer bot. By speaking your customers' language, you can enhance engagement, improve customer satisfaction,

      • Repeat Column merge in ZOHO writer columns doesn't allow to set max columns per row

        I'm using ZOHO writer to merge data from a ZOHO CRM subform and I want it to make a table. We're using Insert Table for Column Repeat, because this is what we need. (Name of column (Teamname) and underneath that a list of names of teammembers). It works

      • Persisting email signals to contact timeline?

        Hello there. Regarding the email signals that are logged when a contact opens an email, am I correct that these signals are not logged to a contact's timeline? Ideally, I would like to have this context available for future follow-ups, if the contact

      • Trouble with using Apostrophe in Name of Customers and Vendors

        We have had an ongoing issue with how the system recognizes an apostrophe in the name of customers and vendors. The search will not return any results for a name that includes the mark; ie one of our vendors names is "L'Heritage" and when entering the

      • Really want the field "Company" in the activities module!

        Hi team! Something we are really missing is able to see the field Company when working in the activities module. We have a lot of tasks and need to see what company it's related to. It's really annoying to not be able to see it.🙈 Thx!

      • Upcoming update to Google Drive integration in Zoho Creator

        Hello everyone, We're writing to inform you about an upcoming update to how Zoho Creator integrates with Google Drive. In Zoho Creator, during actions like importing files to create an app or attaching files to fields, you can choose files directly from

      • Changing Account Type in Chart of Accounts

        Does anyone know how to change/edit the account type for an Account name in Chart of Accounts. Zoho will not let me do this for some reason

      • Managing Scheduled Replies

        Communication is most effective when the intended message reaches the right person at the right time. However, this can be challenging when communication occurs across different time zones. By scheduling replies, businesses can ensure that responses reach

      • Show price book list price

        When using price books, once you add products to the price book in the Products related list you can display the Unit price which is the default list price; however, there is no option to show the price book list price. To see the price book list price

      • Allow Attaching Quartz Recordings to Existing Zoho Support Tickets

        Hi Zoho Team, We would like to request an enhancement to how Zoho Quartz recordings integrate with Zoho Support tickets. Current Behavior: At the moment, each Quartz recording automatically creates a new support ticket. However, in many real-world scenarios:

      • Issue with open-rate reporting in Zoho Campaigns

        Hello, Since yesterday I’ve been experiencing an issue with the open-rate reports in Zoho Campaigns. The campaigns I send appear in the reports as if none of the emails have been opened, even though I know they have. To verify this, I replicated the campaign

      • Can I hide empty Contact fields from view?

        Some contacts have a lot of empty fields, others are mostly filled. Is there a way I can hide/show empty fields without changing the actual Layout? I would like to de-clutter my view, and also be able to add information later as I am able. I would be

      • Manage user roles and profiles from the Bigin app (iOS and macOS)

        Hello everyone! We're happy to announce that you can manage user roles and profiles from the Bigin mobile app. The Users & Controls section has three subsections: Users, Profiles, and Roles. Users Employees who belong to your Bigin organization are called

      • Support Bots and Automations in External Channels

        Hello Zoho Cliq Team, How are you? We actively use Zoho Cliq for collaboration, including with our external developers. For this purpose, external channels are a key tool since they work seamlessly within the same interface as all of our other channels

      • Zoho Books Sandbox environment

        Hello. Is there a free sandbox environment for the developers using Zoho Books API? I am working on the Zoho Books add-on and currently not ready to buy a premium service - maybe later when my add-on will start to bring money. Right now I just need a

      • Form to PDF template

        I am looking to create a Form and when information is inputted, it will merge it into a PDF template we already have. There could be multiple lines in the form and each line inputs information into a new page of a PDF template.

      • Writing SQL Queries - After Comma Auto Suggesting Column

        When writing SQL Queries, does anyone else get super annoyed that after you type a comma and try to return to a new line it is automatically suggest a new column, so hitting return just inputs this suggested column instead of going to a new line? Anyone

      • Please verify my account

        Hi, We have just launched our website and received media coverage in our country. Hundreds of users are signing up for our platform every day. We would like to send them a welcome email, but we are unable to do so because our ZeptoMail account has not

      • Subject character limit increase

        We have an occasional user case where subject lines exceed 500 characters which is the ZeptoMail imposed limit. While I understand long subject lines are not "best practice", these are particular notification emails with specific information for client

      • Account blocked

        Yesterday I got my Zeptomail account blocked due to too many hard bounces. My account is used exclusively for sending transactional emails (eg. your order has been shipped, a form has been filled, etc) and the sudden blocking impacted hundreds of websites

      • Nested notebooks

        Dear Sir/Madam, I would like to know if it is possible to nest notebooks. It would be very helpful when there are too many, as it would improve organization. Thank you for your response. Best regards.

      • Cannot see correct DNS config for mail after moving domain to another provider

        I have moved my domain from one provider to another and after that zoho mail stopped working (expected). Problem is, zoho mail admin panel still shows (10 hours after move) that all records are correct while I haven't changed anything in my domain DNS

      • Proactive Alert for Existing Open Tickets per Contact / Account

        Hello Zoho Desk Team, Greetings, and hope you’re doing well. We would like to submit a feature request aimed at improving agent awareness and efficiency when handling tickets in Zoho Desk. Use case When an agent opens a ticket (new or existing), it is

      • Add the ability to Hide Pages in Page Rules

        Hi, We have Field Rules to show and hide fields and we have page Rules, but we can't hide a page in Page Rules so it isn't completed before the previous page (And then have the Deny Rules to prevent submitting without both pages completed), we can only

      • Ticket resolution field - can you add links, video, and images?

        Seems like the ticket resolution fields is just a text field. Any plans to add the ability to add links, images...the same functionality in the problem description box? I would like to send the customer a link to a KB article, a link to our Wiki, embed

      • Ticket Resolution - Add rich formatting, screenshots and attachments

        The resolution field only allows plain text at the moment. Many of our resolutions involve posting screenshots as evidence, it would be great for us to be able to have rich text formatting, be able to paste screenshots and add attachments in the solution

      • Deprecation Notice: OpenAI Assistants API will be shut down on August 26, 2026

        I recieved this email from openAI what does it means for us that are using the integration and what should we do? Earlier this year, we shared our plan to deprecate the Assistants API once the Responses API reached feature parity. With the launch of Conversations,

      • CRUD actions for Resources via API

        Hello, is it possible to perform CRUD actions through the API for Resources? We want to create a sync from Zoho CRM Car record to Bookings resources to create availabilities for Car bookings. For Test drives, not only the sales person needs to be available,

      • Kaizen #140 - Integrating Blog feed scraping service into Zoho CRM Dashboard

        Howdy Tech Wizards! Welcome to a fresh week of kaizen. This week, we will look at how to create a dashboard widget that displays the most recent blog post of your preferred products/services, updated daily at a specific time. We will leverage the potential

      • Convert Lead Automation Trigger

        Currently, there is only a convert lead action available in workflow rules and blueprints. Also, there is a Convert Lead button available but it doesn't trigger any automations. Once the lead is converted to a Contact/Account the dataset that can be fetched

      • I'm getting this error when I try to link an email to a deal inside the Zohomail Zoho CRM extension.

        When I click "Yes, associate," the system displays an "Oops!! Something went wrong" error message. I have attached a screenshot of the issue for reference.

      • Zoho Sheet - Desktop App or Offline

        Since Zoho Docs is now available as a desktop app and offline, when is a realistic ETA for Sheet to have the same functionality?I am surprised this was not laucned at the same time as Docs.

      • Next Page