Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • ZIA "Generate Content" action doesn't have contexual data from the ticket

        "Generate Content" action doesn't have contexual data from the ticket. I try to get AI to help me with this ticket but it doesn't seem to have any ticket information as context. Although the ticket has a lot of information in it.

      • Zoho Desk - Zoho FSM Integration issue on Mobile and iPad

        Hello Team, I am trying to create a Work Order (WO) using the Zoho FSM integration (Add-on Service) that is integrated with Zoho Desk. The issue is that the integration is not working on mobile devices and iPads. While I am able to create the WO, Request,

      • E-File Form 1099 Directly With the IRS From Zoho Books

        The Form 1099 filing season has begun, and businesses are required to e-file certain forms with the IRS to report payments made to vendors and contractors. If your business made qualifying payments during the year, you must e-file the appropriate Form

      • Suggestions for showing subscribed Topics in CRM (contact record)

        We have several Topics set up in ZMA. We also have a sync set up between ZMA and CRM. I'd like to display the subscribed topics on the CRM Contact record. This will allow the Sales team (who uses CRM) to see at a glance what topics a Contact is subscribed

      • Replies sometimes creating separate ticket

        Sometimes when a customer responds to an email coming from Zoho Desk, instead of adding a reply to the original ticket, a separate ticket is created. This happens even though the response subject line contained the ticket number, and the person responding

      • Re-hide fields when option is unselected

        Hi all Can anyone help me with this - when I create a 'show' field rule for when a dropdown option is selected, how to I make it so the 'show' option re-hides if that option is no longer selected?

      • Custom Fields

        There is no way to add a custom field in the "Timesheet" module. Honestly, the ability to add a custom field should be available in every module.

      • Consultant-Only Booking Page

        Zoho Bookings does not allow for Meeting Type OR Workspace-Wide booking pages to be turned off. This is detrimental to organizations that have territory-based or assigned accounts, because if prospects can go to these booking pages and either select the

      • Call result pop up on call when call ends

        I’d like to be able to create a pop up that appears after a call has finished that allows me to select the Call Result. I'm using RingCentral. I have seen from a previous, now locked, thread on Zoho Cares that this capability has been implemented, but

      • Custom Sorting based on other columns in table

        I need the ability to apply custom sorting to a text-based dimension in the X axis where the sorting is based on another column in the table. For example, I have a chart report where the X axis is a text label. I would like to be able to sort those text

      • Feature Request: Enable Custom PDF Layout Editor for All Modules (Including Package Slips)

        Hello Zoho Community and Product Team, I am writing to share a suggestion that would significantly enhance the customization capabilities within Zoho Books. We all appreciate the power of the Custom PDF Layouts (the "New" template engine) that allows

      • Is there a way to invoke deluge function from within a widget?

        Hi! I have custom functions in deluge and I was wondering whether there is any way to call this function through a widget? Something like on click of a button inside a widget, run the deluge custom function. Would this be possible?

      • Can a default task Priority be set?

        The "Priority" field in the Task layout does not allow a default to be set. Is there another way of doing it? Because the current default is "None" and the Zoho Kanban board design has selected this field as critical information to surface by including

      • Adding a threshold to a line chart based on date range

        I have a line chart that is tracking a percentage over time. It also has a filter for 50 different clients. I would like to create a threshold that is based on a portion of the date range. As I understand it, this would be done by adding a column to the

      • Resize Signature field dynamically

        On the tablet, it is perfect. But on smaller mobile devices and PCs, both web and application, it is too small for people to sign. Is there any plan to make the signature field size dynamically in the future update?

      • Super Admin Logging in as another User

        How can a Super Admin login as another user. For example, I have a sales rep that is having issues with their Accounts and I want to view their Zoho Account with out having to do a GTM and sharing screens. Moderation Update (8th Aug 2025): We are working

      • Zoho Creator Application - New User Not able to access the application

        In Zoho Creator, The newly added user not able to access the " Added Application" - User has received the Invitation Email, but while clicking "confirm Account" in the invitation Email, the following error message has appeared. "Sorry! you cannot accept

      • Deleting Salutation Field

        We have updated our lead input screen and 'Salutation' has appeared. This is not visible in the 'Edit Pgae Layout' screen so cannot be moved to 'List of Removed Fields'  Salutation is visible in the list in 'Customization - Fields' however I can only 'Edit' or 'Replace' I cannot delete and I do not need this field on my lead input screen.  Please can you advise how to get rid of this.  Screen shots can be provided if needed.  Thank you Tasha

      • Auto-Generate & Update Asset Serial Numbers using a custom function (Assets Module)

        Hello Team, I’ve been working on a script to automate one of our processes in Zoho FSM, and the core functionality has been successfully implemented. However, I’m encountering an issue related to serial number allocation, which is not working as expected.

      • Zoho Mail iOS app update: Access Delegated Mailbox.

        Hello everyone! You can now access the delegated mailbox from within the iOS version of the Zoho Mail app. To access the delegated mailbox: Open the Zoho Mail app. Go on to the 'Email' module. Tap the profile picture. Choose the delegated mailbox Please

      • How to convert Lead's country field from Text to Pick List

        Hi, I would like to change the default country field in ZCRM from text to pick list. It looks like not I can't delete default country field and recreate it as pick list nor can i create an new custom field country because such a label belong to default field. So what do I have to do? Any ideas? L

      • How create a draft via workflow?

        I wish to create a workflow rule for specific emails that creates a draft response - not an automatic email reply, but just a draft with a set response ready to be verified by an agent who can then manually select recipients. Alternatively, the workflow

      • Function #51: Transaction Level Profitability for Quotes

        Hello everyone, and welcome back to our series! In a previous post, we shared a custom function that could determine the profitability of a Sales Order. Today, we are presenting a similar function that calculates the profitability of a Quote. The setup

      • New feature: Invite additional guests for your bookings

        Hello everyone, Greetings from Zoho Bookings! We are happy to announce the much-awaited feature Guest Invite, which enhances your booking experience like never before. This feature allows additional participants to be invited for the bookings to make

      • Improved Contact Sync flow in Google Integration with Zoho CRM

        Hello Everyone, Your contact sync in Google integration just got revamped! We have redesigned the sync process to give users more control over what data flows into Google and ensure that this data flows effortlessly between Zoho CRM and Google. With this

      • الخصم على مستوى فاتورة المبيعات

        السلام عليكم ورحمة الله وبركاته مطلوب في إنشاء خصم على مستوى فاتورة المبيعات وليس على مستوى البند أريد معرفة الطريقة؟

      • VAT and Taxes option not available

        Dear ZOHO Team , The VAT and Taxes options in my ZOHO books account not available,I tried to find how to enable or check the way to use this option but unfortunately couldn't find it anywhere ,I'm in UAE ,kindly let me know what to do to solve this issue

      • Default Tagging on API-generated Transactions

        If one assigns tags to an Item or Customer, those tags get auto-populated in each line item of an Invoice or Sales Order when one creates those documents. However, if one creates the Sales Order or Invoice via the API (either directly coding or using

      • Direct Feed (Bank)

        Is Direct feed integration for AlRajhi and ADCB bank supported by Zoho Books in GCC/Saudi

      • Sales Order, Invoice and Payment numbers

        Hi zoho friends, it is me again, the slow learner. I'm wondering if there is a way to have it so the Sales order, invoice and payment numbers are all the same? It would be easier for me if they were the same number so there is not so many reference numbers

      • Customer Satisfaction (CSAT) Report

        From data to decisions: A deep dive into ticketing system reports The customer satisfaction (CSAT) report helps teams understand how customers feel about their support experience, identify service gaps, and continuously improve the help desk. It turns

      • Timeline Tracking Support for records updates via module import and bulk write api

        Note: This update is currently available in Early Access and will soon be rolled out across all data centers (DCs) and for all editions of Zoho CRM. The update will be available to all users within your organization, regardless of their profiles or roles.

      • Shifts in Zoho People vs Zoho Shifts?

        Hello Zoho People Team, We hope you are doing well. We are evaluating the Shifts functionality within Zoho People and comparing it to the standalone Zoho Shifts product. We’ve encountered comments and discussions suggesting that the Shifts feature inside

      • Disable fields in During action in Blueprint?

        Hi there. I've tried field disable (setReadOnly(true)) using client script and the event is onMandatoryFormLoad on detail page, assuming it'll work on blueprint fields, but it bears no result. Is this the expected behaviour? That we can't do this yet?

      • Develop and publish a Zoho Recruit extension on the marketplace

        Hi, I'd like to develop a new extension for Zoho Recruit. I've started to use Zoho Developers creating a Zoho CRM extension. But when I try to create a new extension here https://sigma.zoho.com/workspace/testtesttestest/apps/new I d'ont see the option of Zoho Recruit (only CRM, Desk, Projects...). I do see extensions for Zoho Recruit in the marketplace. How would I go about to create one if the option is not available in sigma ? Cheers, Rémi.

      • Subforms and automation

        If a user updates a field how do we create an automation etc. We have a field for returned parts and i want to get an email when that field is ticked. How please as Zoho tells me no automation on subforms. The Reason- Why having waited for ever for FSM

      • Allow Managers to Create Shifts for Their Departments in Zoho People

        Hello Zoho People Product Team, Greetings and hope you are doing well. This feature request is related to Zoho People - please don't move it to zoho one! We would like to submit a feature request regarding shift management permissions in Zoho People.

      • Zoho Learn and Zoho CRM integration

        I would like to see an integration between Zoho Learn and Zoho CRM. 1. To be able to add articles in a related list in all modules 2. Zia to suggest related articles in a Deal or Case or Lead 3. Ability to read / search articles during a call / follow

      • Maintain steady traffic to your domain: How Domain Aliasing helps

        Consider this scenario: An organization has its primary domain as administrator.com. Now it wants to shorten its domain to admin.com because it's simpler and easier to remember. However, changing the domain completely can cause the following problems:

      • Why Sharing Rules do Not support relative date comparison???

        I am creating a Sharing Rule and simply want to share where "Last Day of Coverage" (Date field) is Greater than TODAY (Starting Tomorrow). However, sharing rules don't have the option to compare a date field to a relative date (like today), only to Static

      • Next Page