Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • CRM Deluge how can link attach file with workdrive file.

        Hi, We have create file at workdrive and would like to link to attachment at crm module by deluge. Right now can only upload it to attachment but can not link the file from workdrive. Thank you

      • Feature announcement: AI-powered document generator in Zoho Sign

        Hi there! We recently announced manual document creation by offering an editor within Zoho Sign. Today, we are excited to announce AI-powered document generation to help you draft agreements, purchase orders, and other important business documents in

      • Permanently deleted files

        Is there a way to get back permanently deleted files from your account if they were accidentally deleted and no longer recoverable from the recycling bin? I know I am probably screwed but I just figured I would ask.

      • Shared Mailbox in iOS app

        Hi, On the desktop I am using a Shared mailbox for our company's general mailbox that somebody else should be able to view. In the iOS app however I can't find it? Old sources on the internet says this isn't possible but that can't be right? Somebody

      • Tip #41- Simplify Compliance and Efficiency with Zoho Assist’s Data Cleanup Feature- 'Insider Insights'

        Managing device data efficiently is just as important as securing remote access. With Zoho Assist’s Data Cleanup feature, admins can easily remove outdated or unnecessary device records, keeping the console organized and secure. This action is carried

      • Revenue Management: #6 Revenue Recognition in Professional Services Industry

        If you run a consulting firm, agency, or any service-based business, you have a situation where you have signed the contract, maybe even sent the invoice, and have received some initial payments. So, can you consider this as revenue? What if the project

      • Zoho fsm scheduled maintenance dispatch

        Hi, I'm reaching out regarding an issue I'm encountering with the Scheduled Maintenance feature in Zoho FSM. I’m trying to configure it so that it automatically dispatches all maintenance jobs scheduled within the next 150 days, starting from today. However,

      • zoho.inventory.getRecords returns no records

        I running a custom function (Deluge) in Zoho Flow with a connection to Zoho Inventory. To debug the problem, I created the test function below. It returns "Number of contacts fetched: 0". This confirms that there's no issue with the org ID or connection

      • Feature announcement - Simplifying document creation with Zoho Sign

        Hi there, We are pleased to announce the release of our document creation feature, enabling you to create, edit, and finalize documents like agreements, purchase orders, and contracts without relying on external tools or applications. &lt;br&gt; This

      • Integrating File Attachments in Zoho Flow Email Notifications

        Hi, I would like to be able to send an email once an entry is made in a form, and in this email, I would like to attach a file that has been added to the form using the ImageUpload (or FileUpload) widget. So, I started by creating a flow that will trigger

      • Zoho project purchases

        Hi Folks, I think having a purchase module in projects will help in putting together a project p and L. Project wise purchases and expenses.  Do let me know your thoughts.

      • How can I restore all the deleted Time entries of a Task in Zoho Projects

        How can I restore all the deleted Time entries of a Task in Zoho Projects? Query: In Zoho Projects, I cloned a task and deleted the time entries from the cloned task. However, this also deleted the time entries from the original task. How can I restore

      • Is there a way to make an account inactive in ZoHo Desk

        We have a few Clients "Accounts" that we no longer do business with. It would be beneficial for them to not show up in lists. However, we want to go back and view tickets, time, etc.

      • How to set the value of the Phone field importing contacts in Zoho Desk

        Hi everyone, i'm new in Zoho Desk, we're setting up the environment and i'm importing contacts from another CRM using a file CSV and i'm getting a problem with phone numbers (italian): the leading zero is cut away, also if the value is inside double quotes.

      • How to center a field inside a section?

        I’ve been trying to center a field inside a section in Zoho Canvas. When I align it visually, it looks centered in the editor, but after clicking Save, the field appears misaligned on the actual canvas. I also tried setting the field to full width, but

      • Showing the map along with mileage expense

        When you use the GPS to track mileage, it shows you the map of the actual path travelled. It would be very useful and practical to save that map with the mileage expense, so that when the report is created, it provides a map of each mileage expense associated

      • Enable Validation Rule for Multi-Select Picklist Field

        Zoho, Please allow validation rules for multi-select fields.

      • CRM Custom function updating a module record shows the Super Admin user as the record modifier

        Dear Zoho CRM Team, Is there any way to update this so that when a custom function has updated a record the Super Admin user doesn't become the modifier? This happens on the record as a modifier and shows up in the audit logs. It would be more useful

      • Best way to fetch employee names from Zoho People into Zoho Creator Inventory Stock Form field Employee Name Lookup

        Hi Team, I have a requirement in my Zoho Creator application (Inventory Stock Adjustment) where I need to fetch employee names from Zoho People and use them as a lookup in a form. Currently, I am considering using an integration field to fetch this data

      • get file api is returning Junk data

        I am working on extension development where at one point I need to retrieve attachments on records I found out I can use only invokeconnection and not invokeurl in extension development The invoke connection returns the image in raw binary format. When

      • Need help with message box

        End user updates many records at once. Each have unique serial number. They want a confirmation box that says starting number, ending number and qty. Is there any way to do this? I been searching and asking support but seems like no great solution.

      • [Webinar] The Transformative Power of Gen BI

        Traditional decision-making tools are no longer enough. The integration of generative AI into business intelligence (BI) is proving to be a true game changer—enabling businesses to make faster, smarter, and better informed decisions. Early adopters of

      • Backorder process review - Automating Removal of Sales Order from "On Hold" When PO is Received

        Hello Zoho Inventory Team, Currently, sales orders in On Hold status are released only when the bill for the purchase order is created. In our workflow, it would be much more efficient if the sales order could automatically move out of On Hold as soon

      • How to access the saved Query using API?

        I have created a query in setting. Is it possible to access it from a API? What's the endpoint? I tried /v8/queries/important_deals but didnt' work.

      • création d'une base de données

        base de donnee d'un lycée

      • User Session variables to recall after form submit & redirect to menu

        Hey, Forgive me if this is a simple answer I have overlooked. I have built a big onboarding process for my SAAS platform which is built and managed by an external Dev team. I am hoping to embed my new creator app on the SAAS platform. I have a menu page

      • Zoho Writer Merge Template with data from Zoho Analytics Views

        Hello, Is there any way to pull information from Zoho Analytics and put it into a Merge Template? I am trying to create a bulk export of one report that filters on a field to create a single document for each of the filter fields.

      • Having to enter bill before stock shows as available.

        Hi, Am I right in thinking you must create a bill from the purchase order receipt before the goods are available for shipping?

      • Agent Availability Report

        From data to decisions: A deep dive into ticketing system reports Businesses need to track when their support agents check in and check out from work. This report, titled Agent Availability, is one of the static reports that helps managers track the number

      • AI-driven construction analytics using Zoho Projects Plus

        Construction projects thrive on timelines; if a project is completed on or before the expected time, it makes headlines, but if it gets delayed, the company’s credibility will be at risk. Such projects require precise planning and constant monitoring,

      • Canadian payroll

        Hello ZOHO, is any updates on when payroll for Canada will be available?

      • Unified customer view: Bringing product intelligence into your CRM data

        For businesses today, applications and digital interfaces are the main touch points in the customer journey. While Zoho CRM data already tells who your customers are and what they buy, Zoho Apptics tells you how they interact and engage with your product.

      • Zoho Creator SaaS app with multiple client AND multiple users

        I am close to finalizing a Zoho Creator app that will be used by different companies to manage sale info, and each company can have multiple users using the app. Things were going perfectly until I tried a search on a report. I use a database field in each table to separate the data for each company, which works great. BUT when I do a search, I get to see ALL the data. This means that Company A could easily see the data from Companies B and C via the built-in search. Is there a way to set up the

      • Possible to send Zoom AI Companion transcripts and summaries to contacts in CRM?

        Title says it all. Is it possible to send Zoom AI Companion transcripts and summaries to contacts in Zoho CRM?

      • Clarity on extended contract status

        Clarity on “extended” status- How does Zoho “extend” a contract?  E.g. if client extends by 1 month, can the “end date” be adjusted accordingly in Zoho and tagged as “Extended” with a clear audit trail that also captures the client’s email? Note- Email

      • Customizing contract status

        Can we customize/add/remove status on Zoho contracts ourselves or does Zoho needs to do this? Context- There is a long list of status on Zoho but not all of them are relevant for us . There are few status which are missing and needs to be added. E.g-

      • Set Custom Business Calendars and Holidays for Global Teams

        Managing a project across diverse teams means accounting for more than just tasks and deadlines; it means acknowledging how and when each team actually works. Users might follow different working days or observe region-specific holidays that cannot be

      • [Integration edition] Deluge learning series - Integrations between Zoho apps and Third-party services with Deluge | 28 August 2025

        We’re excited to welcome you to the Deluge learning series: Integration edition! The Integration edition will run for three months: Session 1 – Integrating Zoho Apps with Deluge using inbuilt integration tasks Session 2 – Integrating Zoho Apps with Deluge

      • Critical Vulnerability in all major password manager plugins - is Zoho Vault affected?

        Hi Zoho, a security researcher found a critical clickjacking vulnerability in all major browser password managers: DOM-based Extension Clickjacking: Your Password Manager Data at Risk | Marek Tóth Is Zoho Vault affected as well? Apparently the Vault extension

      • Need Your Insights

        Hi Zoho, I'm confused why the flow only sends to one output. setVariable15 is from a list. It doesn't consider the 2nd entry. Any thoughts?

      • Next Page