Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Clarity on extended contract status

        Clarity on “extended” status- How does Zoho “extend” a contract?  E.g. if client extends by 1 month, can the “end date” be adjusted accordingly in Zoho and tagged as “Extended” with a clear audit trail that also captures the client’s email? Note- Email

      • Customizing contract status

        Can we customize/add/remove status on Zoho contracts ourselves or does Zoho needs to do this? Context- There is a long list of status on Zoho but not all of them are relevant for us . There are few status which are missing and needs to be added. E.g-

      • Set Custom Business Calendars and Holidays for Global Teams

        Managing a project across diverse teams means accounting for more than just tasks and deadlines; it means acknowledging how and when each team actually works. Users might follow different working days or observe region-specific holidays that cannot be

      • [Integration edition] Deluge learning series - Integrations between Zoho apps and Third-party services with Deluge | 28 August 2025

        We’re excited to welcome you to the Deluge learning series: Integration edition! The Integration edition will run for three months: Session 1 – Integrating Zoho Apps with Deluge using inbuilt integration tasks Session 2 – Integrating Zoho Apps with Deluge

      • Critical Vulnerability in all major password manager plugins - is Zoho Vault affected?

        Hi Zoho, a security researcher found a critical clickjacking vulnerability in all major browser password managers: DOM-based Extension Clickjacking: Your Password Manager Data at Risk | Marek Tóth Is Zoho Vault affected as well? Apparently the Vault extension

      • Need Your Insights

        Hi Zoho, I'm confused why the flow only sends to one output. setVariable15 is from a list. It doesn't consider the 2nd entry. Any thoughts?

      • Map Dependency Fields - How can can I limit picklist values from a multi-select Parent field?

        I know the "Map Dependency Fields" feature doesn't allow a multi-select parent field. However, I need the functionality. Can I do it through a deluge function, and would it still support color coding picklists? We need the color coding for quick scanning

      • DYK 1 - Color Palette for Enhanced Visual Identification of Status

        Introducing the Did You Know series of posts. The goal of this series is to familiarize users with certain features or enhancements in Zoho Projects that may not be evident at first glance. The first post in this series deals with color palettes for indicating

      • Invalid field in the COQL query

        Dear Zoho Support! I believe that you already helped me with a similar problem a few years ago. One of my clients has a custom field named "LOB" in the "Deals" Module (see the field's metadata below). The COQL query using this field: : "select id, Deal_Name,

      • Transferring domain registration to new registrar and switching email hosting at the same time?

        I need to transfer an existing domain uv cure adhesive that's currently with SiteGround to Porkbun. I also need to move the existing custom email addresses from SiteGround to Zoho Mail. I'm not sure if I should transfer the domain first and then tackle

      • Split deposits

        Can Zoho do split deposits. One deposit, two checks for two separate invoices from different customers. This is one of the most common tasks I can imaging. When I mark the two invoices paid, there are two deposits in bank register. When I try to match,

      • Deactivate Desk Contact without Deleting Contat

        We have a client who has multiple tenants for regulatory purposes, and as such, has a few users that have email addresses in both tenants. They've then emailed into the ticketing system, so we have multiple contacts (no big deal, we want to keep their

      • [Free Webinar] Learning Table Series - Zoho Creator for Asset Management with AI Enhancements

        Hello Everyone! We’re excited to invite you to another edition of Learning Table Series, where we showcase how Zoho Creator empowers industries with innovative and automated solutions. Each month highlights a specific sector, and this time our focus is

      • Is there any way to get notified when a failure execution happens on a Workflow ?

        Hello, I wanted to know if it was possible to get notified (like with a mail for example) when a single failure happens on a specific flow of my organization ? I know that there's a mail which warns you when there's 20 failure in a row on the same flow,

      • Menu Building is completely broken

        I have been 3 hours, I have not been able to edit the menu. Either it is completely broken, very little intuitive or I do now know anything... There is no way to create a megamenu, no way to create a menu. Despite the fact I go to menu configurartion

      • Can you sell Subscriptions using Zoho Commerce?

        In addition to physical products and the apparently coming soon 'Digital Products', it is possible to sell Subscriptions using Zoho Commerce?

      • Kaizen #197: Frequently Asked Questions on GraphQL APIs

        🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

      • Multiple Languages for Product Names

        Hi, I use 2 languages: spanish and english. I want to have for every product a name in spanish and a name on english. I want to have to possibility of choosing one of these languages when making an invoice or a purchase order. Is there any way to do

      • Item with name in different languate

        Hello, is there a way to have an item with its name in different languages? For example: I sell an item in different markets and I'd like to have a Proposal and the Invoice with the Item Name in a specific language. Rino Bertolotto Zoho Specialist, STESA srl

      • Contacts with most tickets? Alarm for multiple tickets?

        Is it possible to see through the analytics/reports which contacts are creating the most tickets (not the most discussed ones)? Also, is there a way to set up a notification if a contact creates multiple tickets within a certain time frame?

      • Issue with Template Subject Line Format in Zoho CRM

        Hi Team, I’ve noticed that when I update the subject line of an email template in Zoho CRM, it sometimes appears in an incorrect format when used. Please see the attached screenshot for reference. Kindly look into this issue and fix this issue from backend

      • Two Data Labels in Bar Chart

        I need to create a bar chart that has both the SUM and COUNT. I've concatenated them into a formula but it converts it into a stacked bar / scattered chart. The bar chart is no longer accessible. Since i'm comparing YOY, it would be best to have it in

      • Disable field on subform row

        Hi, Is it currently possible to disable a row item on a subform? I was just trying to do something whereby until another value is entered the field is disable but for the deluge scripting interface threw up an error saying such a function is not supported on a subform. Thanks in advance for your help. Shaheed

      • Leads - Kanban view fit to screen

        Hey guys, I created a custom layout for my leads, staged by lead status. I have 10 types of status. In Kanban view I see only 4 columns/stages and need to scroll to the right to see the rest. Is there a way to make columns/stages be displayed all together?

      • Request to Differentiate Auto-Closed WhatsApp Conversations in SalesIQ

        Hi Zoho Support, I’d like to raise a request related to the way WhatsApp conversations are auto-closed in SalesIQ. Every Monday, our Sales team has to manually review each closed WhatsApp conversation from the weekend to identify which ones were automatically

      • Kanban View UI gets a revamp

        Hello everyone, In the coming week you will notice design related enhancements in Kanban View. The UI has been changed and a new option is introduced under Kanban View Settings that allows to change the color of the category headers.  Please, note that the functionality is not changed. These changes will not apply to the Activities and Visits modules. Here are the details of the changes: 1. The column widths have been fixed to 300 px. The records will have a box around them for clear distinction.

      • Issue with Anchor Link on Zoho Landing Page (Mobile/Tablet View)

        Hi Team, I have created a landing page using Zoho Landing Page and added an anchor link to it. The anchor link is working fine on desktop view; however, it does not work properly on mobile or tablet view. I’ve tried debugging this issue in multiple ways,

      • Simplest way to convert XML to a map?

        I've reviewed the help info and some great posts on the forum here by Stephen Rhyne (srhyne). At the moment I'm using XPath to generate a list of xml nodes, iterating through that to fetch the field name/value pairs and adding them to a map (one map for each record in the data). I then convert the row map to a string and add it to a list. Here's the function: list xml.getRecordListFromXML(string xml_data, string ele_name) {     result = List();     // get list of record nodes     rec_list = input.xml_data.toXML().executeXPath("//"

      • Introducing Creator Simplified: An exclusive learning series to enhance your app development skills

        Hey Creators! Welcome to Zoho Creator's new learning series, Creator Simplified. In this series, we'll dive into real-world business use cases and explore how to translate your requirements into solutions in your Creator application. You can also expect

      • [Product update] Updated Data Synchronization Process for QuickBooks - Zoho Analytics Integration.

        Dear QuickBooks integration users, We’re making an important update in the way data is currently synced in your QuickBooks integration within Analytics workspace. What’s changing: Previously, with every data synchronization, Zoho Analytics used to fetch

      • Zoho CRM new calander format cannot strikethrough completed task

        Hi, Recently there is a new format for calendar within Zoho CRM However, found out that a completed task will not cross out or strikethrough like previous format. Without strikethrough, it will be difficult to identify which task is still in Open status.

      • Employees not Users

        Hello, We are a construction company that has +180 employees and most of them are in remote location working onsite with no access to internet. Is it possible that we have data stored for all employees but have only 5-10 users who will be in charge of entering employees data? or do we have to pay for all +180 employees? even though they won't be using the system?

      • Zoho people generatimg pdf

        Hello , now i want to make a customm button in zoho people that is inside a deduction module , that fetches all the records and generate a pdf with a template that i have done in the mail merges template , i was told that i have to upload template on

      • zohorecruit.com career form postcode bug

        Dear, When I select a postcode from the drop down on a zohorecruit.com career form, the street text field is automatically filled with the name of the city, which should not happen. Any idea how I can fix this? Thanks, Bart

      • Office-365-agenda and Microsoft Teams Integration

        Dear, I have a trial version of Zoho Recruit and trying to evaluate the Microsoft Teams Integration in Zoho Recruit. After registering with my Office 365 account and checking the result of the registration/sign-in at https://mysignins.microsoft.com/ (which

      • How to add Connector in developer platform zoho?

        Hi, I am working on creating an Extension, and part of the development is to retrieve Email templates. In my CRM instance I can invokeURL by creating Zoho OAuth connection and get the template. But developer platform does not provide Zoho OAuth or any

      • How to archive Lost/Junk Leads so sales reps don’t see them, but keep them for reporting?

        Hi everyone, In our Zoho CRM we have two Lead Status values: Lost Lead and Junk Lead. What I want to achieve is: When a lead is marked as Lost or Junk, it should disappear from my sales reps’ Lead views (so they only see active leads). At the same time,

      • Zoho CRM Canvas Copy Original Layout

        Hello all, I want to use Canvas to make small changes to certain views, not to make huge changes. Is it possible to copy the original Zoho layout and set-up and start from there? I checked and all I can find are some templates which are far from the original

      • Revenue Management: #5 Revenue Recognition in SaaS

        If you're building or running a SaaS business, you've probably encountered this. You get paid upfront for a subscription and a one-time onboarding fee, but you end up with confusion about when to consider it revenue. Can I book all of it now? Should I

      • Zoho Learn Course Access Issue

        One of the learners in a specific course can't see any lessons. They are registered as both a user and learner for this course in Zoo Learn. What could be the reason?

      • Next Page