Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Create a table in the layout section of modules

        Dear Sirs, I would like to know whether is possible to create a table under new fields in layouts section of modules. In fact in this section there are single line, multi line, pick list, multi-select, date, formula etc. but I am unable to add a table

      • Nimble enhancements to WhatsApp for Business integration in Zoho CRM: Enjoy context and clarity in business messaging

        Dear Customers, We hope you're well! WhatsApp for business is a renowned business messaging platform that takes your business closer to your customers; it gives your business the power of personalized outreach. Using the WhatsApp for Business integration

      • Power of Automation :: Automatic Task Status Handling for Users

        Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate

      • Question - why no way to input a 'tool description' and 'tree of 'tools'

        Every business is different, with different business processes. To be truly useful Zoho MCP needs to have user editable tool descriptions (or ruleset) and a 'tool tree' so that the LLM is context aware when being used. For example, the tool description

      • Develop Zoho Meeting as a Full Native Application (Not a Browser Wrapper)

        Hello Zoho Meeting Team, Hope you are doing well. We would like to suggest an important improvement regarding the Zoho Meeting desktop application. At the moment, the Zoho Meeting app feels more like a mini browser window or an iframe that loads the web

      • Can I create a custom function for chart of accounts?

        I'm trying to add a custom function that should run whenever a record in the Chart of Accounts module is edited, but on the form for creating a new function, the module dropdown does not include the chart of accounts as an option.

      • billing

        hi, I am being billed $12/year, and I can't remember why. My User ID is 691273115 Thanks for your help, --Kitty Pearl

      • Unable to confirm Super Admin assignment — confirmation button not working

        I’m trying to change the roles within my organization. I am currently a super admin and would like to add another user as a super admin. When I attempt to confirm the action, a screen appears asking for my password to verify my identity. However, when

      • Feature Request: Reviews

        Any chance we could get a "Reviews" element to put on sites? It should be extremely simple to do. Basically just take the comments box and add an option for a star rating. At the very least, it would be nice to have more customization options for the comment box so the text could be changed from "Comments" to "Reviews". I've looked at a bunch of embeddable review widgets but nothing meets my needs so this would be a great feature to have added to the site builder.

      • Can we handle a support like (incident management) project in Zoho Projects?

        Hi, I have a new profile of a project whereby we provide "ticket" base support to a client. They have a request and ideally we would handle comms via a email exchange logged in Zoho. Today we use Zoho Projects for all out projects, which means that we

      • Trying to show the actual Terms & Conditions on PDF

        Hi, On Zoho forms I am trying to have the actual terms and conditions that the user needs to accept also show on the pdf that they receive after. Right now it only says "Agreed". Please help.

      • Installing EMAIL Setup in New Domain

        Respected Support team, I'm facing an issue with cloudflare in Pakistan, I want to setup Zoho Mail Setup but I Don't know how to enable Zoho mail setup without cloudflare. My Website https://stumbleguyzzapk.com/, https://fakservices.com/ is using CF,

      • Enhancements to Zoho Meeting Annotator

        Hello Zoho Meeting Team, Hope you are doing well. We would like to share a few improvement suggestions regarding the Zoho Meeting Annotator used during screen sharing. While the current version provides helpful annotation tools, there are several limitations

      • Zoho Social/Marketing Plus - Addition to "Monitor" function

        It would be very helpful if the Monitor function would allow us to add a column to monitor hashtags in addition to pages and mentions. This is a common and very valuable function in other social listening tools.

      • Zoho forms - > Zoho desk multiple agents

        Hi! I would like to use a standard form to be used when we hiring people, so we looked at zoho forms for that. Now we would like to get that info in to Zoho desk and their different users shall take care of this ticket simultaneous . So, what would be

      • Almacenamiento

        Hola, Quisiera saber como podría hacer para bajar el almacenamiento de 5gb a mis usuarios, en otras palabras los quiero ir limitando de la cuota real, y luego ir agregando poco a poco la cantidad hasta llegar a los 5gb que me dan en el plan free. 

      • Will I Get a Refund If I Downgrade Zoho Mail?

        Hello, We upgraded an email account for our new employee. However, the employee left after one month, and now I've reduced the number of Zoho Mail users from 7 to 6. Can we get a refund for the remaining portion of our annual payment?

      • Introducing the revamped What's New page

        Hello everyone! We're happy to announce that Zoho Campaigns' What's New page has undergone a complete revamp. We've bid the old page adieu after a long time and have introduced a new, sleeker-looking page. Without further ado, let's dive into the main

      • How do i follow up my email campaign in-thread

        Is there a way to follow up the email campaign so that it is in-thread using zoho campaigns? eg customer gets original email with subject line "hello" then 5 days later follow up would be with subject line "RE: hello".

      • Announcing new features in Trident for Mac (1.29.0)

        Hello everyone! Trident for macOS (v1.29.0) is here with new features and enhancements to enhance your business communication. Let's take a quick look at them. Access shared mailboxes. You can now view and access shared mailboxes in Trident, which are

      • Books is extremely slow again today !

        Everything is running slowly even with 500mb connection speed

      • Cyclic dependencies in many-to-many relationships...

        I have an application which includes a form for companies, and a form for contacts. Each company can be assigned 1 technical and 1 administrative contact. I have this working okay so far, but I want to copy the scripts used so far to a new empty application. When I import the scripts it fails with a message that says: Problem encountered while creating the application Error in resolving form dependency:Cyclic dependency among the forms:[Company, Contact] What can I do to resolve this? After all,

      • Zoho API to create ticket

        I'm developing an integration to create tickets via API, but, locally it works (send and recieve requests). In production it also works sending requests, but, my file don't recieve any response data. My URL is available in Zoho API Console and I have

      • Automate Timesheet Approvals with Multi-level Approval Rules

        Introducing Approval Rules for Timesheets in Zoho Projects. With this automation, teams can manage how timesheets are reviewed and approved by setting up rules with criteria and assigning approvers to handle submissions. Timesheet, when associated to

      • Labels Part 2

         Hey Zoho Mail Team, On the labels window on the left-hand pane, alphabetize label name display, rather than displaying in order created.  This one should be easy. Thanks, Drew

      • Building Toppings #1 - Serving your needs with Bigin Toppings

        Hey Biginners! We're excited to kick off our Developer Community series on building toppings for Bigin, and our goal is to provide an accessible, beginner-friendly, and relevant path for every developer. Imagine creating tiny pieces of software that unlock

      • Can we create Sprint with tasks from Multiple projects?

        Hi Team, We were using Zoho Sprints for quite sometime. Currently we have started the process of Sprint method. We couldnt create the active sprint board with the tasks from multiple projects. I would like to know whether this is possible or Any timeline

      • Tip of the Week #74– Create automated workflows in MS Power Automate

        Zoho TeamInbox now connects directly with Microsoft Power Automate, letting you streamline everyday routines tasks such as from sending emails to managing threads, with automated workflows. About the integration Zoho TeamInbox integrates with Microsoft

      • Account validation

        Hello everyone, I registered my account on ZeptoMail to use the system, but the problem is that the verification period on Zepto's end has already passed and I have limited functionality.

      • Paste issues in ZOHO crm notes

        Hi, since a week or so I have issues with the paste function in ZOHO CRM. I use "notes" to copy paste texts from Outlook emails and since a week or so, the pasting doesnt function as it should: some text just disappears and it gives a lot of empty lines/enters.....

      • Is it possible to add a gradient color to a newsletter im designing?

        From where i sit it looks like you can only choose a single color but not combine 2 colors?

      • Account disabled

        I have an issue I need help with. Whilst trialing ZOHO CRM I created the following: Account1 (-------------) using m__ame@m__rg___s__i__.___.__ and 2 personal emails Account2 (-------------) using a personal email and 2 users _al__1@______________._o_.__

      • Blocked Email

        We are a Zoho One subscriber and use Yahoo as our MX provider. A few times each year, for the past four years, CRM blocks one or more of my Zoho One users from receiving internal email from CRM. This includes "@mentions" in all modules, and emails from

      • message var is empty in bot mention handler

        Hi, I'm encountering a problem: in my bot's mention handler, I want to retrieve the text the user typed when mentioning the bot. Example: On the #tests-cyril channel, I send this message: “@Donna hello how are you ?” I expect the system variable "message"

      • Remembrance Day to Remember – Recalling Values

        The phrase “at the eleventh hour” refers to the 11th hour of the 11th day of the 11th month in 1918, when the hostilities of World War I came to an end—but it still holds meaning today. Remembrance Day (Veterans Day in the US) is observed on November

      • How to display two measures (sales and price) divided by categories on one line chart

        Hi everyone, I’m having trouble figuring out how to display two columns on a line chart with category breakdowns. What I need is a line chart where one line represents Sales and the other represents Price. However, the Price data is divided into around

      • Create custom rollup summary fields in Zoho CRM

        Hello everyone, In Zoho CRM, rollup summary fields have been essential tools for summarizing data across related records and enabling users to gain quick insights without having to jump across modules. Previously, only predefined summary functions were

      • Introducing the locking option for CRM records

        Last modified on 06/04/2023: Record locking option in CRM is now available for all Zoho CRM users in all DCs. Note that it was an early access feature available only upon request. Hello All, Hope you're doing well! We're thrilled to introduce our latest

      • Two new enhancements in Zoho CRM: Introducing new criteria for user fields and displaying group information in users page

        Announcement moderated on 14th June, 2023. Dear All, These enhancements are opened for all users in all DCs. ------------------------------------------------------------------------------------- Dear All, Hope you're well! We are here with two useful

      • Tip #49- Navigating the Remote Support Dashboard in Zoho Assist- 'Insider Insights'

        The Remote Support dashboard in Zoho Assist is designed to help technicians quickly access, manage, and monitor all their support sessions from a single, intuitive interface. Whether you’re starting a new session, managing ongoing connections, or reviewing

      • Next Page