Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Calendar not working

        Are we the only ones. On any browser we cannot click on any of our calendar appointments and get them to open. They just make the browser loop. WE have reached out and have been told they are working on it. The office staff are really stuck. The point

      • WhatsApp Message Template Quick Reply Buttons

        Hello, I created my first Message Template to overcome the 24-hr messaging window and it was approved by WhatsApp/Meta. When I go to Meta, I also see that template brought from Zoho Desk, and used it via Zoho Desk IM. However, I notice that when in Meta's

      • Account in Quick View Filter

        I have a report that I often run against a specific Account. Every time, I have to go into the edit menu and change the Advanced Filter. I would prefer to use the Quick View Filter, but it does not allow me to use the one and only field that makes any

      • Set Display for Numbers/Currency/etc with Client Script/Customization in Canvas List Page

        Is it possible to set a display mask for a number/currency field using Client Script or customization? I have custom fields that I would like to keep the decimal places for calculation purposes, but do not need them displayed to the user. So 101.3568

      • 136.143.188.51 blocked by spamcop

        Zoho mailserver seems to be blocked by spamcop. Cant send mail to my customer. Not sure what to do.

      • 🚀 WorkDrive 5.0: Evolving from a file sharing app to an intelligent content management platform: Phase 2

        Hello everyone, WorkDrive's primary focus has always been to provide an intelligent and secure content management platform, simplify collaboration, and be the central repository of files for all Zoho apps. In our previous announcement, we unveiled the

      • i keep see there is a connetion issue connecting 3rd party api on zoho when using zia

        hi there , i have set up open ai api to zoho zia (copied and pasted to zoho zia) but I keep getting notificaiton "there is a connetion issue connecting 3rd party api on zoho" when using zia on top when click zia and try to type in word there

      • Deleting Fields in CRM Deletes Views in Analytics

        Hey friends! I'm having some issues when we modify some fields within ZohoCRM. There are times where we need to sunset a field and eventually completely remove it. In these instances, it seems like a lot of views are removed in Analytics. This ranges

      • Zoho Desk Android and iOS app update: Access multilingual templates on the IM module

        Hello everyone! We have introduced multilingual templates on the IM module of the Zoho Desk mobile app allowing you to send IM template messages in translated versions. iOS: Android: Please update the app to the latest version directly from the App Store/

      • Does Zoho offer a full WhatsApp Marketing Platform (like WATI / GallaBox)?

        lm exploring WhatsApp marketing and I’d like to understand what options are available within the Zoho ecosystem. Currently, I see: Zoho SalesIQ has a broadcast option, but it seems very limited — for example, it doesn’t support uploading contacts via

      • Custom buttons

        Getting Error "You can only create 10 custom buttons per module" ! I already have more than 10 for 1 module before trying to create another. Is this a new restriction brought about by the all knowing what's best for businesses Zoho guru ? It would be

      • The Social Wall: October 2025

        Hello everyone, As we head toward the end of the year, we’re bringing you a few updates to help give your social media efforts a strong finish. This month, we’re rolling out new enhancements across both the web and mobile app. Post Preview Have you ever

      • Client Script | Update - Support for List Page (Canvas) and Notes

        Hello everyone! We are glad to announce two exciting updates in Client Script. Client Script support for List Page (Canvas) Client Script support for Notes These open the door to more dynamic and interactive customizations! 1. Client Script support for

      • Pincode Based Pricing

        We want to give extra discount to customers within city limits because our shipping and packing costs are less. My proposal is assign list of pincodes to price list and assign the price list to customers whenever they sign up with pincodes. Also show

      • feature question: lookup field in Projects task layout

        Hello, Below we have interface to configure a custom task layout, however, we need a lookup field to let pickup data from other apps, such as Zoho CRM Products module, is that possible or already in your roadmap? or a workaround to achieve it? Thanks in advance.

      • What's New in Zoho Analytics - October 2025

        Hello Users! We're are back with a fresh set of updates and enhancements to make data analysis faster and more insightful. Take a quick look at what’s new and see how these updates can power up your reports and dashboards. Explore What's New! Extreme

      • Zoho Analytics & SQL Server - Live connect

        Below are the steps I did Created a fresh database and table in my own virtual server Created a new data source connection with live connect I was able to select the tables and created the data source successfully I am getting the error when I try to

      • Explore the Redesigned Quotes Module in Zoho Billing

        Dear users, We’re excited to introduce a refreshed look for the Quotes details page in Zoho Billing! This update brings you a more efficient user interface experience without changing your existing workflow. We've enhanced layouts with organized tabs,

      • Latest updates in Zoho Meeting | Create departments, Share PDF files

        Hello all,   You can now create departments to group team members within your organization. This will make it easier for you to organize department-level meetings and invite members. In webinars, use the Share material feature to share PDF files directly

      • Unify All Zoho Video Meeting Experiences into One Standardized Platform

        Hi Zoho Team, We would like to share an important user experience concern regarding the current state of video meeting functionality across the Zoho ecosystem. The Problem Within Zoho, there are multiple ways to initiate or schedule a video meeting: Zoho

      • Import VAT Kenya

        Hi. I am quite lost on how to account for VAT when importing. We are an import and distribution company and when importing we get a customs entry with VAT which we pay and claim for this VAT as we do our monthly sales. My question is how to record this

      • [Free Webinar] Product Updates: Quick Catch-Up Session - Part I - Creator Tech Connect

        Hello Everyone! We welcome you all to the upcoming free webinar on the Creator Tech Connect Series. The Creator Tech Connect series is a free monthly webinar featuring deep-dive technical sessions designed for developers, administrators, and app builders.

      • Project expenses in Zoho Books

        Just the way timesheet is used to calculate labour cost for a particular project, how do i record other expenses against a project such as materials and consumables used for the project? So that under report, I can view the total amount expended on each

      • Blueprint status change

        I want to change the Blueprint status of a workflow. The status is completed automatically when I added the record through API call. But I want to make it active, I don't see any options for this. Even if I draw transition from any stage , it is not visible

      • What is going on with GMail extension.

        Suddenly, it is popping up and saying "Please wait while we work our magic... 10%" and it sits there. Yesterday I waited for an entire hour call and it never got further. Today, it's not resolved. Is there a fix I can do? I've tried all the typical

      • Marketing Tip #6: Cross-sell with related products

        Don’t let customers stop at just one product. Suggesting related or complementary items improves the shopping experience and increases order value. Try this today: Use the Item Recommendations feature in Zoho Commerce and add two to three suggested items

      • i cannot use <b></b> to bold the message in Creator C6!!!!???What?

        Dear experts and friends, Now Creator 6 blocking us from using <b></b> Anyone facing this issue? I faced it on Creator C6 Previously, it used to work. Now failed to work. Faint~ Seek guidance from everyone on how to bold the message. The super simple

      • Zoho Writer - Option to Export as .zdoc format

        I've noticed that it's not possible to export a Zoho Writer Document in the .zdoc format. Isn't zdoc, Zoho Writer's own format? My use case is that I sometimes need to create quite complex documents with floating elements, which sometimes need to become

      • [Announcement] Enhanced Merge APIs of Zoho Writer for automating document generation using Zoho Apps data

        Hi users, With the record_id parameter in Zoho Writer's Merge APIs, fields related to the given record id is automatically fetched from the Zoho app. You don't have to spend time manually mapping the records and fields in the Deluge code. Here is a sample

      • Pin multiple columns and adjust column widths in CRM subforms

        Hello all, Subforms act as secondary forms or tables in which you can associate multiple line items to a primary record and thereby ensure more structured and comprehensive data organization. We've made some recent enhancements to subforms. Here's what's

      • [Webinar] Automate sales and presales workflows with Writer

        Sales involves sharing a wide range of documents with customers across the presales, sales, and post-sales stages: NDAs, quotes, invoices, sales orders, and delivery paperwork. Generating and managing these documents manually slows down the overall sales

      • Automate insurance document workflows with Zoho Writer

        Insurance companies have to deal with creating and managing complex documents and forms, such as policy applications, explanation of benefits documents, brochures, renewals, and claim forms. Handling all of this manually is hugely time and effort intensive,

      • Latest updates in Zoho Meeting | New End of session notification to remind everyone about the session end time

        Hello everyone, We’re excited to share a new feature for Zoho Meeting ; End of session notification. With this new setting, you can choose to remind all participants or only the host about the scheduled end time of a meeting. You can also choose when

      • [Webinar] Zoho Writer for SCM and logistics

        Creating, sharing, and distributing complex documents are constant challenges in the SCM and logistics industry. That's why our next webinar is focused on how to simplify these document workflows. Join us on Jul 10, 2025 for a Zoho Writer webinar designed

      • [Webinar] Zoho Writer for the manufacturing industry

        Creating, sharing, and distributing complex documents are constant challenges in the manufacturing services industry. That's why our next webinar is focused on how to simplify these document workflows. Join us on May 8, 2025, for a Zoho Writer webinar

      • Add Comprehensive Accessibility Features to Zoho Writer

        Hello Zoho Writer Team, We hope you are doing well. We would like to submit a feature request to enhance Zoho Writer with a full set of accessibility tools, similar to the accessibility options already available in the Zoho Desk agent interface. 🚧 Current

      • [Webinar] Simplifying document workflows in financial services with Zoho Writer

        Creating, sharing, and distributing complex documents are constant challenges in the financial services industry. That's why our next webinar is focused on how to simplify these document workflows. Join us on April 10, 2025, for a Zoho Writer webinar

      • [Zoho Writer Webinar] Learn how to simplify your HR operations: Part 2

        Hi Zoho Writer users, We're excited to announce the Zoho Writer webinar for July 2024: Learn how Zoho Writer can simplify your HR operations: Part 2. This webinar will focus on how Zoho Writer can help HR teams streamline and automate their entire hiring

      • [Zoho Writer Webinar] Learn how to simplify your day-to-day HR operations

        Hi Zoho Writer users, We're excited to announce the Zoho Writer webinar for June 2024: Learn how Zoho Writer can simplify your day-to-day HR operations. This webinar will focus on how to automate your entire hiring process and generate various types of

      • Branding of native system emails

        Make system emails adjustable in terms of branding. We want our system to be completely white label, because it is not a CRM anymore, it's way more than that. We are following the strategy of "CRM for everyone" to use the CRM in all departments, not only

      • Next Page