Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------


      • Sticky Posts

      • Kaizen #198: Using Client Script for Custom Validation in Blueprint

        Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

      • Kaizen #226: Using ZRC in Client Script

        Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

      • Kaizen #222 - Client Script Support for Notes Related List

        Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

      • Kaizen #217 - Actions APIs : Tasks

        Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

      • Kaizen #216 - Actions APIs : Email Notifications

        Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

        • Recent Topics

        • What is the difference between Zoho Campaigns and marketing hub?

          Zoho Campaigns is a permission-based email marketing tool for sending marketing emails and mass emails. Marketing Automation is a multichannel marketing automation tool that caters to organizations looking for tools to engage their users across multiple

        • Webhook not firing.

          I created a webhook using the Web UI, it looks very nice and the testing worked without an issue, but when i save/ update a ticket, the webhook is not firing. Here are the details of the web-hook i get from using the API "modifiedTime": "2019-10-22T09:23:37.380Z",

        • Duplicate customers being created in Desk

          Hi I've trying to work out why I've getting duplicated customers being created in my desk. I have an external booking system that generates an email when I get a customer booking a job. A booking email gets sent to Desk where I manage the booking and

        • Creating a custom CSV file using deluge script/

          I have an application I have developed and the client wants us to place an export file in csv onto an ftp server daily. Now I don't see au options in creator to change the separator to anything else. The client wants the separator to be the pipe symbol "|"  I think i would be able to create schedule with some code to create the appropriate data in a string using deluge script but I haven't seen any functionality that would allow me to deposit that data as a file anywhere or attach it to an email

        • Can we add custom fields to portal community profiles?

          How do we add custom fields to our profile pages in our portal community? If we have the ability to add custom fields, will we be able to access those fields via API? We want to use our Desk community in our help portal as our primary community and would

        • E-Invoicing in Belgium with Zoho Billing

          Starting January 1, 2026, Belgium is introducing mandatory electronic invoices (e-invoicing) for all B2B transactions between VAT-registered businesses. Invoices and credits notes must be exchanged in a prescribed digital format. How E-Invoicing works

        • delayed: host mx2.zoho.com (136.143.183.44): Network error: Unexpectedly disconnected (STARTTLS)

          Hi Team, I'm not receiving any email. Seems to be an issue. Sender sent me the error message they received: "delayed: host mx2.zoho.com (136.143.183.44): Network error: Unexpectedly disconnected (STARTTLS)" Please advise.

        • Additional Address - Company Name

          It would be very helpful to have a "Company Name" field in the Additional Address shipping section.  In a situation where you drop-ship orders to a different address or different company entirely, it may be necessary to list the company name of the receiver. I understand that the Attention field can be used for that purpose, but that's not really the intended purpose of that field.

        • Incorrect Email Notifications for Product Reviews

          Dear Zoho Commerce Support Team, I am writing to report a technical issue that occurs frequently on our platform. Problem Description: We regularly receive email notifications informing us of new product reviews awaiting approval. However, when we access

        • How to Rank Tables by Row Count in Descending Order

          I am trying to understand the consume of lines that grow up so fast in the last week. Is there any way to create a pivot table or query to get TABLE NAME LINES in descending order?

        • Can we do Image swatches for color variants?

          We want to do something like the attached screenshot on our new zoho store. We need image swatches instead of normal text selection. We want to user to select an image as color option. Is this doable? I don't see any option on zoho backend. Please h

        • annualy customer report

          we need a report per customer that looks like this invoic number cleaning laundry repair management 01 january xxx euro xx euro xx euro xxx euro 02 february xxx euro xxx euro x euro xxxx euro and so on the years 12 months is that possible to make and

        • Export your notes from Notebook!

          Dear users, The long awaited feature is now live. Yes, you can now export your notes from Notebook app in bulk. But the feature has just started with web app alone for now. You can try the export feature as mentioned below: Go to our web app, https://notebook.zoho.com Go to 'Settings' > 'Export' Now, select the format: You can select either ZNote or HTML Once done, you can use the same to import or can have this a local backup of your notes. Note: Export for other platforms are in development and

        • Canva Integration

          Hello! As many marketing departments are streamlining their teams, many have begun utilizing Canva for all design mockups and approvals prior to its integration into Marketing automation software. While Zoho Social has this integration already accomplished,

        • Prepopulating Fields

          Hello, I have a form (Assets) with 2 lookup fields: Client (from Clients) Site (from Client Sites) I modified the code (highlighted in red below), so the Site dropdown shows the list of sites related to the Client. must have Client_Site ( type = picklist

        • REPORT THAT SHOWS SOH FOR ITEMS THAT HAVE ZERO SALES

          When we started Zoho we imported a lot of inventory lines directly off our suppliers price list and have never ordered / sold. I want to clean up our data base and remove all these lines. What reeport will show me what lines have zero sales as well as

        • Automation series : Close all tasks once the project is completed

          When a project is marked as Completed, it might still have open tasks such as recurring tasks that were not marked as complete, tasks that are no longer relevant, or tasks that no longer need attention after closure. To ensure the project reflects its

        • Converting Sales Order to Purchase Order

          Hi All, Firstly, this code works to convert a sales order(SO) to a purchase order (PO) via a button, however I am running into an issue when I convert the SO where the values from the line items are not pulled across from the SO to the PO. The ones in

        • Horrible Connectivity!

          I have used Meetings several times, but most of the time the connection is horrible. The video freezes, the audio freezes, and we end up cancelling the meeting. I am on a high speed internet connection, and Zoom works fine, so I know it's not a problem

        • Is it really true that I can't set the default 'deposit to' account in 2025?

          I've been using Books for 7 years and the default account has never been a problem. I usually manually reconcile invoices and have never had a thought about which account. It has always been my account. However, I recently noticed that for the past 4

        • Zoho DataPrep switching Date Format

          When using a pipeline that is importing Zoho Analytics data into Zoho DataPrep, the month and day of date fields are switched for some columns. For example, a Zoho Analytics record of "Nov. 8, 2025" will appear in Zoho DataPrep as "2025/08/11" in "yyyy/MM/dd"

        • Using Another Field Value for Workflow Field Update

          I'm trying to setup a Workflow with a "Field Update" action on the Lead module, but I would like the new value to actually be taken from a DIFFERENT Field's on the Lead record (vs just defining some static value..) Is this possible? Could I simply use

        • Tax information

          Hello, I need help/guidance on how to add my organization's Tax/VAT information. Thank you Pavly

        • Build smarter Guided Conversations with Fork Blocks

          When your customers arrive on your support channel, they're not there to explore. They are usually confused and stuck while trying to fix something important. We understand how stressful that moment can feel and we want your bot to make things easier,

        • Custom item field won't allow decimal.

          Hello, I have a custom item field that needs to be able to have a value with a decimal place such as 6.7 or 6.18. I have tried custom formatting the input format but can not get the correct syntax to allow this. Is this possible in Zoho?

        • Free webinar: Zoho Sign unwrapped – 2025 in review

          Hey there! 2025 is coming to an end, and this year has been all about AI. Join our exclusive year-end webinar, where we'll walk you through the features that went live in 2025, provide answers to your questions, and give you a sneak peek on what to expect

        • HubSpot CRM to Zoho Creator Integration

          I'm trying to create an integration between HubSpot CRM and Zoho Creator with HubSpot being the push source (if a contact is created or updated in HubSpot, it pushes the information to Zoho Creator). I have two questions: 1- Is it best to use Zoho Flow

        • Gain control over record sharing with portal users through our new enhancement: criteria-based data exposure

          Dear Customers, We hope you're well! Portals is a self-service avenue through which your clients can access and manage their direct and related data in Zoho CRM. This empowers them to be more independent and enables them to experience a sense of transparency

        • Best-practice setup in Zoho One for managing combined candidate pools and exporting anonymised CVs

          We are new users of the Zoho One bundle and operate a consulting and engineering company. Our workforce model includes a mix of permanent employees, active job applicants, and freelance/independent consultants. All three groups need to be searchable,

        • Meet Canvas' Grid component: Your easiest way to build responsive record templates

          Visual design can be exciting—until you're knee-deep in the details. Whether it's aligning text boxes to prevent overlaps, fixing negative space, or simply making sure the right data stands out, just ironing out inconsistencies takes a lot of moving parts.

        • hard-bounced email list

          Hi, Below pops up when I try to send an Email to some of my customer. Please guide me how to take it out from hard-bounced list? I am not sure how they were marked in hard-bounced list

        • Register the 'Contact Role' addition and change as a Potential edition so it can trigger Workflows

          We are trying to use "Contact Roles" in Potentials. Contact Roles are special and different than the other Related lists, so, it may have a special behavior. Something to keep in mind is that you will never have 100 Contact Roles as you can have 100 Tasks, Calls, or any other Related list. In our case we will have 2 in average and up to 4 or 5 maximum. The problem is that we need to bring information from 3 key Contact Roles to the Potential and adding a Contact to the Contacts Roles area never trigger

        • Can I change the format of the buttons in the email templates?

          Hi all! We have been working hard trying to brand our email templates, and have some way to go yet. One of the things we can't seem to edit is the green ${Cases.CUSTOMER_PORTAL_BUTTON} button and the font of the View Ticket text. Is there any way of doing

        • Best practice to structure reporting to include events covering multiple months / quarters.

          Hi, I'm new to Zoho, have some experience of more "enterprise" tools, looking for some input from the community. I'm looking to create a report that includes events that cover a long period, each event has a start / end date and I'm struggling undertanding

        • Marketing Tip #11: Turn features into benefits that sell

          We all love talking about our products, but here’s a secret: customers don’t just buy features, they buy benefits. Instead of just saying "Made from 100% organic cotton," try "Soft, breathable comfort that lasts all day." Benefits tell shoppers how your

        • Synchronise item image between Zoho Commerce and Zoho Books/Inventory/CRM

          Here is a blindingly simple idea to tie several Zoho products together. Zoho - please include a method to synchronise the item image (or images) from one Zoho application to another. For example, if you upload an item image in Zoho Inventory, a user should

        • How do I migrate from zoho mail to Office 365?

          The manual migration of Zoho Mail to Microsoft 365 typically requires using IMAP to move emails and configuring Microsoft Outlook to sync Zoho Mail. This approach can be error-prone, especially if there are multiple accounts or large email archives. Moreover,

        • New Update: Convert External Users in Bulk in Zoho Directory

          Greetings to all Zoho Directory users out there! We’re excited to introduce a new update that makes user management in Zoho Directory even more efficient; you can now convert external users in bulk! Earlier, admins could convert only one external user

        • New Update: Convert External Users in Bulk in Zoho One

          Greetings to all Zoho One users out there! We’re excited to introduce a new update that makes user management in Zoho One even more efficient; you can now convert external users in bulk! Earlier, admins could convert only one external user at a time.

        • How to edit form layout for extension

          I am working on extension development. I have created all the fields. I want to rearrange the layout in Sigma platform. But there is no layout module in Sigma. How can I achieve this for extensions other than Zet CLI and putting the fields into widget

        • Next Page