Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Report on opportunities showing only the last note added.

        Hi I need to create a report that shows the most recent note added to each opportunity. This is so management can see what the latest update is according to the assigned salesperson. One workaround is to use the status field but this implies added manual work and mistakes as the salesperson would have to copy the existing status to a note before adding the latest status... otherwise the activity history would be lost. My current workaround is a report on Notes with Opportunities as the related module.

      • Please add Zelle as an online payment option

        Hello, I would like to request Zelle be added to the online payment service providers for Zoho Invoice. Considering how ubiquitous Zelle has become as a way to pay people via the major banking institutions, I feel like many freelancers would benefit from

      • Getting Project Template List using the REST API

        I am trying to confirm that I can use the REST API to create a project using a project template. The API documentation indicates this is possible by providing the Template ID, but it is not clear at all how to get a list of available Project Templates

      • How to display the CONTACT ID in the Contact page

        Hi, I've seen this conversation below and it is exactly the same question I'm raising now, but unfortunately the last message is not solved https://help.zoho.com/portal/community/topic/show-contact-id-while-editing-contact-form I need to show the ContactId and I don't know how to do this. The last message included in the conversaton shows the way but not it is not completed. "I am sorry by default we do not have the option to show the record ID for the contacts in the field in a record. When you

      • How to change margins in templates

        Hi, I'm new to Zoho Campaigns, and am trying to figure out how to adjust the margins within an email template.  The default margins make the text go all the way to the edges of the page.  Please advise.

      • Create Canvas list view templates from images powered by Zia

        Currently available for all paid editions of Zoho CRM in the US, EU, IN, JP & CN DCs. Designing a personalized CRM interface just got even easier. In today’s fast-evolving digital landscape, AI is transforming the way we work by automating complex tasks

      • Painfully Slow Zoho mail

        Since yesterday Zoho Mail seems to have starting functioning very slowly and having a few bugs. It's slow to open mails, slow to send, slow to change between email accounts. Sometimes clicking on a particular folder (eg Sent folder) stops working and

      • "Wrong password or login" Problem to configure Zoho on MAIL App on my Macbook

        Hi, I'm having problems to configure my e-mail on my MAIL App(Macbook pro). My e-mail is hari@trespontoum.net Actually was working perfectly, and still working on my Iphone. My MAIL App prompt me that my login or password is wrong. I tried to change 3

      • "User already exist in your org"

        Hello, I've just read a discussion about this issue, which didn't solve my problem. I'm trying to add the following emails: sales@kiss-my-boutique.co.uk returns@kiss-my-boutique.co.uk orders@kiss-my-boutique.co.uk I'm getting an error message each time I try and add them. None of them are primary or secondary emails and none of them have been created as users before. I know this as when I try and login and do 'forgot my password' all I get is an error message saying 'user invalid'. Please advise.

      • Operation Not Permitted

        Hi, I have problem in adding user after verifying the domain but it seems like error appeared which is "operation not permitted". For your information, I had delete other domain before did it.

      • Email forwarding setup fails

        I'm trying to set up email forwarding from my Zoho email to my gmail address. I followed the directions to set up email forwarding here: https://www.zoho.com/mail/help/email-forwarding.html. I did only steps 1-6. After doing this, rather than setting

      • Shopify store email issues- Not getting emails

        Hi We have migrated from Microsoft outlook to Zoho back in March, we have a shopify store, the domain is hosted on namesilo, not shopify, I have seen some people here complaining about not getting emails from customers who fill out the contact form on

      • Invoice Discount Account

        Is there a way to change the account used for Discounts applied to an invoice? The current Discount account (ZB native account) type is an "Income" type. I would like to change it to "Other Income", but that is not possible, I am assuming because it contains

      • Need Inactive accounts to be visible in Reports in Zoho Books

        I N=need Inactive accounts to be visible in Reports in Zoho Books to do recons of the accounts but when i see the same they are not visible in the Accountant - Account Transactions report

      • javax.mail.authenticationfailedexception 535 authentication failed

        Hi, I am facing 535 authentication failed error when trying to send email from zoho desktop as well as in webmail. Can you suggest to fix this issue,. Regards, Rekha

      • Phantom Opening Balance

        While experimenting w/ creating a composite item, Zoho added funds to Petty Cash and added Bank Account equal to previous day's balance. I'm unable to undo the adding of funds. Attaching screenshots of both bank account and petty cash transactions.

      • Importing customer comments and multiple shipto addresses

        I am evaluating Zoho Books and have most of my data imported with two exceptions: (1) Import customer comments - in my current system I have several comments of different types, and I need to bring these into Zoho. Preferably Books customer records, but

      • Detailed Account Reports - Add Running Balance

        When one clicks into an account from a report (say P/L or Balance Sheet), the default reports have the Debit, Credit, and then the Amount. The Amount column (last one) is duplicate information. It would be far better and fit more peoples' use cases if

      • How to Billed from two different GST Numbers

        How to Billed from two different GST Numbers. Suppose ABC & Co had GST registration in Delhi and Haryana and Zoho account is created with Delhi GST Registration number. Now i also want to issue invoice from Haryana GST Registration number. How can i proceed ?

      • Zoho IP blocked by SpamHaus

        ERROR CODE :550 - 5.7.0 Your server IP address is in the SpamHaus SBL-XBL database, bye

      • IMAP Communication Problems - IMAP Down Again

        We are experiencing the same problems over the last week or so that were supposedly 'solved' according to this thread. Zoho Imap Down Again This is affecting attempts to use IMAP on multiple ISPs and users so is definitely narrowed down to Zoho problems.

      • Can't login to Zoho mail

        I'm logged into Zoho but when I try to go in zoho mail I get: Invalid request! The input passed is invalid or the URL is invoked without valid parameters. Please check your input and try again. I just set up my mx records and stuff with namecheap a few

      • Hotmail is STILL blocking the zoho mail IP

        Greetings, as I already mentioned in my last message, my Zoho IP has been blocked by Hotmail for more than 15 days. Although someone said that the issue was "solved," it still isn't, and this amount of time seems neither normal for a "serious" company,

      • Keyboard shortcut M key

        I'm trying to customize my keyboard shortcuts and assign the "m" key. But it says the key is already assigned. I've looked through all my shortcuts and can't see any assigned "m". When I select an email and press the M key, nothing happens. What is the

      • Manage customer access across multi-brand help centers

        Hello everyone! Managing customer's access across multiple brands has been streamlined with the ability to invite, reinvite, approve, reject, and deactivate contacts individually for each help center brand . This enhancement gives administrators precise

      • White Label certificate

        Hi all, About 72 hours ago, we fulfilled all steps for white labeling (CNAME to point to mail.cs.zohohost.eu), but since then the configuration page is now stuck at "generating certificate" since 3 days. When I execute an NSLOOKUP mail.webfoundry.be it

      • Unusual activity detected from this IP. Please try again after some time

        When i try to create new addresses on my account i am getting this error, it has been 24 hours now and i am still getting this error can anyone help

      • How to add a new domain in my account?

        Probably zoho has the worst UI in the market. Cannot add a new domain to manage a new email address.

      • Report sorting not working

        I make extensive use of Summary reports and Pivot Tables. Most of these reports are structured such that I have several columns of "information" data that I want displayed, followed by one or more "Data" summarizations. For example, I have the following

      • New Enhancements in Wizards

        Dear All, Wizards help you break a long form into a series of smaller forms on different screens, making it less overwhelming. We are thrilled to introduce a couple of enhancements to wizards in Zoho CRM: Enhancements for conditional rules in Wizards

      • Find and Merge Duplicates to trigger webhook

        My sales team uses the Find and Merge Duplicates feature often to cleanup records in the CRM. We use webhooks to signal to our internal tools database when new Contacts are created, updated, or deleted, in order to keep our DB in sync with Zoho CRM. However,

      • Charting the 2025 Voyage: Zoho Invoice's Year in Review

        2025 has been a year of progress, productivity, and purposeful growth for Zoho Invoice. From expanding into new regions to refining everyday workflows, each update was designed to help businesses stay compliant, get paid faster, and work with confidence.

      • How to Hide System-DefinedTemplates in Service Report

        Is there any option available to hide system-defined templates? these templates are causing confusion for field users.

      • WhatsApp Report in Bigin CRM

        Reporting feature for Bigin CRM’s integrated WhatsApp that provides insights such as: Number of WhatsApp conversations closed Number of messages sent and received Number of conversations replied to Response and closure metrics for WhatsApp chats More

      • Issue with Azure DevOps Integration in Zoho Flow

        Hello, All workflows between Zoho Flow and Azure DevOps have stopped working for several days now. Upon further investigation, it seems that the connection to Azure DevOps is no longer directly supported. Indeed, Microsoft has deprecated the Azure DevOps

      • Narrative 16: Simplify with workflows

        Behind the scenes of a successful ticketing system: BTS Series Narrative 16: Simplify with workflows What is a workflow? A workflow organizes business tasks in a defined sequence that makes each step clear to all participants. This ensures work is completed

      • Zoho Invoice Customer Login Portal

        Are there any plans for a customer portal to Zoho Invoice, ala Freshbooks?  I would like customers that I invoice to be able to login to review invoices and invoice history.  I have not switched from Freshbooks for this very reason.

      • Random Leads are being created

        Hi, Every few days I am getting random leads that are being created with no form interactions at all. The email addresses are from obvious spam, such as 'Easymerchantsx'. When I look at the timeline, there is a Visit record and then a Lead Created record,

      • Print a price list or price book

        Hi Community. Am I right in concluding that Zoho has no functionality to print a price list from either Zoho CRM, Zoho Inventory or Zoho Books? I won't get stuck on the fact that Zoho doesn't sync price books between Zoho CRM and Books/Inventory (more

      • Bar Chart -- sort X-axis

        Hi! I created a bar chart ("Top 10 Products by Revenue") X-axis: Product Name Y-axis: Revenue Filter: Revenue - Top 10 Here's a picture: http://screencast.com/t/ZTJlZTdkNz The x-axis is sorted alphabetically by Product Name. How can I force it to sort numerically by Revenue?

      • Next Page