Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

    • Sticky Posts

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #226: Using ZRC in Client Script

      Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

    • Kaizen #222 - Client Script Support for Notes Related List

      Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

    • Kaizen #217 - Actions APIs : Tasks

      Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

    • Kaizen #216 - Actions APIs : Email Notifications

      Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

      • Recent Topics

      • Non-depreciating fixed asset

        Hi! There are non-depreciable fixed assets (e.g. land). It would be very useful to be able to create a new type of fixed asset (within the fixed assets module) with a ‘No depreciation’ depreciation method. There is always the option of recording land

      • Fixed asset management

        I want to know if there is any individual module for fixed assets management

      • One time sale item in billing automatically detects as service

        if i have some items which i don't want to add in my "item" list because its sold only for one time. but when i type item name in invoice, it (system) automatically takes it as a service and despite of HSN , it shows SAC code to be entered. if its selectable i.e. either item or service , it would be very helpful and a must have feature.

      • Project template after project creation

        How can I apply a project template AFTER the project has been created?

      • convert the project to templet

        i have some deployment ME product for different customer , i need to create a fixed template for use it rather then keeping creating this template every time

      • Related Module in Sharing Rules

        Zoho CRM team recently added the feature to filter records by Related Records It will be really beneficial if we can have this feature for Sharing Rules as well

      • Assignment Thresholds Resetting After Lead Conversion

        Hello everyone, We're facing an issue with Zoho CRM's lead assignment thresholds that makes them unsuitable for our workflow. I'm hoping to find a potential workaround or solution from the community. Here’s our current process: A new lead is created automatically

      • Territory view for custom modules?

        I have recently activated territories however I can't seem to find how to use territories for custom modules? These modules have territories:  Contacts /  Accounts / Opportunities These modules don't have territories:   Buildings (custom module) and 

      • Approval Process Comments

        Is it possible to view the comments entered during the approval or rejection of a record in the approval process? If not, is there a way to require a field to be completed upon approval or rejection?

      • Login for test case

        Had a few questions regarding authentication test cases and couldn't find an answer in the the docs. 1. If an app like Zoho Creator requires authentication before providing access, do I need to create a login function and add it to each test case? 2.

      • Customer members area

        Does FSM support a customer members area? If not what do you propose we use if we want the data used in FSM for customers to give them an area / login to see past orders, create new orders and general announcements.

      • Notes created in mobile can no longer be accessed in desktop

        Working with a 2013 Mac running OS 10.14.6; Desktop Notebook version 4.5.3. Using Motorola Moto G Power 5G - 2024; Android app version 6.7 I have been using Notebook for some years. Starting several weeks ago, the notes newly created ion the phone can

      • Function #49: Manage varying installment payments using Zoho Books

        . Hello everyone, and welcome back to our series! Last week, we discussed automating the collection of fixed installment payments in Zoho Books. But what if your payment structure involves charging varying percentages of the invoice total as installments?

      • Is it possible for contacts to "Re-enter" a workflow in Zoho Campaign?

        We are currently working on a way to automatically add users to from one list to other lists based on specific criteria, but can't seem to find a native way of doing this so we are trying to use Workflows to do this. So, for example, if a user's status is set to "Active," then they should be added to the list "Active Users." If the same user's status is then set to "Paused," they should be added to the list "Paused Users" and removed from the list "Active Users." This works fine for the first go

      • Bulk upload images and specifications to products

        Hi, Many users have asked this over the years and I am also asking the same. Is there any way in which we can bulk upload product (variant) images and product specifications. The current way to upload/select image for every variant is too cumbersome.

      • Out of Office for Just One of My Alias Email

        Can I set up the Out of Office Reply for Just One of my Alias Email Addresses?

      • Can I map multiple Surveys into the CRM using the same fields?

        Hello, We are a healthcare practice that offers two distinct services (Nutrition and Primary Care). We use Zoho Survey for our lead generation form (Get Started Survey), which allows people to express interest in one of the two services and even allows

      • Dealing with API responses where integers have more than 16 digits

        Hi there How do I deal with an api response contaning an int or float with more than 16 digits (before any decimal places for a float). I constantly receive the response "Unable to cast the 'BigInteger' value into a 'BIGINT' value because the input is

      • Add Zoho Mail for users who do not need Zoho One

        We have licenses for ZOho One for teams that need to use the suite of products that Zoho One offers.  We have 8 more people who only need email access and we would like to add just a Zoho Mail.  They do not need the Zoho One license.  We are currently

      • ZML vs HTML Snippet - which is better?

        Are there certain use cases where one is better than the other?

      • Load PO_Date field (Purchase Order) with current date in Deluge

        Hi, I'm not a full time developer, just helping to customize our CRM, in the small company I work for. There must be something wrong with me, because I can't do something so simple as complete a field with the current date in a function using Deluge.

      • Zoho CRM in Microsoft Power Automate Custom Connector

        Hi everyone, I’m building a Power Automate flow that integrates Microsoft Bookings with Zoho CRM. The goal is to automatically create a meeting (event) in Zoho CRM whenever a new appointment is booked via Microsoft Bookings. To achieve this, I created

      • Spell check sucks

        Come on guys, it's 2024 and your spell check is completely retarded. You gotta fix it.

      • Zoho One Login Issue - Unable to receive OTP

        Hi Support Team, I am experiencing a unique login issue with Zoho One. I am attempting to log in from India using Zoho Login credentials provided by a USA-based client. Their Zoho account is hosted on a US data center. After entering the username and

      • In the Zoho Creator Customer Payment form i Have customer field on select of the field Data want to fetch from the invoice from based on the customer name In the Customer Payment form i Have subf

        In the Zoho Creator Customer Payment form i Have customer field on select of the field Data want to fetch from the invoice from based on the customer name In the Customer Payment form i Have subform update Invoice , there i have date field,Invoice number

      • Problem of Import Client Users From CRM and or Expense

        I am premium plan user on Projects. I have about 500 customers on Expense and CRM that integrated with each other. According to at below link, I am trying to import clients from CRM, system not allowed to select any customer. If I import from Expense,

      • equest to Disassociate Bigin from Zoho One and Migrate to Standalone (Upgrade to Bigin Premier – 3 Seats, Annual)

        Dear Zoho One Support Team, I’m writing to request your assistance to disassociate (remove) the Bigin application from our Zoho One organization while preserving all existing Bigin data. After the disconnection is successfully completed, we intend to

      • Unable to enable tax checkboxes

        Hi Zoho Commerce Support, I'm writing to report an issue I'm having with the tax settings in my Zoho Commerce store. I've created several tax rates under Settings > Taxes, but all of them appear with the checkbox disabled. When I try to enable a checkbox,

      • Zoho Commerce - Enable Company Name and Tax Number collection for B2B orders in Global Edition

        Please enable Company Name and Tax Details option on checkout settings in Zoho Commerce Global Edition. It is still important to collect Company Name and Tax Number for B2B sales in many countries. My business is based in Ireland (in the EU) and I have

      • ZohoSign and ZohoBooks Integration/Workflow

        Hello All, We utilize ZohoSign for signatures on tax eFiles. We utilize Dynamic KBA. Additionally, we use ZohoBooks for invoicing for these services. Is there a way to accomplish the following: Send a copy of the Tax Return, Invoice and eFiles in one

      • Manage monthly tasks with projectsf

        Hi All I run a finance and operations team where we need both teams to complete monthly tasks to ensure we hit our deadlines. Can Zoho projects be used for this. There many finance focused tools but we have Zoho one so want to explore Thanks Will

      • Zoho Suite is very slow

        Since today Zoho is incredibly slow over all applications! What's going on?

      • Combine and hide invoice lines

        In quickbooks we are able to create a invoice line that combines and hides invoices lines below. eg. Brochure design         $1000 (total of lines below, the client can see this line) Graphic Design           $600 (hidden but entered to reporting and

      • Zoho Devops

        We have a Zoho one account which we have integrated with an SAS educational product, sold on a subscription model, using webhooks and API calls. We make some use of custom fields and cross module lookups and relationships. We utilize CRM, Books and billing

      • Fuel up your sales with the Zoho SalesIQ + Bigin integration

        Hi everyone! We’re happy to bring you the all-new Zoho SalesIQ + Bigin integration. With this, every prospect from your website instantly becomes a contact in Bigin, complete with transcripts and follow-up tasks, so you never lose a lead again. Let's

      • Introducing AI-powered Assessments & Zoho's native LLM, Zia

        We’ve shipped a cleaner, faster way to create assessments in Zoho Recruit. 🚀 Instead of manually building question banks or copying old templates, you can now generate ready-to-use assessments in just a few clicks, all tailored to the role you’re hiring

      • Ability to Reset Visitor Fields During an Active Chat Flow

        Hello Zoho SalesIQ Team, We hope you are doing well. We would like to propose a feature enhancement to Zoho SalesIQ regarding the management of visitor fields within Zobot flows. Use Case: Our bot asks the visitor to provide information about a 3rd person

      • External ID in Zoho CRM

        Hello everyone! We know that Zoho CRM allows you to integrate third-party apps and manipulate data through APIs. While you integrate a third-party application, you may want to store the third-party reference IDs in Zoho CRM's records. To meet this need

      • Some emails are not being delivered

        I have this problem where some of my mail just seems to disappear. When I send it, it appears as sent with no mention of any problem, but my recipient never gets it, not even in the Spam folder. Same for receiving, I have a secondary e-mail address, and

      • New in Zoho Chat : Search for contacts, files, links & conversations with the all new powerful 'Smart Search' bar.

        With the newly revamped 'Smart Search' bar in Zoho Chat, we have made your search for contacts, chats, files and links super quick and easy using Search Quantifiers.   Search for a contact or specific conversations using quantifiers, such as, from: @user_name - to find chats or channel conversations received from a specific user. to: @user_name - to find chats or channel conversations sent to a specific user. in: #channel_name - to find a particular instance in a channel. in: #chat_name - to find

      • Next Page