Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

      Zoho Campaigns Resources


        • Desk Community Learning Series


        • Digest


        • Functions


        • Meetups


        • Kbase


        • Resources


        • Glossary


        • Desk Marketplace


        • MVP Corner


        • Word of the Day


        • Ask the Experts


          • Sticky Posts

          • Kaizen #198: Using Client Script for Custom Validation in Blueprint

            Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

          • Kaizen #226: Using ZRC in Client Script

            Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

          • Kaizen #222 - Client Script Support for Notes Related List

            Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

          • Kaizen #217 - Actions APIs : Tasks

            Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

          • Kaizen #216 - Actions APIs : Email Notifications

            Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

          Zoho CRM Plus Resources

            Zoho Books Resources


              Zoho Subscriptions Resources

                Zoho Projects Resources


                  Zoho Sprints Resources


                    Zoho Orchestly Resources


                      Zoho Creator Resources


                        Zoho WorkDrive Resources



                          Zoho CRM Resources

                          • CRM Community Learning Series

                            CRM Community Learning Series


                          • Tips

                            Tips

                          • Functions

                            Functions

                          • Meetups

                            Meetups

                          • Kbase

                            Kbase

                          • Resources

                            Resources

                          • Digest

                            Digest

                          • CRM Marketplace

                            CRM Marketplace

                          • MVP Corner

                            MVP Corner




                            Zoho Writer Writer

                            Get Started. Write Away!

                            Writer is a powerful online word processor, designed for collaborative work.

                              Zoho CRM コンテンツ




                                ご検討中の方

                                  • Recent Topics

                                  • Creator Add Records through API - Workflows not triggered ?

                                    Hi Everyone, I am trying to add records to my Creator application through a third party app that I am developing. Currently, I am testing this through Postman. The records I need to add have a lot of workflows to run to populate dropdowns, fields, use

                                  • Important Update: Changes to Google Translate Support in Zoho SalesIQ

                                    We’re updating our default chat translation options across all Zoho SalesIQ data centres to offer a more secure, in-house solution for your translation needs. What’s changing? We will be offering Zoho Translate (our in-house tool) as the default translation

                                  • Zoho CRM Community Digest - July 2025 | Part 1

                                    Hey everyone, The start of July 2025 marked a special milestone: 200 posts in our Kaizen series! For those new here, Kaizen is a go-to series for Zoho CRM developers, where we regularly share best practices, tips, and expert insights to help you build

                                  • What’s New in Zoho Expense (April – July 2025)

                                    Hello users, We're happy to bring you the latest updates and enhancements we've made to Zoho Expense over the past three months, which include introducing the Trip Expense Summary report in Analytics, extending Chatbot support to more editions, rolling

                                  • Weekly Tips: Insert frequently used phrases in a jiffy using Hotkeys

                                    You often find yourself using similar phrases in an email —like confirming appointments or providing standard information. Constantly switching between the mouse and keyboard interrupts your flow and slows you down.Instead of typing the same phrases over

                                  • Undo Command in Notebook

                                    It would be handy to have an Undo in the row of icons at the bottom of the notes.

                                  • Narrative 9: GC—Meaningful conversations that benefit your customers

                                    Behind the scenes of a successful ticketing system - BTS Series Narrative 9: GC—Meaningful conversations that benefit your customers Customers often seek immediate solutions, which is why self-service options are essential. Guided Conversations provides

                                  • FSM App Oddity

                                    We recently rolled out FSM to our technicians, and only one technician is having an issue. He's using an iPhone with iOS 18.6 installed. When he goes to service appointments using the calendar icon at the bottom of the app, he gets a list view only. Typically,

                                  • Cliq Not Working !

                                    Zoho Cliq has been experiencing connectivity issues since this morning. The app is unable to establish a connection with the server.

                                  • Injecting CSS into ZML pages

                                    Is there a way to inject CSS into ZML pages? Use case: 1. Dashboard displays 'Recent Activities' card displaying a list of newly added records 2. Each item in list links to the record onClick 3. When a user points the cursor over an item in the list,

                                  • Power of Automation :: Automated Weekly Notifications for Unapproved Timesheets

                                    Hello Everyone, A custom function is a software code that can be used to automate a process and this allows you to automate a notification, call a webhook, or perform logic immediately after a workflow rule is triggered. This feature helps to automate

                                  • Extracting Data from Sitelink APIs

                                    Hi All, I'm working to pull data on Sitelink using API calls but i can't seem to extract it. If you click expand, you will see it has a lot of data in it. I tried extracting from the map but it's not giving me anything. Is there a workaround on this or

                                  • Export a list of fields for all modules in a spreadsheet with specific field data

                                    Many of my clients are using spreadsheets to create lists of fields for all modules when starting a new implementation or when updating an existing setup. This is a useful process but also a very time consuming one. It would be good a list of fields could

                                  • [Important announcement] Zoho Writer will mandate DKIM configuration for automation users

                                    Hi all, Effective Dec. 31, 2024, configuring DKIM for From addresses will be mandatory to send emails via Zoho Writer. DKIM configuration allows recipient email servers to identify your emails as valid and not spam. Emails sent from domains without DKIM

                                  • 【Zoho Backstage】2025年7月のアップデート紹介

                                    本投稿は、本社のZoho Desk コミュニティに投稿された以下の記事を翻訳し、一部抜粋したものです。 What's New - July 2025 | Zoho Backstage ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 本投稿では、Zoho Backstage の直近のアップデートを3点抜粋して、ご紹介します。 目次 1.参加可否の確認：RSVP 2.証明書作成ツール：Certificate Builder 3.登録の承認 ：Registration Approval 1.参加可否の確認：RSVP

                                  • Can we have Backorder Management ?

                                    Can we have Backorder Management ?

                                  • Display a field of an Account on the Ticket view

                                    Hi there, We would like to display of the Account of the user submitting a ticket on the ticket view. See for example: How can this be achieved? It doesn't really matter where it will be placed as long as it is shown there. Thanks, Snir

                                  • Insurance Agencies

                                    I am reaching out to see if anyone has any experience in the Insurance Agency world with using Zoho exclusively for the CRM and commissions side of things. Lots of strong features like drip campaigns, meeting, calendars, emails, can all be found in here.

                                  • Problem with Egyptian internet ISP

                                    Dears,  We have problem with our ADSL internet SP, That your site not working with. So, we contact them and they asking us to contact you to solve this problem.  The problem that when we are connecting to Tedata ADSL your website not working (Business Email).  BR,, Mohamed Omar 

                                  • Mail Not Showing In Inbox or Sent Box

                                    Hi, there are mails that are not displaying in both my inbox and sent box. I just tried the iPad app and it is the same but there is a blank field where a mail should be and it refers to a server error. Please fix this.

                                  • Marketing Automation List Entry Criteria is no longer an option

                                    For a couple of years now we have used the "List Entry Criteria" option to filter our Journey recipient list. All of a suddent the option no longer exists for New Lists and I can only delete the option from existing lists but can no longer edit it. Anyone

                                  • GCLID arrives not in CRM with iframe integrated ZOHO Form

                                    Hello amazing community, I enabled Adwords integration in ZOHO CRM. I have a ZOHO Form integrated in a wordpress. I tested iframe and Javascript. I enabled the "handover" GCLID in the ZOHO Form. When I add a GLID with http://www.example.com/?gclid=TeSter-123.

                                  • Bidirectional sync between Zoho Bookings and Zoho CRM - Part 1

                                    Hey, community members! The default integration between Zoho CRM and Zoho Bookings is unidirectional, which means that any appointments booked, rescheduled, or deleted in Zoho Bookings will be reflected in Zoho CRM. However, any modifications made to

                                  • Is it possible to edit placeholder text?

                                    In the master slides I want to make sure my templates have more meaning for the users and want to replace the standard placeholder text with something more instructional. Instead of "Click to edit Mast subtitle styles" I want to have "Click to enter reporting

                                  • How to allow download of a file AFTER information is collected?

                                    I can't find this anywhere. Can someone help with what seems to be a pretty basic web function today? - File is stored in Workdrive - Prospect goes to a URL where the file is located - System requests prospect to enter "Name" and "Email" - An email is

                                  • Add Baseline methods to API

                                    There are no API methods for creating or querying baselines,  Use case: Customer is notified of the current project start date according to plan via another product/integration. We would like to auto create a baseline (Automated via API) at the point

                                  • Fixed Assets beg balance

                                    Hello, I trust you are well I'm a new zoho books user and I find it difficult to plug in the fixed assets balance and here's what I have done: 1- I imported the fixed assets data into the fixed assets module as per the instructions 2- the import process

                                  • CRM Deluge how can link attach file with workdrive file.

                                    Hi, We have create file at workdrive and would like to link to attachment at crm module by deluge. Right now can only upload it to attachment but can not link the file from workdrive. Thank you

                                  • Feature announcement: AI-powered document generator in Zoho Sign

                                    Hi there! We recently announced manual document creation by offering an editor within Zoho Sign. Today, we are excited to announce AI-powered document generation to help you draft agreements, purchase orders, and other important business documents in

                                  • Permanently deleted files

                                    Is there a way to get back permanently deleted files from your account if they were accidentally deleted and no longer recoverable from the recycling bin? I know I am probably screwed but I just figured I would ask.

                                  • Shared Mailbox in iOS app

                                    Hi, On the desktop I am using a Shared mailbox for our company's general mailbox that somebody else should be able to view. In the iOS app however I can't find it? Old sources on the internet says this isn't possible but that can't be right? Somebody

                                  • Tip #41- Simplify Compliance and Efficiency with Zoho Assist’s Data Cleanup Feature- 'Insider Insights'

                                    Managing device data efficiently is just as important as securing remote access. With Zoho Assist’s Data Cleanup feature, admins can easily remove outdated or unnecessary device records, keeping the console organized and secure. This action is carried

                                  • Revenue Management: #6 Revenue Recognition in Professional Services Industry

                                    If you run a consulting firm, agency, or any service-based business, you have a situation where you have signed the contract, maybe even sent the invoice, and have received some initial payments. So, can you consider this as revenue? What if the project

                                  • Zoho fsm scheduled maintenance dispatch

                                    Hi, I'm reaching out regarding an issue I'm encountering with the Scheduled Maintenance feature in Zoho FSM. I’m trying to configure it so that it automatically dispatches all maintenance jobs scheduled within the next 150 days, starting from today. However,

                                  • zoho.inventory.getRecords returns no records

                                    I running a custom function (Deluge) in Zoho Flow with a connection to Zoho Inventory. To debug the problem, I created the test function below. It returns "Number of contacts fetched: 0". This confirms that there's no issue with the org ID or connection

                                  • Feature announcement - Simplifying document creation with Zoho Sign

                                    Hi there, We are pleased to announce the release of our document creation feature, enabling you to create, edit, and finalize documents like agreements, purchase orders, and contracts without relying on external tools or applications. &lt;br&gt; This

                                  • Integrating File Attachments in Zoho Flow Email Notifications

                                    Hi, I would like to be able to send an email once an entry is made in a form, and in this email, I would like to attach a file that has been added to the form using the ImageUpload (or FileUpload) widget. So, I started by creating a flow that will trigger

                                  • Zoho project purchases

                                    Hi Folks, I think having a purchase module in projects will help in putting together a project p and L. Project wise purchases and expenses.  Do let me know your thoughts.

                                  • How can I restore all the deleted Time entries of a Task in Zoho Projects

                                    How can I restore all the deleted Time entries of a Task in Zoho Projects? Query: In Zoho Projects, I cloned a task and deleted the time entries from the cloned task. However, this also deleted the time entries from the original task. How can I restore

                                  • Is there a way to make an account inactive in ZoHo Desk

                                    We have a few Clients "Accounts" that we no longer do business with. It would be beneficial for them to not show up in lists. However, we want to go back and view tickets, time, etc.

                                  • Next Page