Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions



Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:
  1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
  2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
  3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 



Post-migration, the Lead data showed the following inconsistencies and challenges:
  1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
  2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 
Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:
  1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
  2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
  3. The Country field value is used to identify and append the correct country code to the phone number.
  4. The processed records are updated back into Zoho CRM using Bulk Write API.
All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 
  1. Last Activity Days
  2. Web Engagement Score
Refer to the Working with Custom Fields help page for detailed guide. 


2. Create the following custom Single Line fields to store the calculated values during bulk processing:
  1. Lead Score Value
  2. Lead Segment
  3. Sales Priority
  4. Data Quality Status
3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.



5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>


7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.


9. Run the following command to install the CRM Bulk Data Processor CodeLib.



Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:
  1. Provide a Name and Description for the cron job. 
  2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
  3. Enter the following parameters and values:

    MODULES -> Leads
    FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status

    Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
     
  4. Choose the Schedule Type as One Time for our use case.
Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:
  1. ZohoFiles.files.ALL
  2. ZohoCRM.bulk.ALL
  3. ZohoCRM.modules.ALL
  4. ZohoCRM.settings.ALL
  5. ZohoCRM.org.ALL


3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.


Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 



Notes
These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.


The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Notes
Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 
  1. Country Names (India, Unites States, United Kingdom)
  2. Abbreviations (Ind, IN, USA, US, UK)
  3. Calling Codes (+91, +1, +44)
With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 


Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {
    private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");
    public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {
        if (rawPhone == null || rawPhone.trim().isEmpty()) {
            return Optional.empty();
        }
        String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");
        if (rawPhone.startsWith("+") && digits.length() >= 10) {
            return Optional.of("+" + digits);
        }
        switch (countryCode) {
            case "IN":
                return normalizeIndia(digits);
            case "US":
                return normalizeUS(digits);
            case "UK":
                return normalizeUK(digits);
            default:
                return Optional.empty();
        }
    }
    private static Optional<String> normalizeIndia(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+91" + digits);
        }
        if (digits.startsWith("91") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUS(String digits) {
        if (digits.length() == 10) {
            return Optional.of("+1" + digits);
        }
        if (digits.startsWith("1") && digits.length() == 11) {
            return Optional.of("+" + digits);
        }
        return Optional.empty();
    }
    private static Optional<String> normalizeUK(String digits) {
        if (digits.startsWith("44") && digits.length() == 12) {
            return Optional.of("+" + digits);
        }
        if (digits.length() == 10) {
            return Optional.of("+44" + digits);
        }
        return Optional.empty();
    }

}

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:
  1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
  2. If the company size is 200 employees or more, 20 points are added.
  3. If the Lead has been active within the last 7 days, 15 points are added.
  4. If the web engagement score is 70 or higher, 10 points are added.
The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:
  1. High (Hot): score ≥ 70
  2. Medium (Warm): score ≥ 40
  3. Low (Cold): score < 40
public static int calculateLeadScore(
            String jobTitle,
            Integer companySize,
            Integer lastActivityDays,
            Integer webEngagementScore) {
        int score = 0;
        // Decision Maker
        if (jobTitle != null) {
            String title = jobTitle.toLowerCase();
            if (title.contains("ceo") || title.contains("cto")
                    || title.contains("director") || title.contains("vp")) {
                score += 25;
            }
        }
        // Company Size
        if (companySize != null && companySize >= 200) {
            score += 20;
        }
        // Recency
        if (lastActivityDays != null && lastActivityDays <= 7) {
            score += 15;
        }
        // Engagement
        if (webEngagementScore != null && webEngagementScore >= 70) {
            score += 10;
        }
        return score;
    }
    public static String deriveSegment(int score) {
        if (score >= 70) return "Hot";
        if (score >= 40) return "Warm";
        return "Cold";
    }
    public static String derivePriority(String segment) {
        switch (segment) {
            case "Hot":
                return "High";
            case "Warm":
                return "Medium";
            default:
                return "Low";
        }
    }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.
  1. Clean: Both field values are valid.
  2. Critical: Both field values are invalid.
  3. Needs Review: Only one of the field values is invalid.

Write Back to Records:

The processed data is written back to Zoho CRM as follows:
  1. Mobile: Formatted phone numbers.
  2. Lead Score: Total score calculated from all applicable fields.
  3. Lead Segment: Segment derived from the lead score.
  4. Sales Priority: Priority assigned based on the lead segment.
  5. Data Quality Status: Overall quality of the Lead record.
if (normalizedMobile.isPresent()) {
data.put("Mobile", normalizedMobile.get());
}
data.put("Lead_Score", leadScore);
data.put("Lead_Segment", leadSegment);
data.put("Sales_Priority", salesPriority);
data.put("Data_Quality_Status", dataQualityStatus);
}

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

Info
You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.



We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Idea
Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

      Zoho Campaigns Resources


        • Desk Community Learning Series


        • Digest


        • Functions


        • Meetups


        • Kbase


        • Resources


        • Glossary


        • Desk Marketplace


        • MVP Corner


        • Word of the Day


        • Ask the Experts


          • Sticky Posts

          • Kaizen #198: Using Client Script for Custom Validation in Blueprint

            Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

          • Kaizen #226: Using ZRC in Client Script

            Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

          • Kaizen #222 - Client Script Support for Notes Related List

            Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

          • Kaizen #217 - Actions APIs : Tasks

            Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

          • Kaizen #216 - Actions APIs : Email Notifications

            Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

          Zoho CRM Plus Resources

            Zoho Books Resources


              Zoho Subscriptions Resources

                Zoho Projects Resources


                  Zoho Sprints Resources


                    Zoho Orchestly Resources


                      Zoho Creator Resources


                        Zoho WorkDrive Resources



                          Zoho CRM Resources

                          • CRM Community Learning Series

                            CRM Community Learning Series


                          • Tips

                            Tips

                          • Functions

                            Functions

                          • Meetups

                            Meetups

                          • Kbase

                            Kbase

                          • Resources

                            Resources

                          • Digest

                            Digest

                          • CRM Marketplace

                            CRM Marketplace

                          • MVP Corner

                            MVP Corner




                            Zoho Writer Writer

                            Get Started. Write Away!

                            Writer is a powerful online word processor, designed for collaborative work.

                              Zoho CRM コンテンツ




                                ご検討中の方

                                  • Recent Topics

                                  • Multi Page/Step Forms in creator

                                    Greetings i was wondering if it's possible to create multipage/step forms on creator similar to what we have on zoho forms. is that possbile? Thanks

                                  • Package Geometry

                                    how can i add the dimensions and weight capacity of the available boxes to be default in the system everytime we use it ?

                                  • How to create a Master Kanban Board that syncs with Child Projects?

                                    Hello, We're currently using Zoho Sprints for managing our interdepartmental teams, and we're looking to enhance our workflow using Kanban boards as part of a company-wide productivity improvement initiative. Our goal is to implement a project structure

                                  • Writer.. Broken?

                                    Hello,  Writer has been really good to me during the months I've used it, up until now.  I usually launch the app by tapping the icon and I could immediately pick up where I left off.  Now I'm greeted by a loading circle not reaching 100% and I only have the option to create a new account.  By pressing that button it now switches to a login screen and I can access my account. However, it seems (only speculating ofc) to be stuck in cell-phone mode? everything looks scrambled.  I can't access any of

                                  • How to access Recruit Variables in a Deluge function?

                                    I have set up Recruit Variables in Zoho Recruit, and I would like to know how to retrieve these variables from within a Recruit custom function (Deluge). Could someone please explain the correct way to access them? I tried the following code, but it did

                                  • Upon De activate a user what name doe sthe contacts candidates go under?

                                    When deactivating a user, does the user name remain the same, as the candidate owner? If not what/who, does it change to? Do I need to change the user name in contacts and candidates before I deactivate the user?

                                  • Weekly Tips: Customize alerts from your Priority Users

                                    You might receive hundreds of emails daily, but messages from your manager, clients, or team leads often require immediate attention, as they may contain urgent requests or critical updates. How would you ensure you never miss important messages from

                                  • Maximum 100 records in Sheet View is limiting. How can I increase this?

                                    Thanks in advance for any help with this. There was a similar post that showed answered but it did not help with increasing the number of records you see in a Sheet View. Editing in the Sheet View is fast and efficient but I have 3500 records and I need

                                  • Revenue Management: #3 Revenue Recognition Simplified

                                    In continuation of the previous post on how to compute revenue recognition, let's explore a solution that helps businesses handle real-world complexities. While the Accounting Standards provide a clear framework for recognizing revenue, the real challenge

                                  • Push Invoices to Xero Manually

                                    Hi guys, I'm wondering if anyone has wanted to do this and has a workaround or knows of an app that may be able to help with this. I sell B2B and B2C. The customers can purchase on our website or through marketplace, all of which send sales to zoho. The

                                  • OpenAI error code: 1010 in a Zobot

                                    Please see short linked screen recording. Insights welcome. Please and thank you! https://workdrive.zohoexternal.com/external/f3247ba9c872639157b707700c0300c433c7664aea924a034f4da3c3ad2e355f

                                  • Ability to Create Sub-Modules in Zoho CRM

                                    We believe there needs to be a better, more native way to manage related records in Zoho CRM without creating clutter. Ideally, Zoho would support "sub-modules" that we can create and associate under a parent module. Our use case: We have a custom module

                                  • Installing EMAIL Setup in New Domain

                                    Respected Support team, I'm facing an issue with cloudflare in Pakistan, I want to setup Zoho Mail Setup but I Don't know how to enable Zoho mail setup without cloudflare. My Website https://stumbleguymod.com/ is using CF, and I want a different Zoho

                                  • Signature change

                                    I cannot see how to change signature or out of office details easily now in the new format.

                                  • Inventory API - Retrieve all uploaded product / item images

                                    I know that I can get the primary image for each product / item or composite item, by using the /image endpoint.  https://inventory.zoho.com/api/v1/compositeitems/<item-id>/image?authtoken=<TOKEN> This will return only one photo, even if the item has multiple images uploaded. Is there a way to retrieve all images stored for an item via the Zoho Inventory API?

                                  • Ebay Integration malfunction

                                    My eBay integration in Inventory has always worked well. It suddenly malfunctioned. It is creating its own parts in Inventory that are unavailable instead of selling the parts I've always sold. Tech help was unable to resolve this. The latest sale attempt

                                  • how to get all the records in the custom View more than 200 records , Without using the page Concept

                                    how to get all the records in the custom View more than 200 records , Without giveing page as default in the Loop Concept Pls help how We can Achive this void schedule.Lead_Attempt_To_contact_schedule_10_30() { pages = {1,2}; for each pg in pages { query_map

                                  • The way that Users can view the ticket

                                    I have created users. What I would like to achieve is the following: All users under the same company account should be able to view each other’s tickets.

                                  • Zoho UAE SMS/WHATSAPP

                                    Hello everyone, so I have a question as regards DC and their impact on automation, integration and app usage. For example I am working with a UAE clientniw but each time I tried to connect their WhatsApp and sms then automate their process I tend to receive

                                  • Looking to Flag or Tag contacts/ accounts on Zoho Desk?

                                    I am looking for a way to flag certain accounts and make it obvious on the views pages. So for example if a has a certain package or needs extra attention it is clear before even clicking on the ticket. This could be via adding a tag or flag onto an account,

                                  • setting date-time field from string

                                    hello everyone, i hope someone could help me. i have a date-time field in a form that i want to fill in from two separate fields of date, and time. i need to combine the two fields to a one date-time field but can make it work. i tried to convert the

                                  • Calendars and CRM Contacts

                                    I'm finding having multiple calendars in Zoho One so confusing. I have a few questions so I can get this straight. We have a meeting room that we have set up as a resource in Calendar. Can this be set up in Bookings and the CRM Calendar? Using Zoho Calendar,

                                  • Announcing new features in Trident for Mac (v1.23.0)

                                    Hello everyone! Trident for macOS (v.1.23.0) is here with interesting features and thoughtful enhancements to elevate your workplace communication and productivity. Here's a quick look at what's new. Record your meetings. You can now record audio and

                                  • Applying a record template

                                    Hi all, I can't figure this out. I hope you can help. The scenario: We have learners who have to complete a 'digital' journal with tasks in order to qualify. Those tasks, once completed, need a final signature from their 'Mentor', which will trigger their

                                  • Quickbooks invoice with Zoho Creator

                                    Is it possible to push data from Zoho Creator directly to an invoice on QuickBooks? If so, where can I find information on how to do this?

                                  • Help: Capture full page URL in hidden field when same Zoho Form is embedded on multiple pages (iframe)

                                    Hi all, Goal Use one Zoho Form across multiple pages and record the exact page URL (incl. subdomain + path + hash) where the user submitted it. Example pages: https://www.example.com/cargo/ https://www.example.com/cargo/containers/#contact https://cargo.example.com/auto/

                                  • Automatically Populate fields - HELP!

                                    There have been many discussions on this but I still can't seem to get it to work for me. I am trying to create a lookup field and have other fields automatically populate. Based on the instructions in the Help Center, I should be using the "on user input". It's just not working, here is the layout...   Both forms are in the same application. Current form is called Add Note, form to fetch records from is called Add Client. Lookup field is called Select_Client_ID related field in fetch form is called

                                  • Push notifications to portal users

                                    Hi all, it is possible to send push notifications to portal users?

                                  • Rename the attachment from record template pdf in the sendmail deluge script

                                    Hello Zoho, I urgently need a feature to rename record templates that I send via the sendmail feature. The program I created sends emails to clients with an invoices that have been created in the invoice database. If the user selects 3 invoice numbers,

                                  • Alt Text On Multiple Images

                                    I'm using Zoho Social to post to a charity website. Often the posts have multiple images, but it seems there is only one field for Alt Text. Does that mean I can only include it for the first image? Or is there a way to add alt text for all the imag

                                  • [SOLVED] Getting 401 when trying to download ticket attachment via API

                                    I'm able to use the API just fine to access ticket content. But I cannot download ticket attachment, keep getting 401 Client Error. Example: https://desk.zoho.com.au/api/v1/tickets/{ticket_id}/attachments/{attachment_id}/content?orgId={org_id} For headers,

                                  • Zoho project – Workdrive integration.

                                    Hello everyone, I was wondering something, we did the Zoho projects integration with Zoho Workdrive but nowhere during this integration we could set the location of the folders that would be automatically created in Workdrive. As I understand it, it creates

                                  • Scriptを埋め込みたくてOn User Inputを探しているのですが・・・

                                    編集モードで、Scriptを埋め込みたい項目を選択し、「項目のプロパティ」パネルで、その下のほうに「フィールドアクション（Field Actions）」という見出しがあると聞いたのですが、そもそも、それが見つかりません。そのために、On User Inputなどのイベントが選べません。 画面の英語を日本語に訳しているためにわけわからん状態になっているのかも知れませんが、わかる方、いらっしゃいますか？

                                  • Problem with cloud query exceeded

                                    When making a call I get this error, It is associated with a function in node that calls external APIs This is the code //tokenConsultar = thisapp.ObtenerToken(); //input.token = tokenConsultar.get("output").toMap().get("token");

                                  • Showing Total of Funnel Chart Legend (With Values) Items

                                    Hi, We are using the funnel type chart for displaying our zCRM sales pipeline stages and associated sum of deals in each stage.  We have configured the legend to show the associated value (Deal Amount (Sum)) for each stage adjacent to the legend items (Stages). Is there a way to display the total of the values in the same chart?  For example, at the bottom of the legend or maybe as a #merge placeholder in the legend title?

                                  • Field authorization for Linking module in Zoho CRM portal

                                    Hi guys! Currently building a customer portal for one of my clients, and I ran into a bit of a roadblock while using a multi-select lookup field. The issue is that there is no way to hide or define access in the linking module created by this multi-select

                                  • Kaizen #63 - Layout Rules in Zoho CRM

                                    Hello and welcome to another week of Kaizen! This week, we will be discussing Layout Rules in Zoho CRM. If you need to modify the layout of a module based on user inputs, or to show or hide sections based on the value of a specific field, we have got

                                  • "Trigger flow for each entry" not working

                                    I've read this article on how to trigger a flow for each entry in my array. https://help.zoho.com/portal/en/kb/flow/user-guide/create-a-flow/articles/webhook-trigger#Trigger_flow_for_each_entry I cannot get the flow to run for each variant in this JSON

                                  • Zoho CRM Portals - allow access per account

                                    Hello all, I am trying to set up a portal for our customer but I seem to be hitting an obstacle and I am not sure if it is my problem or a limitation in the software. So basically the way I understand the portal Contact Email > Each Record or Related

                                  • Android - Writer não acentua em Português com teclado bluetooth

                                    Gosto muito do Zoho, tanto o Note quando o Writer. Infelizmente, o Writer sofre de um problema sério: ao usar um teclado bluetooth, forma mais cômoda de lidar com um processador de texto, os acentos (todos!) da Língua Portuguesa não são aceitos. Todos

                                  • Next Page