Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Automation#33: Automate Splitting Names for Existing Contact Records

  An organized directory – who doesn't love one? Previously, we explored how to split contact names into First Name and Last Name for new contacts in Zoho Desk. But what about existing contacts already in your database? This week, we bring you a custom

• Workqueue

  I really like the idea of the Workqueue generally - it will be really useful. What it seems to lack however, is the ability to customise it properly. I want to be able to show a custom view rather than just "My Leads" and "Leads Assigned in Last 3 hours".

• Webform & spam

  Hi, We set up 2 webform on our website, fowarding the content to Zoho CRM. Since it has been opened up, we are getting lot of spam message (for now about 20 a day). To lower the amount of false new leads we added the captcha field and new enquieries are

• Zoho CRM Web form page URL

  Hi there, I am getting quite a bit of spam through my Zoho CRM Web form and want to restrict the URLs it is submittable from. I can see that I can add Location URLs but I need to be able to just add a domain so can I enter mydomain.com/* to cover all

• Receiving too many Spam Leads. Why?

  I am receiving so many junk leads from web forms created by zoho's platform. The junk queries are increasing day by day and are affecting our business. I am continuously following up with zoho team from the past one year but not getting any satisfactory

• Mail Merge is not working properly as far as the AUTOMATE section is concerned

  Hi there, I created a Mail Merge template for the Deal module. I would like Deal owners to mail merge their Deal records, download the Mail Merge document as a Word doc and make a few changes before sending it to the customer. Thing is, neither the "Merge

• Canvas Detail View Related List Sorting

  Hello, I am having an issue finding a way to sort a related list within a canvas detail view. I have sorted the related list on the page layout associated with the canvas view, but that does not transfer to the canvas view. What am I missing?

• project name field issue- n8n

  Hey guys, I have a question. I want to create a new product using the workflow. The problem is with the product name field; I don't know how to fill it in. The workflow starts with retrieving information from the leads table, retrieving links to scrape

• Function #20: Custom calculation in item table of invoices

  When you create an invoice in Zoho Books, the 'Amount' of a line item is typically calculated as the product of the "Quantity" and the "Rate" of the item. For instance, if an item has a sales rate of $50 and a quantity of 5 is sold, then the amount would

• Critical:- Eneble TDS filing for 26Q from the zoho book

  We currently extract TDS data from Zoho Books and manually input it into a separate TDS software to generate the FUV file and file returns. Previously, while using Tally, we benefited from an integrated feature that seamlessly recorded transactions and

• How to track repeat customers?

  I own a food business and every order is entered into Zoho with: a unique Customer ID total order amount date of order With this information, I want to be able to see a list of my "best" customers. In other words, descending lists arranged according to:

• How to hide or archive a blog post temporarily in Zoho commerce website builder?

  I would like to temporarily hide or archive a blog post in zoho commerce website builder so that it doesnt appear on my website till I enable it again. I tried to look for this option but could not find it.  It only allows me to permanently delete a blog

• Zoho Books - Breaking A Working App

  We've been using Zoho for many years now. Across all apps, entering phone numbers in standard formats was enabled in all apps. These formats are: xxx.yyy.zzzz xxx-yyy-zzzz (xxx) yyy-zzzz and we were able also to add extension numbers in these formats:

• Build data protection into your support

  At Zoho, privacy is our principle. Every Zoho product is built with privacy as the foundation and the finishing touch, guiding every decision we make. Security, privacy, and compliance are woven into the software development lifecycle, starting from how

• Conditional formatting: before/after "today" not available

  When setting conditional formatting, it only allows me to set a specific calendar date when choosing "Before" or "After" conditions. Typing "today" returns the error "Value must be of type date". Is there a workaround? Thanks for any help!

• Display Client Name in Zoho Creator Client Portal Dashboard

  Hello Zoho Creator Team, We hope you are doing well. Zoho Creator recently introduced the option to set a client’s display name in the Client Portal settings, which is very helpful for providing a personalized portal experience. However, there is currently

• Customizable UI components in pages | Theme builder

  Anyone know when these roadmap items are scheduled for release? They were originally scheduled for Q4 2025. https://www.zoho.com/creator/product-roadmap.html

• Amazon.in FBA multiple warehouse integration with Zoho Inventory

  My organisation subscribed to Zoho One looking at the opportunity to integrate Amazon.in with Inventory. But during the configuration, we understood the integration has severe limitations when it involves multiple warehouses in the same Organisation.

• Feature Request - Set Default Values for Meetings

  Hi Zoho CRM Team, It would be very useful if we could set default values for meeting parameters. For example, if you always wanted Reminder 1 Day before. Currently you need to remember to choose it for every meeting. Also being able to use merge tags

• Issues with Actions By Zoho Flow

  Hi, I have a workflow that fires when a deal reaches a stage. This then sends out a contract for the client to sign. I have connected this up through Actions by Zoho Flow. Unfortunately this fails to send out. I have tracked it down to the date fields.

• CRM x WorkDrive: We're rolling out the WorkDrive-powered file storage experience for existing users

  Release plan: Gradual rollout to customers without file storage add-ons, in this order: 1. Standalone CRM 2. CRM Plus and Zoho One DCs: All | Editions: All Available now for: - Standalone CRM accounts in Free and Standard editions without file storage

• Windows Desktop App - request to add minimization/startup options

  Support Team, Can you submit the following request to your development team? Here is what would be optimal in my opinion from UX perspective: 1) In the "Application Menu", add a menu item to Exit the app, as well as an alt-key shortcut for these menus

• Ability for admin to access or make changes in zoho form without asking for ownership

  Currently in zoho form only form owner can make the changes in the form and if someone else has to make changes then we have to transfer the ownership to them and even admin also cant access it . So i think admin must have the ability or option to access

• Issue with WhatsApp Template Approval and Marketing Message Limit in Zoho Bigin

  We are facing issues while creating and using WhatsApp message templates through Zoho Bigin, and we request your clarification and support regarding the same. 1. Utility Template Approval Issue Until December, we were able to create WhatsApp templates

• Zoho CRM Calendar View

  Hello Zoho team, We need desperately a calendar view next to list, kandan and other views. I think it should be easy to implement as you already have the logic from Projects and also from Kanban View in CRM. In calendar view when we set it up - we choose

• Camera

  I can sign on to a meeting and see the other participants, but my screen is dark. The instructions for Zoho "Camera Settings" say "click on lock icon in address bar," but I don't see that icon! Suggestions?

• What is Workqueue and how to hide it?

  Hi， My CRM suddenly have this "Workqueue", may I ask how to set the permission of this tab?

• Batch/lot # and Storage bin location

  Hi I want to ask for a feature on Zoho inventory I own a warehouse and I've gone through different management software solutions with no luck until I found Zoho, it has been a game changer for my business with up to the minute information, I'm extremely happy with it. It's almost perfect. And I say Almost because the only thing missing for me (and I'm sure I'm not alone) is the need of being able to identify the lot number of my inventory and where it is located in the warehouse. Due to the nature

• Adding Sender Address with Basic Plan

  According to the knowledge base, I should be able to add Sender addresses with the Basic Plan. But whenever I try to add an email, it takes me to a search window and I cannot find any emails in the list. Even mine, which is the admin. email.

• Conditional Field Visibility in Bigin CRM

  I would like to request support for conditional field visibility within Bigin CRM. This feature should allow administrators to configure show/hide rules for fields based on predefined criteria (e.g., field values, picklist selections, stage changes,

• Bill automation in Zoho Books

  Hi I am looking for 3rd-party options for bill automation in zoho which are economical and preferably have accurate scanning. What options do I have? Zoho's native scanning is a bit pricey

• Reporting Tags

  We've been using reporting tags for years (before itemizing was available) and now we are finding reporting these tags are impossible to track. Reports have changed in the customization and our columns of reporting tags no longer show up. We do not use

• Consumption based inventory

  I am currently using Zoho Books for my hospitality business, which includes lodging and restaurant services. We purchase many items in bulk for storage and consumption as needed. I'd like these items to be recorded as inventory when purchased and categorized

• Smarter Access Control: Role-Based Access vs. Responsibility-Based Profiles

  Every business has roles, responsibilities, and workflows. While roles help define structure, responsibilities within those roles are rarely the same. As your team grows, some members need access to only a specific set of features. Others require visibility

• Partner with HDFC And Sbi Bank.

  Hdfc and sbi both are very popular bank if zoho books become partner with this banks then many of the zoho books users will benefit premium features of partnered banks.

• API in E-Invoice/GST portal

  Hi, Do I have to change the api in gst/e-invoice portal as I use zoho e books for my e-invoicing. If yes, please confirm the process.

• Member role in zoho meeting

  does a user with member role can see other users in the organization

• How to post more than 4 Images on Instagram?

  Hi I read several articles to the topic od how to post more than 4 images on instagram, but i can't figure out how it works. I can't find the content editor and i installesd the z share extension for google chrome. Could someone please help me? Than

• Clone Recurring Expenses

  Our bookkeeping practices make extensive use of the "clone" feature for bills, expenses, invoices, etc. This cuts down significantly on both the amount of typing that needs to be done manually and, more importantly, the mental overhead of choosing the

• Zoho Books - How to Invoke a Custom Function in Schedulers

  We have multiple schedulers that send emails to customers in batches. Currently, we are maintaining the same code across several schedulers. Is it possible to use a custom function inside a scheduler script? If yes, how can we invoke the custom function

• Next Page