Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Does anyone know how to setup Zoho Desk or Zoho CRM as a custom outgoing/incoming Call Centre?

  I need to setup a call center so I can setup agents to make phone calls across Canada to market our services.  I am trying to figure out the most reliable and cost efficient way to do this? I am currently paying for two phone services and neither seem

• Display actual mileage on an invoice

  My users are creating expenses in Zoho expense. For example, they expense 10 miles and get paid 7 dollars (10 miles * IRS rate of .70). If I look at the expenses in Zoho Books, it does show them at 10 miles at .70 cent When I add these expense to an invoice

• Prevent Unapproved Quotes from Exporting to Zoho CRM Finance Module

  Is it possible to prevent unapproved quotes in Zoho Books from being exported from Zoho Finance module inside Zoho CRM?

• Default font size for printing is too big

  A recent issue in printing e mails is that the default font size is huge. This happens in both Edge and Firefox. In order to get what I would call a "normal" printout of an e mail it is necessary to go into the printer preferences / options and set the scale to 75%, otherwise a short e mail with signature and logos etc printed at 100% can take 3 or 4 pages. The annoying thing is that it is necessary to do this each time a printout is to be made as a change in scale only applies to that particular

• Request for Auto PO - Min–Max based Automated Purchase Feature

  Dear Zoho POS Team, I’m writing to request a feature enhancement that would significantly streamline inventory management for businesses using Zoho POS — particularly supermarkets, FMCG retail, and multi-store operations like ours. Feature Requested:

• Export Invoices to XML file

  Namaste! ZOHO suite of Apps is awesome and we as Partner, would like to use and implement the app´s from the Financial suite like ZOHO Invoice, but, in Portugal, we can only use certified Invoice Software and for this reason, we need to develop/customize on top of ZOHO Invoice to create an XML file with specific information and after this, go to the government and certified the software. As soon as we have for example, ZOHO CRM integrated with ZOHO Invoice up and running, our business opportunities

• issue with deluge script

  i used chat gpt to build this script and I am getting 2 errors which I cannot figure out how to fix: void monthly_sales_order_generation() { try { // ---------------- CONFIG ------------------- analytics_url_1 = "https://analytics.zoho.com/api/<workspaceID>/report/<reportID1>/data";

• Introducing LinkedIn Apply Connect for Zoho Recruit

  Attract up to 3x more qualified candidates and hire more efficiently with LinkedIn Apply Connect. Let candidates fill-in job applications without any redirections, gain deeper insights with applicant highlights within Zoho Recruit, and keep candidates

• Recruit paid support?

  Hi all, Could anyone who has paid support package advise if it provides value for money with regards to support response times? Exploring the idea as unfortunately when we have faced issues with Recruit it has been a 7+ day timescale from reporting to

• Ticket Status email

  Good day, This was discussed in the past, but it would be helpful if we could have the system assign a custom response to a status. We have various statuses for tickets, e.g. "closed due to no response", or "Pending Status", it would be helpful for the

• Future Orders - Due Date

  Hi In my role, I can receive tickets where the work required is requested months in advance. Using a Future Orders option, which I believe was setup under the On Hold status type, hides the Due Date, in all views/ticket etc. Whilst I understand the reasoning

• Introducing Withdrawal Reasons for Offers

  We’re excited to introduce a new enhancement to the Offer module that brings more clarity and accountability to every withdrawn offer. The Withdrawal Reason update ensures that each withdrawal — manual or automatic — is backed by a clear explanation,

• Zoho Creator customer portal users

  Hi, I'm in a Zoho One subscription with our company. I'm running a project now that involves creating a Zoho Creater application and using the Zoho Creator Customer Portal.  At most we need 25 customer portal users. In our Zoho One plan we only get 3

• GPS tracking only or Check out facility

  Dear Team, Zoho CRM is quite robust CRM but till date i was waiting for the feature of having GPS tracking of Sales employees which is a common demand by all customers for thier field sales executives. We cover them by saying that CRM provides Checkin

• Appraisals on Employee Information Profile

  Is it possible to show completed appraisals on each employee's "Employee Information" page? I would presume at the bottom - similar to the "Related Lists" concept in Zoho CRM. Obviously view access would be limited to employee and appropriate other roles

• Kaizen #220: Actions API - Webhooks APIs - Part 2

  Hello all!! Welcome back to the follow-up Kaizen post of Kaizen #219: Actions API - Webhooks APIs - Part 1. In the previous week, we covered how to configure a basic Webhook and how to include Headers, Body, and URL Parameters using both the POST Webhook

• Standard Payment Term is not pulled from account to quotation

  Hey Team There seems to be something off. I do have "Net 30" as my default payment term in Zoho Books for my customers. If, from the customer overview or quote section, I create a new Quotation, the payment terms field stays blank and doesn't get the

• Manage your invoices from Bigin's mobile app (iOS)

  Hello everyone! We're happy to announce that users can now integrate Zoho Books with the latest version of Bigin's iOS app. Zoho Books can be integrated with Bigin only via the web application. Users can view the Zoho Books tab in the detail pages of

• Systematic SPF alignment issues with Zoho subdomains

  Analysis Period: August 19 - September 1, 2025 PROBLEM SUMMARY Multiple Zoho services are causing systematic SPF authentication failures in DMARC reports from major email providers (Google, Microsoft, Zoho). While emails are successfully delivered due

• Accessibility controls and multilingual captcha authorization for Help Center users ​

  Hello everyone, As part of our ongoing effort to improve user experience, we are excited about the accessibility controls and predefined accessibility personas added in the Help Center, similar to what is already available in Zoho Desk. Help Center users

• Zoho Sign + Zoho CRM : la solution pour simplifier vos accords de vente

  La conclusion d’un contrat de vente nécessite de nombreuses étapes : préparation de documents, validations successives et collecte des signatures. Les équipes commerciales passent souvent par plusieurs cycles de collaboration et de révision, tout en assurant

• Item/service subtotal

  Just discovered & really pleased that we can drag to re-order the line items in Sales orders & Invoices, a very nice feature which doesn't seem to be documented? It would be nice to be able to insert a subtotal as a line item to complete this great feature

• Send email template "permission denied to access the api"

  Hello, Per the title, I'm trying to send a Zoho CRM Email template based on the advice given here: https://help.zoho.com/portal/en/community/topic/specify-an-email-template-when-sending-emails-in-custom-functions (I'd prefer to send right from Deluge

• how to Solve Conflict Invoices in Zoho POS

  Hello Team, I am facing a repeated issue in Zoho POS while saving a sale that contains service-based items. My products are intentionally created as Service (Non-Inventory) items because I do not want to track stock for them. However, every time I try

• Copy, Paste, Highlighted text

  when i copy paste text it gets highlighted. Fix this.  WTF is with this black highlight ? Fix copy pasting. Some people pay for using this mail service. 

• Pricing Strategies: #6 Common Mistakes while Configuring Pricing

  "Why does this month's billing feel messy again?" That was the exact line Priya muttered while staring at the spreadsheet full of edited pricing, one-off discounts and mismatched subscription details. Her business was growing, but the pricing machine

• Publish to LinkedIn via API

  Hi, Is it possible to publish a job opening to LinkedIn (paid job slots) if creating a job opening via api / deluge function? Or is the user required to manually publish via the job boards process? Many Thanks Adrian

• FSM - Associating and selecting Contacts based on Service Addresses

  Hi FSM team, I've come across an FSM limitation I wanted to share for improvement. I'm currently configuring FSM for a client who provides heating system install and maintenance services. The are often sub contracted by building management companies to

• Do Unpublished Products in Shopify Sync to Zoho POS?

  Will an unpublished product in Shopify sync to Zoho POS? I would like to know how Zoho POS handles products that are in draft/unpublished status in Shopify.

• Problem in Zoho POS frontend ERP.

  We are facing some difficulties in our billing and inventory processes and require your support to resolve them at the earliest. Customer Advance Payments We receive advance payments from customers for upcoming sales. Please guide us on how to record

• Categorize Items with Item Headers

  Hello customers, Did you ever want to classify items based on specific categories to help your customers understand your invoice better? With the new Item Header feature, you can easily categorize items in your invoices and estimates and give them a common title.  Item Headers are available in the Invoices, the Recurring Invoices and the Estimates module. It can be carried forward from estimates to invoices at the time of converting the estimates.  To add an item header: Go to the Estimates, Invoices

• Discount on the transaction level in POS

  "Is it possible to apply a discount on the total bill before tax in Zoho POS?" I want to apply a discount on the total transaction amount in Zoho POS, but the discount should be calculated before the taxes are applied. Is there an option to do this in

• Send Automated WhatsApp Messages and Leverage the Improved WhatsApp Templates

  Greetings, I hope all of you are doing well. We're excited to announce a major upgrade to Bigin's WhatsApp integration that brings more flexibility, interactivity, and automation to your customer messaging. WhatsApp message automation You can now use

• Jobs at multiple locations

  Hi A lot of times we have same job open at multiple locations. So I would like an option where we can create a job and choose multiple locations. Currently we have to create individual jobs for each location. If we can create one job and mention multiple

• Tag Adding Option in Kanban Card Customization Bigin CRM Mobile App

  I would like to request an option to add and display tags on Kanban cards in the Bigin CRM mobile app. This feature would make it easier to categorize deals and quickly identify priorities while working on the go.

• Empowered Custom Views: Cross-Module Criteria Now Supported in Zoho CRM

  Hello everyone, We’re excited to introduce cross-module criteria support in custom views! Custom views provide personalized perspectives on your data and that you can save for future use. You can share these views with all users or specific individuals

• Associating a Candidate to a Job Listing

  Hello, I am trying to use Zoho Forms embedded on my website for candidates to apply for a job opening. I want the form then to tie directly with zoho recruit and have the candidate be automatically inputed into Recruit as well as associated with the specific

• WhatsApp Voice Message Sending Option in Bigin CRM

  I would like to request a feature that allows users to send WhatsApp voice messages directly from Bigin CRM. This would help improve communication efficiency and make it easier for teams to respond quickly to customer inquiries.

• CRM: hosting a single html file in Zoho and displaying it as a widget

  I have seen that CRM offers the option of uploading a web project to Zoho itself and displaying it as a widget in CRM. The instructions then talk about setting a development environment with Node and developing an application to upload to Zoho. But I

• How to change position button transtition of Blueprint?

  Hi Everyone, Look at my screenshoot, it is possible move the reject button to the right? I couldn't find that setting in the blueprint. Thank you.

• Next Page