Kaizen 228 - Process Large-Scale Migrated Data Using Catalyst Solutions

Howdy, tech wizards!

This week’s Kaizen explores how Catalyst Solutions in the Zoho CRM Developer Hub help import large volumes of data into Zoho CRM while improving data quality and simplifying the migration process.

Why Catalyst Solutions?

Catalyst Solutions provide a library of independent, ready-to-use backend solutions called CodeLibs. The CRM Bulk Data Processor CodeLib helps handle complex data operations on large datasets. Following are the key benefits:

1. Highly Scalable: Processes large datasets efficiently, supporting up to 200,000 records per API call.
2. Automation Made Simple: Automate complex tasks like segmentation, data cleansing and lead scoring with minimal effort.  
3. Effortless Integration: Comes with pre-configured Catalyst resources such as serverless functions and data stores that integrate directly with Zoho CRM.
   

Business Scenario

Zylker, a manufacturing company, migrated over 3 lakh Lead records into Zoho CRM from spreadsheets. 

Post-migration, the Lead data showed the following inconsistencies and challenges:

1. Phone number field did not include country codes, or contained incorrect country codes. This inconsistency made it difficult for sales teams to contact Leads reliably.
2. Identifying potential Leads for follow-up required manual effort and marketing team has to run generic campaigns without lead segmentation. 

Using workflows before record creation to fix phone numbers for this volume would result in 3 lakh individual workflow executions, which can slow imports and cause partial failures.

Solution Overview

Zylker can use the CRM Bulk Data Processor available in Catalyst Solutions to address these challenges at scale.

The solution works as follows:

1. Lead records are fetched in bulk from Zoho CRM using the Bulk Read API.
2. The fetched data is temporarily stored to evaluate the priority and quality of each record. Based on this evaluation, the records are also segmented accordingly.
3. The Country field value is used to identify and append the correct country code to the phone number.
4. The processed records are updated back into Zoho CRM using Bulk Write API.

All operations are executed using pre-configured Catalyst resources provided by the CodeLib. You can explore the Bulk Data Flow in Catalyst help page to understand how each resource participates in the processing pipeline.

Prerequisites

Before you begin, ensure the following:

1. Log into your Zoho CRM and navigate to Setup > Customization > Modules and Fields > Leads.

Create the following custom Single Line fields to map existing record data in Zoho CRM. 

1. Last Activity Days
2. Web Engagement Score

Refer to the Working with Custom Fields help page for detailed guide. 

2. Create the following custom Single Line fields to store the calculated values during bulk processing:

1. Lead Score Value
2. Lead Segment
3. Sales Priority
4. Data Quality Status
   

3. Go to Setup > Developer Hub > Catalyst Solutions > Zoho CRM Bulk Data Processor. 

4. Click Go to Catalyst and create a Catalyst account if required.

5. Refer to the Catalyst CLI Installation Guide and install it on your local system.

6. Create a new folder on your local system, which will act as a local project directory. Use the following command:

mkdir -p <folder_name>

7. Navigate to the project folder in your terminal and execute the following command: 

catalyst init

8. Follow the prompts to select and initialize your desired project from the Catalyst console. 

When prompted to choose a feature, press Enter and proceed without choosing any feature.

9. Run the following command to install the CRM Bulk Data Processor CodeLib.

catalyst codelib:install https://github.com/catalystbyzoho/codelib-zoho-crm-bulk-processor

Step 1: Create a Cron Job

1. Open the Catalyst Console and navigate to the same project in which you have installed the CRM Bulk Data Processor CodeLib.

2. Go to Cloud Scale > Triggers > Cron and click Create Cron.

3. Fill in the following details:

1. Provide a Name and Description for the cron job. 
2. Select Function and choose BulkJobScheduler from the dropdown in the Target Function field.
3. Enter the following parameters and values:
   
   MODULES -> Leads
   FIELDS_TO_BE_PROCESSED -> Last_Name, Company, Mobile, Country, Designation, No_of_Employees, Web_Engagement_Score, Lead_Score_Value, Lead_Segment, Sales_Priority, Data_Quality_Status
   
   Use GET Modules Metadata API and GET Fields Metadata API to get the API names of the required module and fields.
    
4. Choose the Schedule Type as One Time for our use case.
   

Refer to the Implementing Cron Jobs Help Guide for more details.

Step 2: Configure Zoho CRM API Credentials 

1. Create a self-client application in Zoho's API Console.

2. Generate a Grant Token with the following scopes:

1. ZohoFiles.files.ALL
2. ZohoCRM.bulk.ALL
3. ZohoCRM.modules.ALL
4. ZohoCRM.settings.ALL
5. ZohoCRM.org.ALL
   

3. Follow the OAuth Help Section for a step-by-step guide to generate Access and Refresh tokens for your Zoho CRM organization.

Store all the API credentials securely for later use.

4. To allow Catalyst to access Zoho CRM data, configure the stored API credentials as environment variables in the catalyst-config.json file within the BulkDataProcessor function.

For detailed guidance, refer to the Configure Function Components help guide in CRM Bulk Data Processor. 

These environment variables are used by Catalyst Connectors to create access tokens for establishing secure connection between Zoho CRM and Catalyst.

Step 3: Add Business Logic

1. Go to com.processor.record.ZCRMRecordsProcessorImpl.java file of the BulkDataProcessor in the functions directory.

2. Override the ZCRMRecordsProcessor method with the business logic.

The complete code sample for this use case is available on GitHub for reference.

Following is the major custom logic used here:

Use the GET Fields Metadata API to fetch the API names of the fields required throughout the custom logic.

Phone Number Normalization:

This can be implemented in two stages, Country standardization and Phone Number formatting. 

For country standardization, CountryStandardizerUtil maps different variations of the Country field in records to a single standardized country identifier. These variations may include, 

1. Country Names (India, Unites States, United Kingdom)
2. Abbreviations (Ind, IN, USA, US, UK)
3. Calling Codes (+91, +1, +44)
   

With this, we can ensure different interpretations of the same country are interpreted consistently during the process. 

Once the country is standardized, Phone Numbers are formatted to E.164 format in the PhoneNormalizerUtil. 

It removes all non-numeric characters and converts the Phone input to E.164 format with the respective country codes. 

public class PhoneNormalizerUtil {     private static final Pattern NON_DIGIT = Pattern.compile("[^0-9]");     public static Optional<String> normalizeToE164(String rawPhone, String countryCode) {         if (rawPhone == null || rawPhone.trim().isEmpty()) {             return Optional.empty();         }         String digits = NON_DIGIT.matcher(rawPhone).replaceAll("");         if (rawPhone.startsWith("+") && digits.length() >= 10) {             return Optional.of("+" + digits);         }         switch (countryCode) {             case "IN":                 return normalizeIndia(digits);             case "US":                 return normalizeUS(digits);             case "UK":                 return normalizeUK(digits);             default:                 return Optional.empty();         }     }     private static Optional<String> normalizeIndia(String digits) {         if (digits.length() == 10) {             return Optional.of("+91" + digits);         }         if (digits.startsWith("91") && digits.length() == 12) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUS(String digits) {         if (digits.length() == 10) {             return Optional.of("+1" + digits);         }         if (digits.startsWith("1") && digits.length() == 11) {             return Optional.of("+" + digits);         }         return Optional.empty();     }     private static Optional<String> normalizeUK(String digits) {         if (digits.startsWith("44") && digits.length() == 12) {             return Optional.of("+" + digits);         }         if (digits.length() == 10) {             return Optional.of("+44" + digits);         }         return Optional.empty();     } }

Lead Priority and Segmentation

The lead priority is calculated using the Title, No of employees, Last Activity Days and Web Engagement Score fields in the record.  

Each field value contributes a predefined number of points:

1. If the job title indicates a senior role such as CEO, CTO, Director, or VP, 25 points are added.
2. If the company size is 200 employees or more, 20 points are added.
3. If the Lead has been active within the last 7 days, 15 points are added.
4. If the web engagement score is 70 or higher, 10 points are added.

The sum of all applicable factors is returned as an integer and based on this score the lead's priority and segmentation is assigned as follows:

1. High (Hot): score ≥ 70
2. Medium (Warm): score ≥ 40
3. Low (Cold): score < 40
   

public static int calculateLeadScore(             String jobTitle,             Integer companySize,             Integer lastActivityDays,             Integer webEngagementScore) {         int score = 0;         // Decision Maker         if (jobTitle != null) {             String title = jobTitle.toLowerCase();             if (title.contains("ceo") || title.contains("cto")                     || title.contains("director") || title.contains("vp")) {                 score += 25;             }         }         // Company Size         if (companySize != null && companySize >= 200) {             score += 20;         }         // Recency         if (lastActivityDays != null && lastActivityDays <= 7) {             score += 15;         }         // Engagement         if (webEngagementScore != null && webEngagementScore >= 70) {             score += 10;         }         return score;     }     public static String deriveSegment(int score) {         if (score >= 70) return "Hot";         if (score >= 40) return "Warm";         return "Cold";     }     public static String derivePriority(String segment) {         switch (segment) {             case "Hot":                 return "High";             case "Warm":                 return "Medium";             default:                 return "Low";         }     }

Lead Quality:

Lead quality is evaluated based on the validity of the Mobile and Country fields in the records.

1. Clean: Both field values are valid.
2. Critical: Both field values are invalid.
3. Needs Review: Only one of the field values is invalid.
   

Write Back to Records:

The processed data is written back to Zoho CRM as follows:

1. Mobile: Formatted phone numbers.
2. Lead Score: Total score calculated from all applicable fields.
3. Lead Segment: Segment derived from the lead score.
4. Sales Priority: Priority assigned based on the lead segment.
5. Data Quality Status: Overall quality of the Lead record.
   

if (normalizedMobile.isPresent()) { data.put("Mobile", normalizedMobile.get()); } data.put("Lead_Score", leadScore); data.put("Lead_Segment", leadSegment); data.put("Sales_Priority", salesPriority); data.put("Data_Quality_Status", dataQualityStatus); }

Step 4: Deploy to Development

Run the following command to deploy your local customizations to the Development Environment of Catalyst.

catalyst deploy

You can find the complete code sample on GitHub for reference.

Try it Out!

With the changes deployed to the Development environment, let us now test the solution.

We hope this Kaizen helps you import large volumes of data into Zoho CRM while improving data quality and lead intelligence using the CRM Bulk Data Processor.

Have questions or suggestions? Drop them in the comments or write to us at  support@zohocrm.com. 

We will circle back to you next Friday with another interesting topic. 

On to Better Building!

--------------------------------------------------------------------------------------------------------------------------

Previous Kaizen: Client Script Support for List Page (Canvas) | Kaizen Collection: Directory

--------------------------------------------------------------------------------------------------------------------------

• Topic Participants

• Kiruthiga G

  

• Sticky Posts

• Kaizen #198: Using Client Script for Custom Validation in Blueprint

  Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

• Kaizen #226: Using ZRC in Client Script

  Hello everyone! Welcome to another week of Kaizen. In today's post, lets see what is ZRC (Zoho Request Client) and how we can use ZRC methods in Client Script to get inputs from a Salesperson and update the Lead status with a single button click. In this

• Kaizen #222 - Client Script Support for Notes Related List

  Hello everyone! Welcome to another week of Kaizen. The final Kaizen post of the year 2025 is here! With the new Client Script support for the Notes Related List, you can validate, enrich, and manage notes across modules. In this post, we’ll explore how

• Kaizen #217 - Actions APIs : Tasks

  Welcome to another week of Kaizen! In last week's post we discussed Email Notifications APIs which act as the link between your Workflow automations and you. We have discussed how Zylker Cloud Services uses Email Notifications API in their custom dashboard.

• Kaizen #216 - Actions APIs : Email Notifications

  Welcome to another week of Kaizen! For the last three weeks, we have been discussing Zylker's workflows. We successfully updated a dormant workflow, built a new one from the ground up and more. But our work is not finished—these automated processes are

• Recent Topics

• Issue with Azure DevOps Integration in Zoho Flow

  Hello, All workflows between Zoho Flow and Azure DevOps have stopped working for several days now. Upon further investigation, it seems that the connection to Azure DevOps is no longer directly supported. Indeed, Microsoft has deprecated the Azure DevOps

• Narrative 16: Simplify with workflows

  Behind the scenes of a successful ticketing system: BTS Series Narrative 16: Simplify with workflows What is a workflow? A workflow organizes business tasks in a defined sequence that makes each step clear to all participants. This ensures work is completed

• Zoho Invoice Customer Login Portal

  Are there any plans for a customer portal to Zoho Invoice, ala Freshbooks?  I would like customers that I invoice to be able to login to review invoices and invoice history.  I have not switched from Freshbooks for this very reason.

• Random Leads are being created

  Hi, Every few days I am getting random leads that are being created with no form interactions at all. The email addresses are from obvious spam, such as 'Easymerchantsx'. When I look at the timeline, there is a Visit record and then a Lead Created record,

• Print a price list or price book

  Hi Community. Am I right in concluding that Zoho has no functionality to print a price list from either Zoho CRM, Zoho Inventory or Zoho Books? I won't get stuck on the fact that Zoho doesn't sync price books between Zoho CRM and Books/Inventory (more

• Bar Chart -- sort X-axis

  Hi! I created a bar chart ("Top 10 Products by Revenue") X-axis: Product Name Y-axis: Revenue Filter: Revenue - Top 10 Here's a picture: http://screencast.com/t/ZTJlZTdkNz The x-axis is sorted alphabetically by Product Name. How can I force it to sort numerically by Revenue?

• ZoHo Flow Custom Function not Processing JSON

  JSON is being escaped and converting all the quotes in the JSON to '&quot;' Here's the Code: void InsertRowInE123EligibitySheet(map input) { rawData = get("data",""); decodedData = rawData.htmlDecode(); data = decodedData.toMap(); sheet_id = "ID GOES

• Managing functions

  Can someone let me know if there are any plans to improve the features for managing functions in CRM? I have lots of functions and finding them is hard. The search only works on the function name and the filter only works on function type. I have created

• Zoho FSM - Service Appointment Trouble Adding Field Agent

  Hello, I just started using Zoho FSM and I'm currently adding older work orders from prior Field Service application I use. The work order that im trying to add is from an earlier date, and when I try to add a field agent to a service appointment it does

• 年内最後のユーザー向けイベント：5名限定！ 課題解決型ワークショップイベント Zoho ワークアウト開催のお知らせ (12/18)

  ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 12月開催のZoho ワークアウトについてお知らせします。 今回はZoomにて、オンライン開催します。 参加登録はこちら（無料） https://us02web.zoom.us/meeting/register/QHn6kJAcRs-znJ1l5jk0ww ━━━━━━━━━━━━━━━━━━━━━━━━ Zoho ワークアウトとは？ Zoho ユーザー同士で交流しながら、サービスに関する疑問や不明点の解消を目的とした「Zoho ワークアウト」を開催します。

• How do I get my account id?

  Hello, I followed the instructions to get a list of accounts of the currently authenticated user (which is me, and I am logged in). But when I follow the below instructions I get the following error: ERROR: {"data":{"errorCode":"INVALID_TICKET","moreInfo":"Invalid ticket"},"status":{"code":400,"description":"Invalid Input"}} Instructions that I am following: GET - User account details Purpose The API retrieves the list of accounts of the currently authenticated user.  Request URL  http://mail.zoho.com/api/accounts

• Zoho Quartz Screen Recording

  Hello, can we get access to Quartz, please, as a standalone solution? It would be great for creating training videos for current and future staff on how to use Zoho software according to our company requirements. Thank you

• auto close automated alert tickets which are similar

  Hello ZOHO Community, we are using ZOHO Desk to process automated monitoring alerts. Scenario: Our monitoring system creates a ticket when a threshold is exceeded, e.g. Subject: Computer 1 – CPU usage 100% – Error A few minutes later, once the issue resolves

• Maintain knowledge base integrity by moderating article comments

  Hello everyone, A knowledge base provides a self-service platform where customers can refer to articles, user manuals, and other resources to learn about the company's products or services and troubleshoot problems. Often, readers leave a comment on the

• Making another calendar your default calendar

  I am trying to make another calendar my default calendar when I add events to it. It keep going to a single calendar, I need it to go to my google calendar by default, as this is linked to other services / websites. I cannot find an option to make it

• Option to Delete Chats in IM

  Currently, there is no option to delete any chats in IM, regardless of their source.

• Referencing a cell from another sheet

  My workbook has multiple sheets. Each sheet has some calcluated totals in certain cells. The front master sheet has a list of everything that is detailed on the other sheets, with the totals. These could change at any time, so the totals need to be references to the other cell's value, not a fixed number. So on the master sheet, I put in =, then go the other sheet and choose the cell and hit Enter. In regular Excel, this works. But in the Zoho sheet, it doesn't work. I have to edit the result by

• Group mail for external email addresses

  Hello, I was just wondering if the Group mail feature works with external email addresses - e.g. gmail.com or a completely different domain? it seems only internal addresses (hosted with Zoho) receive the mail. Thanks, Oliver

• The email address you have entered belongs to a different deployment/region.

  Hi, I am trying to create the user - mprust@crombiecomputers.co.uk but keep getting the message below -  The email address you have entered belongs to a different deployment/region. Please contact support@zohoaccounts.com for assistance. Look forward

• Use Zoho Flow Credits for CRM ‘Actions by Zoho Flow’

  Hello Team, We would like to submit a feature request regarding credit usage for “Actions by Zoho Flow” in Zoho CRM. Use Case: We are Zoho One users and actively use Zoho Flow, where our organization has 52,000 Flow tasks per month. In Zoho CRM, we use

• Unusual activity detected from this IP. Please try again after some time.

  Hello Zoho admin and IT team We are a registered website in Eloctronic services and we been trying to add our users to the zoho system but this issue faced us ,, hope you unlocked us please.

• Alert if a field is ticked.

  Hi There, We have two modules named Opportunities (Deals) and End Users (CustomModule1), as per the image below. Within Opportunities, we have a lookup field that looks up from the End Users Module. We are looking to get an alert either via email or another

• Zoho CRM Analytics - Allow To Reorder Dashboards

  I would like to suggest that you add the ability to reorder dashboards in the Analytics Module. I can see that this has been requested some time ago, the latest 9 years ago. I am not sure if this is a big or small endeavor, but such a small fix can go

• Sending a Template to Sign

  hi, trying to send a template to be signed using this as a test: $accessToken = "1000.xxx" $templateId = "1234" $uri = "https://sign.zoho.eu/api/v1/templates/$templateId/createdocument" $payload = @{ templates = @( @{ template_id = $templateId request_name

• Adding Choices in a Sub-Form Dropdown

  Hi, Has anybody tried Adding Choices to a Dropdown in a Zoho Creator Sub-Form programmatically? My Deluge code adds rows to a subform with 2 fields A and B. A - text field. B - dropdown. My Deluge script adds the row and displays A successfully. For the

• Zoho CRM Email Templates 100% Width No Background How?

  Hi, On the Zoho CRM Email Templates in setup > customization > templates > new templates > I choose blank template, but still it puts in a gray background and a max width for the email. I just want to make an email that looks like an email I would send from gmail that has no background or max width. How do you do this? 

• Checking client unsubscribe details

  Hi team, Can you please let me know where we can check if a client has unsubscribed, along with the date and time it was done? If this information is not available at our end, please help confirm the unsubscribe date for the below email ID from the backend:

• Scan and Fill CRM Lookup Field

  Not sure if there is a reason why this isn't possible or if I'm just missing it. But I would like to be able to use the scan and fill feature on the mobile app to prefill the CRM lookup field and fetch the rest of the data in the form.

• Customer Management: #2 Organize Customers to Enhance Efficiency

  When Ankit started his digital services firm, things felt simple. A client would call, ask for a website or a one-time consultation, Ankit would send an invoice, get paid, and move on. "Just one client, one invoice. Easy.", he thought. Fast forward a

• Zoho Mail and Zoho Flow integration to automatically create ToDo tasks from outbound emails

  How do i setup Zoho Mail and Zoho Flow integration to automatically create ToDo tasks from outbound emails

• Attachments between Zoho and Clickup, using Flow.

  Olá suporte Flow, tudo bem ? Estamos usando o flow para integrar Zoho Desk com o clickup. Não localizamos a opção de integrar anexos entre do zoho Desk para o clickup. Gostaríamos de saber se migrando para o plano pago, teremos suporte para fazer a integração

• Adding an Account on Zoho Mail Trigger in Zoho Flow

  I'm trying to create a flow using the zoho mail trigger "Email Receive". My problem is that when I select this trigger, it only shows one account from the account dropdown. I'm planning to assign it on a different email. How can I add other email ad

• Linnworks

  Unless I am missing something, the Linnworks integration is very basic and limited. I have reached out to support but the first response was completely useless and trying to get a reply in a timely manner is very difficult. Surely I should be able to

• Test data won't load

  I am using a Flow to receive orders from WooCommerce and add them to a Zoho Creator app. I recently received an order which failed, and when attempting to test the order I found that it just shows a loading animation and shows up in the history as "queued."

• AddHour resets the time to 00:00:00 before adding the hour.

  Based on the documentation here: https://www.zoho.com/deluge/help/functions/datetime/addhour.html Here's my custom function: string ConvertDateFormat(string inputDate) { // Extract only the date-time part (before the timezone) dateTimePart = inputDate.subString(0,19);

• WhatsApp Link is not integrating

  Hello, I am using zoho flow. when new row added in google sheet it sends email to respected person. In email body I have a text "Share via WhatsApp". behind this text I putted a link. But when the recipient receives email and wants to share my given info

• Zoho flow - Webhook

  If I choose an app as a trigger in Zoho Flow, is it still possible to add a webhook later in the same flow?

• Zoho Flow + Bigin + Shopify

  We are testing Zoho Flow for the first time and want to create a flow based in first purchases. When a client makes his first order, we're going to add the "primeiracompra" (first order) tag to his account in Shopify (it's not efficient, but that's the

• Is it The Flow? Or is it me?

  I want to do some basic level stuff, take two fields from a webhook, create a zsheet from a template using one field with date appended, create a folder using both fields as the name, and put the zsheet into that folder. I was going to elaborate - but

• Having problem with data transferring from Google sheet to ZMA

  When connecting Google sheet with Zoho marketing automation it is having the email as a mandatory field. Can I change it as non-mandatory field or is there any other way to trasnfer data from google sheet to ZMA. I have leads which we get from whatsapp,

• Next Page