How to dedupe or remove duplicate records in your data?

Deduplicate




You can remove duplicate records from your data using the Deduplicate transform. This can be done in two ways: row-wise and column-wise.

Row-wise

This method removes rows with duplicate data, allowing only unique rows to be present in your dataset.

To apply row-wise deduplication:

1. Click the Transform menu, click Deduplicate, then select Row-wise

2. You can choose to ignore case and whitespace while removing duplicate rows.

Notes
Note: There could be instances where no duplicates were found in the sample dataset. You can still apply the rule to remove duplicates rows when the entire dataset is processed during export.

3. A live preview will be shown with the duplicate rows highlighted in red.



4. Click Remove duplicates

Column-wise

You can also select single or multiple columns and choose to dedupe. You can use Deduplicate > Column-wise transform to remove rows based on duplicate values present in the selected columns. 

In other words, select all those columns (For example, Region, Address, Product) which has the same entries vertically. The column-wise deduplication will remove those rows which has the same entry vertically in the columns selected.

To apply column-wise deduplication:

1. Click the Transform menu, click Deduplicate then select Column-wise

2. You can choose to ignore case and whitespace to find duplicates. 

3. You can choose one of the two methods to dedupe your dataset based on the selected column: Automatic deduplication or Manual conditions

4. When you choose the Automatic deduplication method, DataPrep works for you to deduplicate your data based on the columns you've selected. 



5. When you choose the Manual conditions method, you will need to enter the conditions and expressions and construct the 'if' statements. You can then select which rows to keep, or remove, within each of the duplicate cluster if the condition is true.

 

6. The following table lists the available If conditions for all the data types. Click here to know more about data types.

Text

Numeric

Datetime

Duration

Boolean

List

Map

 contains

 =equal to

 = equal to

 is smallest

 is true

 has value

 has key

 doesn't contain

 != not equal to

 != not equal to

 is largest

 is false

 is empty list

 is empty map

 begins with

 > more than

 is earliest

 = equal to

 contains

 is not empty list

 is not empty   map

 ends with

 < less than

 is latest

 != not equal to

 doesn't contain

 is empty

 is empty

 is

 >= more than or   equal

 is after

 is empty

 begins with

 is not empty

 is not empty

 is not

 <= less than or   equal to

 is before

 is not empty

 ends with

 use regex

 use regex

 is empty

 is smallest

 on or after

 use regex

 is

 use patterns

 use patterns

 is not empty

 is largest

 on or before

 use patterns

 is not

 

 

 use regex

 is empty

 is empty

 

 is empty

 

 

 use patterns

 is not empty

 is not empty

 

 is not empty

 

 

 

 use regex

 use regex

 

 use regex

 

 

 

 use patterns

 use patterns

 

 use patterns

 

 


7. You can also keep adding more conditions using the AND and OR operators to apply deduplication using a combination of conditions.

Idea
For example, you can write a condition that goes like this, "If the mail column contains zoho.com, keep those rows", i.e., Enter conditions to select which rows to keep if mail contains zoho.com

8. With the Advanced option, you can insert functions and provide conditions to remove duplicates. 



9. Click the Preview button to see which rows will be removed during the transformation.

10. You can also select multiple columns for deduplication using (+) in Columns to de-duplicate. 

      Create. Review. Publish.

      Write, edit, collaborate on, and publish documents to different content management platforms.

      Get Started Now


        Access your files securely from anywhere

          Zoho CRM Training Programs

          Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

          Zoho CRM Training
            Redefine the way you work
            with Zoho Workplace

              Zoho DataPrep Personalized Demo

              If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

              Zoho CRM Training

                Create, share, and deliver

                beautiful slides from anywhere.

                Get Started Now


                  Zoho Sign now offers specialized one-on-one training for both administrators and developers.

                  BOOK A SESSION







                              Quick LinksWorkflow AutomationData Collection
                              Web FormsEnterpriseOnline Data Collection Tool
                              Embeddable FormsBankingBegin Data Collection
                              Interactive FormsWorkplaceData Collection App
                              CRM FormsCustomer ServiceAccessible Forms
                              Digital FormsMarketingForms for Small Business
                              HTML FormsEducationForms for Enterprise
                              Contact FormsE-commerceForms for any business
                              Lead Generation FormsHealthcareForms for Startups
                              Wordpress FormsCustomer onboardingForms for Small Business
                              No Code FormsConstructionRSVP tool for holidays
                              Free FormsTravelFeatures for Order Forms
                              Prefill FormsNon-Profit
                              Intake FormsLegalMobile App
                              Form DesignerHRMobile Forms
                              Card FormsFoodOffline Forms
                              Assign FormsPhotographyMobile Forms Features
                              Translate FormsReal EstateKiosk in Mobile Forms
                              Electronic Forms
                              Drag & drop form builder

                              Notification Emails for FormsAlternativesSecurity & Compliance
                              Holiday FormsGoogle Forms alternative GDPR
                              Form to PDFJotform alternativeHIPAA Forms
                              Email FormsEncrypted Forms

                              Secure Forms

                              WCAG

                                      Create. Review. Publish.

                                      Write, edit, collaborate on, and publish documents to different content management platforms.

                                      Get Started Now






                                                        You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                                            Manage your brands on social media


                                                              • Desk Community Learning Series


                                                              • Digest


                                                              • Functions


                                                              • Meetups


                                                              • Kbase


                                                              • Resources


                                                              • Glossary


                                                              • Desk Marketplace


                                                              • MVP Corner


                                                              • Word of the Day


                                                              • Ask the Experts


                                                                Zoho Sheet Resources

                                                                 

                                                                    Zoho Forms Resources


                                                                      Secure your business
                                                                      communication with Zoho Mail


                                                                      Mail on the move with
                                                                      Zoho Mail mobile application

                                                                        Stay on top of your schedule
                                                                        at all times


                                                                        Carry your calendar with you
                                                                        Anytime, anywhere




                                                                              Zoho Sign Resources

                                                                                Sign, Paperless!

                                                                                Sign and send business documents on the go!

                                                                                Get Started Now




                                                                                        Zoho TeamInbox Resources





                                                                                                  Zoho DataPrep Demo

                                                                                                  Get a personalized demo or POC

                                                                                                  REGISTER NOW


                                                                                                    Design. Discuss. Deliver.

                                                                                                    Create visually engaging stories with Zoho Show.

                                                                                                    Get Started Now









                                                                                                                        • Related Articles

                                                                                                                        • Dataset transforms

                                                                                                                          Dataset transform consists of combining and transforming datasets to suit your needs. You can apply the following transforms in a dataset. Deduplicate Row-wise Column-wise Join Append Pivot Unpivot Calculated columns Derive dataset Mark as personal ...
                                                                                                                        • Derive dataset

                                                                                                                          The Derive dataset transform allows you to create a branch of your dataset from the last applied transform in your dataset. The new dataset will have the most recent state of your data with an empty ruleset. To derive dataset 1. In the DataPrep ...
                                                                                                                        • Filter

                                                                                                                          The filter option allows you to filter data based on custom conditions applied over one or more columns. The filter can also be used without applying them as a rule. However, you can choose to apply them as a rule by either keeping or deleting the ...
                                                                                                                        • Append

                                                                                                                          In DataPrep, you can append one dataset with another and create a new dataset using the Append transform. Here's a quick video on how to use the Append transform. To append datasets 1. In the DataPrep Studio page, click the Transform menu, click ...
                                                                                                                        • Mark PII and ePHI data

                                                                                                                          You can mark a column that contains PII(Personal Identifiable Information) or personal data and ePHI (Electronic Protected Health Information ) data using the Mark PII and ePHI data transform. You can also apply security methods on the columns to ...
                                                                                                                          Wherever you are is as good as
                                                                                                                          your workplace

                                                                                                                            Resources

                                                                                                                            Videos

                                                                                                                            Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                                            eBooks

                                                                                                                            Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                                            Webinars

                                                                                                                            Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                                            CRM Tips

                                                                                                                            Make the most of Zoho CRM with these useful tips.



                                                                                                                              Zoho Show Resources