How to dedupe or remove duplicate records in your data?

Deduplicate




You can remove duplicate records from your data using the Deduplicate transform. This can be done in two ways: row-wise and column-wise.

Row-wise

This method removes rows with duplicate data, allowing only unique rows to be present in your dataset.

To apply row-wise deduplication:

1. Click the Transform menu, click Deduplicate, then select Row-wise

2. You can choose to ignore case and whitespace while removing duplicate rows.

Note: There could be instances where no duplicates were found in the sample dataset. You can still apply the rule to remove duplicates rows when the entire dataset is processed during export.

3. A live preview will be shown with the duplicate rows highlighted in red.



4. Click Remove duplicates

Column-wise

You can also select single or multiple columns and choose to dedupe. You can use Deduplicate > Column-wise transform to remove rows based on duplicate values present in the selected columns. 

In other words, select all those columns (For example, Region, Address, Product) which has the same entries vertically. The column-wise deduplication will remove those rows which has the same entry vertically in the columns selected.

To apply column-wise deduplication:

1. Click the Transform menu, click Deduplicate then select Column-wise

2. You can choose to ignore case and whitespace to find duplicates. 

3. You can choose one of the two methods to dedupe your dataset based on the selected column: Automatic deduplication or Manual conditions

4. When you choose the Automatic deduplication method, DataPrep works for you to deduplicate your data based on the columns you've selected. 



5. When you choose the Manual conditions method, you will need to enter the conditions and expressions and construct the 'if' statements. You can then select which rows to keep, or remove, within each of the duplicate cluster if the condition is true.

 

6. The following table lists the available If conditions for all the data types. Click here to know more about data types.

Text

Numeric

Datetime

Duration

Boolean

List

Map

 contains

 =equal to

 = equal to

 is smallest

 is true

 has value

 has key

 doesn't contain

 != not equal to

 != not equal to

 is largest

 is false

 is empty list

 is empty map

 begins with

 > more than

 is earliest

 = equal to

 contains

 is not empty list

 is not empty   map

 ends with

 < less than

 is latest

 != not equal to

 doesn't contain

 is cell empty

 is cell empty

 is

 >= more than or   equal

 is after

 is cell empty

 begins with

 is cell not empty

 is cell not empty

 is not

 <= less than or   equal to

 is before

 is cell not empty

 ends with

 use regex

 use regex

 is cell empty

 is smallest

 on or after

 use regex

 is

 use patterns

 use patterns

 is cell not empty

 is largest

 on or before

 use patterns

 is not

 

 

 use regex

 is cell empty

 is cell empty

 

 is cell empty

 

 

 use patterns

 is cell not empty

 is cell not empty

 

 is cell not empty

 

 

 

 use regex

 use regex

 

 use regex

 

 

 

 use patterns

 use patterns

 

 use patterns

 

 


7. You can also keep adding more conditions using the AND and OR operators to apply deduplication using a combination of conditions.

For example, you can write a condition that goes like this, "If the mail column contains zoho.com, keep those rows", i.e., Enter conditions to select which 
If mail contains zoho.com

8. With the Advanced option, you can insert functions and provide conditions to remove duplicates. 



9. Click the Preview button to see which rows will be removed during the transformation.

10. You can also select multiple columns for deduplication using (+) in Columns to de-duplicate. 

      Create. Review. Publish.

      Write, edit, collaborate on, and publish documents to different content management platforms.

      Get Started Now


        Access your files securely from anywhere

          Zoho CRM Training Programs

          Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

          Zoho CRM Training
            Redefine the way you work
            with Zoho Workplace

              Zoho DataPrep Personalized Demo

              If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

              Zoho CRM Training

                Create, share, and deliver

                beautiful slides from anywhere.

                Get Started Now


                  Zoho Sign now offers specialized one-on-one training for both administrators and developers.

                  BOOK A SESSION





                              Quick Links Workflow Automation Data Collection
                              Web Forms Enterprise Begin Data Collection
                              Interactive Forms Workplace Data Collection App
                              CRM Forms Customer Service Accessible Forms
                              Digital Forms Marketing Forms for Small Business
                              HTML Forms Education Forms for Enterprise
                              Contact Forms E-commerce Forms for any business
                              Lead Generation Forms Healthcare Forms for Startups
                              Wordpress Forms Customer onboarding Order Forms for Small Business
                              No Code Forms Construction RSVP tool for holidays
                              Free Forms Travel
                              Prefill Forms Non-Profit

                              Intake Forms Legal
                              Mobile App
                              Form Designer HR
                              Mobile Forms
                              Card Forms Food Offline Forms
                              Assign Forms Photography
                              Mobile Forms Features
                              Translate Forms Real Estate Kiosk in Mobile Forms
                              Electronic Forms

                              Notification Emails for Forms Alternatives Security & Compliance
                              Holiday Forms Google Forms alternative  GDPR
                              Form to PDF Jotform alternative HIPAA Forms
                              Email Forms
                              Encrypted Forms
                              Embeddable Forms
                              Secure Forms
                              Drag & drop form builder
                              WCAG

                                    Create. Review. Publish.

                                    Write, edit, collaborate on, and publish documents to different content management platforms.

                                    Get Started Now




                                                      You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                                          Manage your brands on social media


                                                            • Desk Community Learning Series


                                                            • Digest


                                                            • Functions


                                                            • Meetups


                                                            • Kbase


                                                            • Resources


                                                            • Glossary


                                                            • Desk Marketplace


                                                            • MVP Corner


                                                            • Word of the Day


                                                            • Ask the Experts


                                                              Zoho Sheet Resources

                                                               

                                                                  Zoho Forms Resources


                                                                    Secure your business
                                                                    communication with Zoho Mail


                                                                    Mail on the move with
                                                                    Zoho Mail mobile application

                                                                      Stay on top of your schedule
                                                                      at all times


                                                                      Carry your calendar with you
                                                                      Anytime, anywhere




                                                                            Zoho Sign Resources

                                                                              Sign, Paperless!

                                                                              Sign and send business documents on the go!

                                                                              Get Started Now




                                                                                      Zoho TeamInbox Resources





                                                                                                Zoho DataPrep Demo

                                                                                                Get a personalized demo or POC

                                                                                                REGISTER NOW


                                                                                                  Design. Discuss. Deliver.

                                                                                                  Create visually engaging stories with Zoho Show.

                                                                                                  Get Started Now









                                                                                                                      • Related Articles

                                                                                                                      • Dataset transforms

                                                                                                                        Dataset transform consists of combining and transforming datasets to suit your needs. You can apply the following transforms in a dataset. Deduplicate Row-wise Column-wise Join Append Pivot Unpivot Calculated columns Derive dataset Mark as personal ...
                                                                                                                      • Derive dataset

                                                                                                                        The Derive dataset transform allows you to create a branch of your dataset from the last applied transform in your dataset. The new dataset will have the most recent state of your data with an empty ruleset. To derive dataset 1. In the DataPrep ...
                                                                                                                      • Filter

                                                                                                                        The filter option allows you to filter data based on custom conditions applied over one or more columns. The filter can also be used without applying them as a rule. However, you can choose to apply them as a rule by either keeping or deleting the ...
                                                                                                                      • Mark PII and ePHI data

                                                                                                                        You can mark a column that contains PII(Personal Identifiable Information) or personal data and ePHI (Electronic Protected Health Information ) data using the Mark PII and ePHI data transform. You can also apply security methods on the columns to ...
                                                                                                                      • Window functions to add formula columns

                                                                                                                        A window function is used to perform calculations across a group of rows on a dataset. A set of rows is termed a window. You can use window functions to perform summations and calculations based on a rolling window of data, relative to the current ...
                                                                                                                        Wherever you are is as good as
                                                                                                                        your workplace

                                                                                                                          Resources

                                                                                                                          Videos

                                                                                                                          Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                                          eBooks

                                                                                                                          Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                                          Webinars

                                                                                                                          Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                                          CRM Tips

                                                                                                                          Make the most of Zoho CRM with these useful tips.



                                                                                                                            Zoho Show Resources