How to dedupe or remove duplicate records in your data?

Deduplicate




You can remove duplicate records from your data using the Deduplicate transform. This can be done in two ways: row-wise and column-wise.

Row-wise

This method removes rows with duplicate data, allowing only unique rows to be present in your dataset.

To apply row-wise deduplication:

1. Click the Transform menu, click Deduplicate, then select Row-wise

2. You can choose to ignore case and whitespace while removing duplicate rows.

Note: There could be instances where no duplicates were found in the sample dataset. You can still apply the rule to remove duplicates rows when the entire dataset is processed during export.

3. A live preview will be shown with the duplicate rows highlighted in red.



4. Click Remove duplicates

Column-wise

You can also select single or multiple columns and choose to dedupe. You can use Deduplicate > Column-wise transform to remove rows based on duplicate values present in the selected columns. 

In other words, select all those columns (For example, Region, Address, Product) which has the same entries vertically. The column-wise deduplication will remove those rows which has the same entry vertically in the columns selected.

To apply column-wise deduplication:

1. Click the Transform menu, click Deduplicate then select Column-wise

2. You can choose to ignore case and whitespace to find duplicates. 

3. You can choose one of the two methods to dedupe your dataset based on the selected column: Automatic deduplication or Manual conditions

4. When you choose the Automatic deduplication method, DataPrep works for you to dedupe your data based on the columns you've selected. 



5. When you choose the Manual conditions method, you will need to enter the conditions and expressions and construct the 'if' statements. You can then select which rows to keep, or remove, within each of the duplicate cluster if the condition is true.



6. The following table lists the available If conditions for all the data types. Click here to know more about data types.

Text

Numeric

Datetime

Duration

Boolean

List

Map

 contains

 =equal to

 = equal to

 is smallest

 is true

 has value

 has key

 doesn't contain

 != not equal to

 != not equal to

 is largest

 is false

 is empty list

 is empty map

 begins with

 > more than

 is earliest

 = equal to

 contains

 is not empty list

 is not empty   map

 ends with

 < less than

 is latest

 != not equal to

 doesn't contain

 is cell empty

 is cell empty

 is

 >= more than or   equal

 is after

 is cell empty

 begins with

 is cell not empty

 is cell not empty

 is not

 <= less than or   equal to

 is before

 is cell not empty

 ends with

 use regex

 use regex

 is cell empty

 is smallest

 on or after

 use regex

 is

 use patterns

 use patterns

 is cell not empty

 is largest

 on or before

 use patterns

 is not

 

 

 use regex

 is cell empty

 is cell empty

 

 is cell empty

 

 

 use patterns

 is cell not empty

 is cell not empty

 

 is cell not empty

 

 

 

 use regex

 use regex

 

 use regex

 

 

 

 use patterns

 use patterns

 

 use patterns

 

 


7. You can also keep adding more conditions using the AND and OR operators to apply deduplication using a combination of conditions.

For example, you can write a condition that goes like this, "If the mail column contains zoho.com, keep those rows", i.e., Enter conditions to select which 
If mail contains zoho.com

8. With the Advanced option, you can insert functions and provide conditions to remove duplicates.



9. Click the Preview button to see which rows will be removed during the transformation.

10. You can also select multiple columns for deduplication using (+) in Columns to de-duplicate. 


    Zoho CRM Training Programs

    Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

    Zoho CRM Training
      Redefine the way you work
      with Zoho Workplace

        Zoho DataPrep Personalized Demo

        If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

        Zoho CRM Training

          Create, share, and deliver

          beautiful slides from anywhere.

          Get Started Now


            Zoho Sign now offers specialized one-on-one training for both administrators and developers.

            BOOK A SESSION








                                You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                    Manage your brands on social media

                                      Zoho Desk Resources

                                      • Desk Community Learning Series


                                      • Digest


                                      • Functions


                                      • Meetups


                                      • Kbase


                                      • Resources


                                      • Glossary


                                      • Desk Marketplace


                                      • MVP Corner


                                      • Word of the Day


                                        Zoho Marketing Automation

                                          Zoho Sheet Resources

                                           

                                              Zoho Forms Resources


                                                Secure your business
                                                communication with Zoho Mail


                                                Mail on the move with
                                                Zoho Mail mobile application

                                                  Stay on top of your schedule
                                                  at all times


                                                  Carry your calendar with you
                                                  Anytime, anywhere




                                                        Zoho Sign Resources

                                                          Sign, Paperless!

                                                          Sign and send business documents on the go!

                                                          Get Started Now




                                                                  Zoho TeamInbox Resources



                                                                          Zoho DataPrep Resources



                                                                            Zoho DataPrep Demo

                                                                            Get a personalized demo or POC

                                                                            REGISTER NOW


                                                                              Design. Discuss. Deliver.

                                                                              Create visually engaging stories with Zoho Show.

                                                                              Get Started Now







                                                                                            You are currently viewing the help articles of Sprints 1.0. If you are a user of 2.0, please refer here.

                                                                                            You are currently viewing the help articles of Sprints 2.0. If you are a user of 1.0, please refer here.



                                                                                                  • Related Articles

                                                                                                  • Dataset transforms

                                                                                                    Dataset transform consists of combining and transforming datasets to suit your needs. You can apply the following transforms in a dataset. Deduplicate Row-wise Column-wise Join Append Pivot Unpivot Calculated columns Derive dataset Mark as personal ...
                                                                                                  • Derive dataset

                                                                                                    The Derive dataset transform allows you to create a branch of your dataset from the last applied transform in your dataset. The new dataset will have the most recent state of your data with an empty ruleset. To derive dataset 1. Click the Transform ...
                                                                                                  • Mark PII and ePHI data

                                                                                                    You can mark a column that contains PII(Personal Identifiable Information) or personal data and ePHI (Electronic Protected Health Information ) data using the Mark PII and ePHI data transform. You can also apply security methods on the columns to ...
                                                                                                  • Target Matching

                                                                                                    Target matching allows you to set a target and align the source dataset to match with your target before exporting data. You can attach a target dataset to match the column structure, formats, and the data types in the existing source dataset. This ...
                                                                                                  • Append

                                                                                                    In DataPrep, you can append one dataset with another and create a new dataset using the Append transform. Here's a quick video on how to use the Append transform. To append datasets 1. Click the Transform menu, click Combine , then select Append . 2. ...
                                                                                                    Wherever you are is as good as
                                                                                                    your workplace

                                                                                                      Resources

                                                                                                      Videos

                                                                                                      Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                      eBooks

                                                                                                      Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                      Webinars

                                                                                                      Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                      CRM Tips

                                                                                                      Make the most of Zoho CRM with these useful tips.



                                                                                                        Zoho Show Resources