Deduplicate

Deduplicate





You can remove duplicate records from your data using the Deduplicate transform. This can be done in two ways: row-wise and column-wise.

Row-wise

This method removes rows with duplicate data, allowing only unique rows to be present in your dataset.

To apply row-wise deduplication:

1. Click the Transform menu, click Deduplicate, then select Row-wise

2. You can choose to ignore case and whitespace while removing duplicate rows.

Notes
Note: There could be instances where no duplicates were found in the sample dataset. You can still apply the rule to remove duplicates rows when the entire dataset is processed during export.

3. A live preview will be shown with the duplicate rows highlighted in red.



4. Click Remove duplicates

Column-wise

You can also select single or multiple columns and choose to dedupe. You can use Deduplicate > Column-wise transform to remove rows based on duplicate values present in the selected columns. 

In other words, select all those columns (For example, Region, Address, Product) which has the same entries vertically. The column-wise deduplication will remove those rows which has the same entry vertically in the columns selected.

To apply column-wise deduplication:

1. Click the Transform menu, click Deduplicate then select Column-wise

2. You can choose to ignore case and whitespace to find duplicates. 

3. You can choose one of the two methods to dedupe your dataset based on the selected column: Automatic deduplication or Manual conditions

4. When you choose the Automatic deduplication method, DataPrep works for you to dedupe your data based on the columns you've selected. 



5. When you choose the Manual conditions method, you will need to enter the conditions and expressions and construct the 'if' statements. You can then select which rows to keep, or remove, within each of the duplicate cluster if the condition is true.



6. The following table lists the available If conditions for all the data types. Click here to know more about data types.

Text

Numeric

Datetime

Duration

Boolean

List

Map

 contains

 =equal to

 = equal to

 is smallest

 is true

 has value

 has key

 doesn't contain

 != not equal to

 != not equal to

 is largest

 is false

 is empty list

 is empty map

 begins with

 > more than

 is earliest

 = equal to

 contains

 is not empty list

 is not empty   map

 ends with

 < less than

 is latest

 != not equal to

 doesn't contain

 is cell empty

 is cell empty

 is

 >= more than or   equal

 is after

 is cell empty

 begins with

 is cell not empty

 is cell not empty

 is not

 <= less than or   equal to

 is before

 is cell not empty

 ends with

 use regex

 use regex

 is cell empty

 is smallest

 on or after

 use regex

 is

 use patterns

 use patterns

 is cell not empty

 is largest

 on or before

 use patterns

 is not

 

 

 use regex

 is cell empty

 is cell empty

 

 is cell empty

 

 

 use patterns

 is cell not empty

 is cell not empty

 

 is cell not empty

 

 

 

 use regex

 use regex

 

 use regex

 

 

 

 use patterns

 use patterns

 

 use patterns

 

 


7. You can also keep adding more conditions using the AND and OR operators to apply deduplication using a combination of conditions.

Idea
For example, you can write a condition that goes like this, "If the mail column contains zoho.com, keep those rows", i.e., Enter conditions to select which 
If mail contains zoho.com

8. With the Advanced option, you can insert functions and provide conditions to remove duplicates.



9. Click the Preview button to see which rows will be removed during the transformation.

10. You can also select multiple columns for deduplication using (+) in Columns to de-duplicate. 


      Create. Review. Publish.

      Write, edit, collaborate on, and publish documents to different content management platforms.

      Get Started Now


        Access your files securely from anywhere

          Zoho CRM Training Programs

          Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

          Zoho CRM Training
            Redefine the way you work
            with Zoho Workplace

              Zoho DataPrep Personalized Demo

              If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

              Zoho CRM Training

                Create, share, and deliver

                beautiful slides from anywhere.

                Get Started Now


                  Zoho Sign now offers specialized one-on-one training for both administrators and developers.

                  BOOK A SESSION







                              Quick LinksWorkflow AutomationData Collection
                              Web FormsEnterpriseOnline Data Collection Tool
                              Embeddable FormsBankingBegin Data Collection
                              Interactive FormsWorkplaceData Collection App
                              CRM FormsCustomer ServiceAccessible Forms
                              Digital FormsMarketingForms for Small Business
                              HTML FormsEducationForms for Enterprise
                              Contact FormsE-commerceForms for any business
                              Lead Generation FormsHealthcareForms for Startups
                              Wordpress FormsCustomer onboardingForms for Small Business
                              No Code FormsConstructionRSVP tool for holidays
                              Free FormsTravelFeatures for Order Forms
                              Prefill FormsNon-Profit

                              Intake FormsLegal
                              Mobile App
                              Form DesignerHR
                              Mobile Forms
                              Card FormsFoodOffline Forms
                              Assign FormsPhotographyMobile Forms Features
                              Translate FormsReal EstateKiosk in Mobile Forms
                              Electronic Forms
                              Drag & drop form builder

                              Notification Emails for FormsAlternativesSecurity & Compliance
                              Holiday FormsGoogle Forms alternative GDPR
                              Form to PDFJotform alternativeHIPAA Forms
                              Email FormsFormstack alternativeEncrypted Forms

                              Wufoo alternativeSecure Forms

                              WCAG



                                        Create. Review. Publish.

                                        Write, edit, collaborate on, and publish documents to different content management platforms.

                                        Get Started Now







                                                          You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                                              Manage your brands on social media


                                                                • Desk Community Learning Series


                                                                • Digest


                                                                • Functions


                                                                • Meetups


                                                                • Kbase


                                                                • Resources


                                                                • Glossary


                                                                • Desk Marketplace


                                                                • MVP Corner


                                                                • Word of the Day


                                                                • Ask the Experts


                                                                  Zoho Sheet Resources

                                                                   

                                                                      Zoho Forms Resources


                                                                        Secure your business
                                                                        communication with Zoho Mail


                                                                        Mail on the move with
                                                                        Zoho Mail mobile application

                                                                          Stay on top of your schedule
                                                                          at all times


                                                                          Carry your calendar with you
                                                                          Anytime, anywhere




                                                                                Zoho Sign Resources

                                                                                  Sign, Paperless!

                                                                                  Sign and send business documents on the go!

                                                                                  Get Started Now




                                                                                          Zoho TeamInbox Resources





                                                                                                    Zoho DataPrep Demo

                                                                                                    Get a personalized demo or POC

                                                                                                    REGISTER NOW


                                                                                                      Design. Discuss. Deliver.

                                                                                                      Create visually engaging stories with Zoho Show.

                                                                                                      Get Started Now









                                                                                                                          • Related Articles

                                                                                                                          • Window functions to add formula columns

                                                                                                                            A window function is used to perform calculations across a group of rows on a dataset. A set of rows is termed a window. You can use window functions to perform summations and calculations based on a rolling window of data, relative to the current ...
                                                                                                                          • Filter

                                                                                                                            The filter option allows you to filter data based on custom conditions applied over one or more columns. The filter can also be used without applying them as a rule. However, you can choose to apply them as a rule by either keeping or deleting the ...
                                                                                                                          • Filter

                                                                                                                            The filter transform allows you to selectively filter data based on filter conditions applied over one or more columns. To apply advanced filters 1. You can choose an Right-click on a column and select the Filter option from the context menu. The ...
                                                                                                                          • Column Explorer

                                                                                                                            The Column Explorer in Zoho DataPrep allows you to search, navigate and control the viewing options of the columns in a dataset. You can search and filter columns with invalid or missing data and fix them quickly. This helps you focus on the columns ...
                                                                                                                          • Context menu

                                                                                                                            Zoho DataPrep offers various options in the context menu to perform column level operations. A few basic options here are Sort, Rename, Delete and Move. To access the context menu, right click the column name and click any one of the icons to perform ...
                                                                                                                            Wherever you are is as good as
                                                                                                                            your workplace

                                                                                                                              Resources

                                                                                                                              Videos

                                                                                                                              Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                                              eBooks

                                                                                                                              Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                                              Webinars

                                                                                                                              Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                                              CRM Tips

                                                                                                                              Make the most of Zoho CRM with these useful tips.



                                                                                                                                Zoho Show Resources