How to manually run pipeline?

Pipeline Manual run

By triggering the manual run option in your pipeline, you can export the data you have prepared to your destination. A manual run will run your pipeline on the existing data without refreshing, which means no new data will be imported.

NotesNote: You can also use this option for dry runs and to test if your pipeline can be executed without any failures before automating the pipeline using a schedule or backfill run. However, please make sure you configure the pipeline for your testing environment, especially destinations as the transforms and export will happen for whichever data destinations are configured.

InfoInfo: A manual run will run on the latest version of the pipeline. A version is nothing but stages of the actions performed on a pipeline. If you want to do a manual run on any old version, you can restore the required version, and perform a manual run. Click here to know more about versions. 

 To execute manual run in a pipeline 

1. Open your pipeline and go to the pipeline builder view. Click here if you want to know how to create a pipeline in the first place.

2. Click on the Add data option to bring your data from the required data source. Learn more on 50+ data sources.

After data is imported, you will be redirected to the pipeline builder where you can see your data source, and a stage linked to it.

Stages are nodes created for processing data while applying data flow transforms. Every dataset imported from your data source will have a stage created by default.

3. You can right-click the stage and apply the data flow transforms.

4. You can also choose to Prepare Data to open the DataPrep Studio page and explore more data transforms. Click here to know the various elements in a pipeline builder.

5. Once you are done creating your data flow and apply necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

6. Data destination is a place where you want to export your data to. It can be your local database, business applications like Zoho CRM, Zoho Analytics, etc. You can choose your preferred destination out of 50+ data destinations in DataPrep to export your prepared data to.  


 

 Manual run 

7. After adding a destination for your pipeline, click the Run drop down and choose Manual run.

 Settings 

  • Stop export data quality drops below 100%: You can use this toggle if you would like to stop export if data quality drops below 100 percent.
  • Order export: You can use this option when you have configured multiple destinations and would like to determine in what order the data has to be exported to destinations.

NotesNote: This option will be visible only if you have added more than one destination in your pipeline.


Data Refresh options

Under the Data refresh options, you can set the configuration for how you would like to fetch new data during the manual run. For example, you can fetch data from a specific date and time.



Use existing data : This option will use the source file data that was imported. This option will be selected by default.

Refresh data from source : You can refresh and import new data from the source based on the import configuration.  

InfoInfo: The below options are applicable for cloud storage services. These import options may vary based on the data source you have selected. Refer to the import configuration tables below to know about other data sources


How to import data from source?  Choose how you would like to import data using this option.

Import all data

All data that matches the specified pattern will be imported.
File batch size: Data will be imported as a single file with the specified number of files in this field. For example, you have 10 files in your source and you provide 10 as the file batch size, all 10 files will be imported as 1 file. 

Notes
Note: The maximum supported value for a File batch size is 10.

Incremental file fetch 

Incremental file fetch- Using this option, you can incrementally fetch data that match the specified pattern. Incremental data fetch is a method used to import new files from a source after the previous sync.

Use the previously imported file if no file is availableWhen there are no new files in the source during incremental import,
If the checkbox is unchecked: The last fetched files will be imported again.
If the checkbox is checked: The import will be skipped and no files will be imported. 

Which file to import?
Choose one of the below options

All files- All files will be imported based on the option selected in the Fetch based on field.
Newest file - The newest file will be imported based on the option selected in the Fetch based on field.
Oldest file- The oldest file will be imported based on the option selected in the Fetch based on field.
Fetch based on - Files will be fetch based on the Created or modified time. For example, if you want the first created file, choose Created Time and choose Oldest file option in the Which file to import? field.
File batch sizeData will be imported as a single file with the specified number of files in this field. For example, you have 10 files in your source and you provide 10 as the file batch size, all 10 files will be imported as 1 file.

Notes
Note: The maximum supported value for a File batch size is 10.

Example for incremental fetch
Let's say you want to import all data based on the created time from Feb 1, 2025 12:00 AM to Feb 4, 2025 12:00 AM. Below is the configuration and data interval snapshot. This will import all data created from Feb 1, 2025 12:00 AM to Feb 4, 2025 12:00 AM

Import configuration


Data interval


Do not import data 

This option will not import any new data. The source file will be used.
Notes
Note: You can look into the tables in the import configuration tables to know when a data interval is applicable.

8. Click on the Run button to execute the manual run. This will start processing the pipeline and the Job summary page opens. The Job summary shows the progress of a job execution. Click here to know more.  You can view the status of the run on the Job summary page. There are three different status for a job in DataPrep - Initiated, Success and Failure.

If the job fails, you can identify if the error is at the import stage, transform stage, or with the destination, and target matching in the Job summary page. You can hover on the failed stage to view the error details and fix them to proceed exporting.

In the Overview tab, you can view the status of the job along with the details such as user who started the job, storage used, total rows processed, start time, end time and duration of the job. Click here to know more.

In the Stages tab, you can view the details of each pipeline stage such as Import, Transform, and Export. Click here to know more.

In the Output tab, you see the list of all exported data. You can also download the output if needed. Click here to know more.


NotesNote: Jobs are listed under each pipeline's menu, the jobs are stored indefinitely and can be referred whenever needed.

9. When the manual run is completed, the data prepared in your pipeline will be exported to the configured destinations.

Import configuration for simple selection in cloud storage

Importing data without using a regex pattern, can be termed as a simple selection. If you import data from a cloud storage service using the simple selection method, refer to the below options:

Data source How to import data from source Data interval applicable or not
Cloud storage Import source file Not applicable
Do not import data Not applicable

Import configuration for advanced selection in cloud storage

Zoho DataPrep allows you to import data using the Advanced selection option. Advanced selection helps you perform dynamic file selection based on a regex pattern. If you import data using the Advanced selection method, refer to the below options:

Data source How to import data from source Data interval applicable or not
Cloud storage Import all data Not applicable
Incremental file fetch
All files
Newest file
Oldest file
Applicable
Do not import data Not applicable

Import configuration for data sources without using Databridge

If you import data from the below sources without using Zoho Databridge, refer to the below options:

Data source How to import data from source Data interval applicable or not
Local files Import source file
Note: You can upload a new source file that you would like to import using the Upload File option. You can use the Parse file as option to parse unknown file formats.
Not applicable
Do not import data Not applicable
URL Import source file Not applicable
Do not import data Not applicable
Local database Import all data Not applicable
Do not import data Not applicable
FTP servers Import source file Not applicable
Do not import data Not applicable

Import configuration for data sources using Databridge

Zoho DataPrep allows you to import data from your local network using Zoho Databridge. If you import data from the below sources using Zoho Databridge, refer to the below options:

Data source How to import data from source? Data interval applicable or not
Local files Import all data Not applicable
Incremental file fetch
All files
Newest file
Oldest file
Applicable
Do not import data Not applicable
Local database Import all data Not applicable
Only modified and new data Applicable
Do not import data Not applicable
FTP servers Import all data Not applicable
Incremental file fetch
All files
Newest file
Oldest file
Applicable
Do not import data Not applicable

Import configuration for Apps and cloud database

If you import data from any of the below applications or cloud database, refer to the below options:

Data source How to import data from source? Data interval applicable or not
Cloud database Import all data Not applicable
Only modified and new data Applicable
Do not import data Not applicable
Cloud database query import Only modified and new data Applicable
Do not import data Not applicable
Zoho Creator Import all data Not applicable
Only modified and new data Applicable
Only new data Applicable
Do not import data Not applicable
Zoho Analytics Import all data Not applicable
Only modified and new data Applicable
Do not import data Not applicable
Zoho CRM Import all data Not applicable
Only modified and new data Applicable
Only new data Applicable
Do not import data Not applicable
Zoho Bigin Only new data Applicable
Do not import data Not applicable
Import all data Not applicable
Only modified and new data Applicable
Salesforce Import all data Not applicable
Only modified and new data Applicable
Only new data Applicable
Do not import data Not applicable
Dataiku(Beta) Import all data Not applicable
Do not import data Not applicable
Zoho Forms(Beta) Only new data Applicable
Only modified and new data Applicable
Do not import data Not applicable


SEE ALSO

Learn about the different pipeline runs in Zoho DataPrep


    Access your files securely from anywhere

      Zoho CRM Training Programs

      Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

      Zoho CRM Training
        Redefine the way you work
        with Zoho Workplace

          Zoho DataPrep Personalized Demo

          If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

          Zoho CRM Training

            Create, share, and deliver

            beautiful slides from anywhere.

            Get Started Now


              Zoho Sign now offers specialized one-on-one training for both administrators and developers.

              BOOK A SESSION





                          Quick LinksWorkflow AutomationData Collection
                          Web FormsEnterpriseBegin Data Collection
                          Interactive FormsWorkplaceData Collection App
                          CRM FormsCustomer ServiceAccessible Forms
                          Digital FormsMarketingForms for Small Business
                          HTML FormsEducationForms for Enterprise
                          Contact FormsE-commerceForms for any business
                          Lead Generation FormsHealthcareForms for Startups
                          Wordpress FormsCustomer onboardingOrder Forms for Small Business
                          No Code FormsConstructionRSVP tool for holidays
                          Free FormsTravel
                          Prefill FormsNon-Profit

                          Intake FormsLegal
                          Mobile App
                          Form DesignerHR
                          Mobile Forms
                          Card FormsFoodOffline Forms
                          Assign FormsPhotography
                          Mobile Forms Features
                          Translate FormsReal EstateKiosk in Mobile Forms
                          Electronic Forms

                          Notification Emails for FormsAlternativesSecurity & Compliance
                          Holiday FormsGoogle Forms alternative GDPR
                          Form to PDFJotform alternativeHIPAA Forms
                          Email Forms
                          Encrypted Forms
                          Embeddable Forms
                          Secure Forms
                          Drag & drop form builder
                          WCAG



                                            You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                                Manage your brands on social media


                                                  • Desk Community Learning Series


                                                  • Digest


                                                  • Functions


                                                  • Meetups


                                                  • Kbase


                                                  • Resources


                                                  • Glossary


                                                  • Desk Marketplace


                                                  • MVP Corner


                                                  • Word of the Day


                                                  • Ask the Experts


                                                    Zoho Marketing Automation

                                                      Zoho Sheet Resources

                                                       

                                                          Zoho Forms Resources


                                                            Secure your business
                                                            communication with Zoho Mail


                                                            Mail on the move with
                                                            Zoho Mail mobile application

                                                              Stay on top of your schedule
                                                              at all times


                                                              Carry your calendar with you
                                                              Anytime, anywhere




                                                                    Zoho Sign Resources

                                                                      Sign, Paperless!

                                                                      Sign and send business documents on the go!

                                                                      Get Started Now




                                                                              Zoho TeamInbox Resources



                                                                                      Zoho DataPrep Resources



                                                                                        Zoho DataPrep Demo

                                                                                        Get a personalized demo or POC

                                                                                        REGISTER NOW


                                                                                          Design. Discuss. Deliver.

                                                                                          Create visually engaging stories with Zoho Show.

                                                                                          Get Started Now









                                                                                                              • Related Articles

                                                                                                              • Pipeline sectional run

                                                                                                                When data propagation happens in a pipeline, the changes made in the upstream stages are loaded to the current stage that you're working on. This runs a section of a pipeline and therefore called Sectional runs. What is data propagation? Data ...
                                                                                                              • Pipeline Backfill run

                                                                                                                Backfill run can process data that was missed in previous schedules due to a change in data models or data preparation workflow. This way, you can catch up with missed schedules or if you want to import specific data within a timeframe data, or to ...
                                                                                                              • Create a new pipeline

                                                                                                                Pipelines are fundamental entities that allows users to create data flows in DataPrep, with multiple data stages and using various flow level transforms. You can create a new pipeline from scratch from the below places. Home page Pipelines tab ...
                                                                                                              • Pipeline

                                                                                                                Pipelines are fundamental entities that allows users to create data flows in DataPrep, with multiple data stages and various flow level transforms. To view the list of pipelines, you can click the Pipeline tab from the top bar. The Pipelines tab ...
                                                                                                              • Create pipeline from template library

                                                                                                                Pipelines are fundamental entities that allows users to create data flows in DataPrep, with multiple data stages and various flow level transforms. Template library is a place which contains pre-defined templates and all the saved pipeline templates. ...
                                                                                                                Wherever you are is as good as
                                                                                                                your workplace

                                                                                                                  Resources

                                                                                                                  Videos

                                                                                                                  Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                                  eBooks

                                                                                                                  Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                                  Webinars

                                                                                                                  Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                                  CRM Tips

                                                                                                                  Make the most of Zoho CRM with these useful tips.



                                                                                                                    Zoho Show Resources