Pipeline Manual run
By triggering the manual run option in your pipeline, you can export the data you have prepared to your destination. A manual run will run your pipeline on the existing data without refreshing, which means no new data will be imported.
Note: You can also use this option for dry runs and to test if your pipeline can be executed without any failures before automating the pipeline using a schedule or backfill run. However, please make sure you configure the pipeline for your testing environment, especially destinations as the transforms and export will happen for whichever data destinations are configured.
Info: A manual run will run on the latest version of the pipeline. A version is nothing but stages of the actions performed on a pipeline. If you want to do a manual run on any old version, you can restore the required version, and perform a manual run. Click here to know more about versions. To execute manual run in a pipeline
1. Open your pipeline and go to the pipeline builder view. Click here if you want to know how to create a pipeline in the first place.
2. Click on the Add data option to bring your data from the required data source. Learn more on 50+ data sources.
After data is imported, you will be redirected to the pipeline builder where you can see your data source, and a stage linked to it.
Stages are nodes created for processing data while applying data flow transforms. Every dataset imported from your data source will have a stage created by default.
3. You can right-click the stage and apply the data flow transforms.
4. You can also choose to Prepare Data to open the DataPrep Studio page and explore more data transforms. Click here to know the various elements in a pipeline builder.
5. Once you are done creating your data flow and apply necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.
6. Data destination is a place where you want to export your data to. It can be your local database, business applications like Zoho CRM, Zoho Analytics, etc. You can choose your preferred destination out of 50+ data destinations in DataPrep to export your prepared data to.
Manual run
7. After adding a destination for your pipeline, click the Run drop down and choose Manual run.
Settings
- Stop export data quality drops below 100%: You can use this toggle if you would like to stop export if data quality drops below 100 percent.
- Order export: You can use this option when you have configured multiple destinations and would like to determine in what order the data has to be exported to destinations.
Note: This option will be visible only if you have added more than one destination in your pipeline.
Data Refresh options
Under the Data refresh options, you can set the configuration for how you would like to fetch data during the manual run.
Use existing data : This option will use the source file data that was imported. This option will be selected by default.
Refresh data from source : You can refresh and import new data from the source based on the import configuration.
Data Interval
Important: Data interval is mandatory only when you want to import data incrementally.
By default, the From time in the data interval will be pre-filled based on the last successful refresh, meaning the data will be fetched from last successful refresh up to the present date-time. You can always choose to edit it.
Click the Edit option on the top right corner and provide the data interval using the below options:
Click the Edit option on the top right corner and provide the data interval using the below options:
Select range
From - Using this option, you can import data that matches the given pattern from the specified date and time.
Between - Using this option, you can import data that matches the given pattern between two time intervals.
Note: The data will be imported based on the import configuration.
Import configuration
Provide the import configuration for your data sources using the required options.
Info: The below options are applicable for cloud storage services. These import options may vary based on the data source you have selected. Refer to the import configuration tables below to know about other data sources and when a data interval is applicable.
How to import data from source? Choose how you would like to import data using this option.
Import all data
All data that matches the specified pattern will be imported.
File batch size: Data will be imported as a single file with the specified number of files in this field. For example, you have 10 files in your source and you provide 10 as the file batch size, all 10 files will be imported as 1 file.
Note: The maximum supported value for a File batch size is 10.
Incremental file fetch
Incremental file fetch- Using this option, you can incrementally fetch data that match the specified pattern. Incremental data fetch is a method used to import new files from a source after the previous sync.
Use the previously imported data if no new data is available:
During incremental import,
- If the checkbox is checked: When there is no new data in the source, the last fetched data will be imported again.
- If the checkbox is unchecked: When there is no new data in the source, the import will fail and no files will be imported. This will, in turn, cause the entire pipeline job to fail.
Which file to import?
Choose one of the below options:
All files- All files will be imported based on the option selected in the Fetch based on field.
Newest file - The newest file will be imported based on the option selected in the Fetch based on field.
Oldest file- The oldest file will be imported based on the option selected in the Fetch based on field.
Fetch based on - Files will be fetch based on the Created or modified time. For example, if you want the first created file, choose Created Time and choose Oldest file option in the Which file to import? field.
File batch size: Data will be imported as a single file with the specified number of files in this field. For example, you have 10 files in your source and you provide 10 as the file batch size, all 10 files will be imported as 1 file.
Note: The maximum supported value for a File batch size is 10.
Example for incremental fetch
Let's say you want to import all data based on the created time from Feb 1, 2025 12:00 AM to Feb 4, 2025 12:00 AM. Below is the configuration and data interval snapshot. This will import all data created from Feb 1, 2025 12:00 AM to Feb 4, 2025 12:00 AM
Import configuration
Data interval
Do not import data
This option will not import any new data. The source file will be used.
8. Click on the Run button to execute the manual run. This will start processing the pipeline and the Job summary page opens. The Job summary shows the progress of a job execution. Click here to know more. You can view the status of the run on the Job summary page. There are three different status for a job in DataPrep - Initiated, Success and Failure.
If the job fails, you can identify if the error is at the import stage, transform stage, or with the destination, and target matching in the Job summary page. You can hover on the failed stage to view the error details and fix them to proceed exporting.
In the Overview tab, you can view the status of the job along with the details such as user who started the job, storage used, total rows processed, start time, end time and duration of the job. Click here to know more.
In the Stages tab, you can view the details of each pipeline stage such as Import, Transform, and Export. Click here to know more.
In the Output tab, you see the list of all exported data. You can also download the output if needed. Click here to know more.
Note: Jobs are listed under each pipeline's menu, the jobs are stored indefinitely and can be referred whenever needed.9. When the manual run is completed, the data prepared in your pipeline will be exported to the configured destinations.
Import configuration for simple selection in cloud storage
Importing data without using a regex pattern, can be termed as a simple selection. If you import data from a cloud storage service using the simple selection method, refer to the below options:
| How to import data from source |
| Import source file |
| Do not import data |
Import configuration for advanced selection in cloud storage
Zoho DataPrep allows you to import data using the Advanced selection option. Advanced selection helps you perform dynamic file selection based on a regex pattern. If you import data using the Advanced selection method, refer to the below options:
| How to import data from source |
| Import all data |
| Incremental file fetch All files Newest file Oldest file |
| Do not import data |
Import configuration for data sources without using Databridge
If you import data from the below sources without using Zoho Databridge, refer to the below options:
| How to import data from source |
| Import source file Note: You can upload a new source file that you would like to import using the Upload File option. You can use the Parse file as option to parse unknown file formats. |
| Do not import data |
| Import source file |
| Do not import data |
| Import all data |
| Do not import data |
| Import source file |
| Do not import data |
Import configuration for data sources using Databridge
Zoho DataPrep allows you to import data from your local network using Zoho Databridge. If you import data from the below sources using Zoho Databridge, refer to the below options:
| How to import data from source? |
| Import all data |
| Incremental file fetch All files Newest file Oldest file |
| Do not import data |
| Import all data |
| Only modified and new data |
| Do not import data |
| Import all data |
| Incremental file fetch All files Newest file Oldest file |
| Do not import data |
Import configuration for Apps and cloud database
If you import data from any of the below applications or cloud database, refer to the below options:
| Data source | How to import data from source? | Data interval applicable or not |
| Import all data | ||
| Only modified and new data | ||
| Do not import data | ||
| Only modified and new data | ||
| Do not import data | ||
| Import all data | ||
| Only modified and new data | ||
| Only new data | ||
| Do not import data | ||
| Import all data | ||
| Only modified and new data | ||
| Do not import data | ||
| Import all data | ||
| Only modified and new data | ||
| Only new data | ||
| Do not import data | ||
| Only new data | ||
| Do not import data | ||
| Import all data | ||
| Only modified and new data | ||
| Import all data | ||
| Only modified and new data | ||
| Only new data | ||
| Do not import data | ||
| Import all data | ||
| Do not import data | ||
| Only new data | ||
| Only modified and new data | ||
| Do not import data |
Note: For Zoho applications, if you choose the Import all data option, the date provided in the From field will be considered and not the data interval. Data will be imported from the date specified in the From field.
SEE ALSO
Access your files securely from anywhere
Zoho CRM Training Programs
Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.
Zoho DataPrep Personalized Demo
If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.
- Zoho Workerly Home
- Forums
- Connect With Us:
- Zoho Recruit Home
- Forums
- Connect With Us:
New to Zoho Writer?
Create. Review. Publish.
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho CRM Plus?
Deliver unforgettable customer experiences
New to Zoho Marketing Plus?
Everything you need to run your marketing
New to Zoho Marketing Plus?
Everything you need to run your marketing
You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.
New to Zoho Survey?
Zoho Sheet Resources
Zoho Forms Resources
New to Zoho Sign?
Zoho Sign Resources
Sign, Paperless!
New to Zoho TeamInbox?
Zoho TeamInbox Resources
New to Zoho ZeptoMail?
Design. Discuss. Deliver.
New to Zoho Workerly?
New to Zoho Recruit?
New to Zoho CRM?
New to Zoho Projects?
New to Zoho Sprints?
New to Zoho Assist?
New to Bigin?
Related Articles
Pipeline sectional run
When data propagation happens in a pipeline, the changes made in the upstream stages are loaded to the current stage that you're working on. This runs a section of a pipeline and therefore called Sectional runs. What is data propagation? Data ...Pipeline Backfill run
Backfill run can process data that was missed in previous schedules due to a change in data models or data preparation workflow. This way, you can catch up with missed schedules or if you want to import specific data within a timeframe data, or to ...Create a new pipeline
Pipelines are fundamental entities that allow users to create ETL pipelines / data flows in DataPrep, with multiple data stages and using various flow‑level transforms. You can create a new pipeline from scratch (for your data integration and data ...Create pipeline from template library
Pipelines are fundamental entities that allow users to create ETL pipelines / data flows in DataPrep, with multiple data stages and using various flow‑level transforms. Template library is a place which contains pre-defined templates and all the ...Pipeline
Pipelines are fundamental entities that allow users to create data flows in DataPrep, with multiple data stages and various flow-level transforms. To view the list of pipelines, you can click the Pipeline tab from the top bar. The Pipelines tab shows ...
New to Zoho LandingPage?
Zoho LandingPage Resources