You can manually refresh your stage with the latest data by reloading data from your data source. You can use the manual reload option to fetch data that was failed during a schedule. This is part of your ETL workflow for data movement, ensuring fresh records are fetched as needed for accurate data integration downstream.

Note: Manual reload will not affect the schedule or backfill runs if any.

Steps to reload data 

1.  Create a pipeline or open an existing pipeline from the Home Page, Pipelines tab, or Workspaces tab, and click the Add data option. You can also click the Import data option under the Workspaces tab to import data.  

Info: You can also click the Import data icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.

2. In the next screen, choose the required data source. You can also choose one of the below categories from the left pane.  

• Files

• Databases

• Cloud storage

• Cloud databases

• Zoho Apps

• Saved connections
  

3. Once all files are successfully imported, it will take you to the Pipeline builder page where you can start applying transforms. You can also right-click the stage and choose the Prepare data option to prepare your data in the DataPrep Studio page. These transforms are part of the ETL “transform” phase, helping with data cleaning and shaping for effective data integration. Click here to know about transforms. 

4. To reload data manually, click the ellipses icon from the Pipeline builder page, and choose the Reload data option. This reload triggers data movement that ensures the latest source data flows into your ETL pipeline.

Data Interval

Important: Data interval is mandatory only when you want to import data incrementally. Incremental imports are a key strategy in ETL to optimize data movement and reduce overhead.

 By default, the From option in the data interval will be pre-filled based on the last successful refresh, meaning the data will be fetched from last successful refresh up to the present date-time. You can modify it if needed. 

Click the Edit option on the top right corner and provide the data interval using the below options:

Select range

From - Using this option, you can import data that matches the given pattern from the specified date and time.

Between - Using this option, you can import data that matches the given pattern between two time intervals.

Note: The data will be imported based on the import configuration. This helps maintain consistent data integration across runs.

Import configuration

Info: The below options are applicable for cloud storage services. These import options may vary based on the data source you have selected. Refer to the import configuration tables below to know about other data sources and to know when a data interval is applicable.

Import from source file

This option will import all data from the source file. This can be used if you do an import without the advanced selection method.

 Import all data     

This option will import all available data that matches the file pattern.

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage, FTP, and local file systems.

 Incremental file fetch     

You can configure how to import and fetch incremental data from your cloud storage using this option. Incremental data import is a method used to import new or modified records after the previous sync. This strategy helps efficient data movement and supports reliable data integration.

Use the previously imported file if no new data is available: 

1. If the checkbox is checked: When there is no new data in the source, the last fetched data will be imported again.
2. If the checkbox is unchecked: When there is no new data in the source, the import will fail and no files will be imported. This will, in turn, cause the entire pipeline job to fail.

Which file to import? You can choose to import All files, Newest file, or Oldest file using this option.

 All files     

This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.

Note: GoogleDrive, OneDrive, SharePoint, Box has modifiedTime and createdTime.
DropBox, Amazon S3 has only modifiedTime (There is no created time)

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage, FTP, and local file system.

 Newest file     

This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Oldest file     

This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Do not import data     

The data is imported only once. The second time, the rules get applied to the same data and get exported.

6. After configuring the import configuration and data interval, click Reload data. The data reload will be triggered successfully.

7. Once the data reloads, you can apply the required transforms and add destination and export your data by doing a manual run.

Important: Please make sure you do a manual run and export the reloaded data to avoid data loss if there is an ongoing schedule. You can also download the data from the Output tab in the Job summary page as a job will be created each time you perform a manual reload on the ETL pipeline.

Import configuration for simple selection in cloud storage

Importing data without using a regex pattern, can be termed as a simple selection. If you import data from a cloud storage service using the simple selection method, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Cloud storage	Import source file	Not applicable
	Do not import data	Not applicable

Import configuration for advanced selection in cloud storage

Zoho DataPrep allows you to import data using the Advanced selection option. Advanced selection helps you perform dynamic file selection based on a regex pattern. If you import data using the Advanced selection method, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Cloud storage	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable

Import configuration for data sources without using Databridge

If you import data from the below sources without using Zoho Databridge, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Local files	Import source file Note: You can upload a new source file that you would like to import using the Upload File option. You can use the Parse file as option to parse unknown file formats.	Not applicable
	Do not import data	Not applicable
URL	Import source file	Not applicable
	Do not import data	Not applicable
Local database	Import all data	Not applicable
	Do not import data	Not applicable
FTP servers	Import source file	Not applicable
	Do not import data	Not applicable

Import configuration for data sources using Databridge

Zoho DataPrep allows you to import data from your local network using Zoho Databridge. If you import data from the below sources using Zoho Databridge, refer to the below options:

Data source	How to import data from source?	Data interval applicable or not
Local files	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable
Local database	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
FTP servers	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable

Import configuration for Apps and cloud database

If you import data from any of the below applications or cloud database, refer to the below options:

Data source	How to import data from source?	Data interval applicable or not
Cloud database	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
Cloud database query import	Only modified and new data	Applicable
	Do not import data	Not applicable
Zoho Creator	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Zoho Analytics	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
Zoho CRM	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Zoho Bigin	Only new data	Applicable
	Do not import data	Not applicable
	Import all data	Not applicable
	Only modified and new data	Applicable
Salesforce	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Dataiku(Beta)	Import all data	Not applicable
	Do not import data	Not applicable
Zoho Forms(Beta)	Only new data	Applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable

Note: For Zoho applications, if you choose the Import all data option, the date provided in the From field will be considered and not the data interval. Data will be imported from the date specified in the From field.

  

Incremental fetch for manual reload

In an incremental file fetch, when a manual reload is configured for a pipeline, all new or modified files within the specified data interval will be fetched.

 

For instance, the Manual reload is set from August 1st to 2nd.

August 1st has 3 files, August 2nd has 4 files. The import configuration is set to fetch all files incrementally with a batch size of 5 based on the oldest created time.

During the manual reload job, the 3 files created on August 1st, the first 2 files created on August 2nd will be fetched and exported as a single file.

SEE ALSO

Import configuration for other data sources in Zoho DataPrep