You can manually refresh your stage with the latest data by reloading data from your data source. You can use the manual reload option to fetch data that was failed during a schedule.

Note: Manual reload will not affect the schedule or backfill runs if any.

Steps to reload data 

1.  Create a pipeline or open an existing pipeline from the Home Page, Pipelines tab, or Workspaces tab, and click the Add data option. You can also click the Import data option under the Workspaces tab to import data.  

Info: You can also click the Import data icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.

2. In the next screen, choose the required data source. You can also choose one of the below categories from the left pane.  

• Files

• Databases

• Cloud storage

• Cloud databases

• Zoho Apps

• Saved connections
  

3. Once all files are successfully imported, it will take you to the Pipeline builder page where you can start applying transforms. You can also right-click the stage and choose the Prepare data option to prepare your data in the DataPrep Studio page. Click here to know about transforms. 

4. To reload data manually, click the ellipses icon from the Pipeline builder page, and choose the Reload data option.

5. If you would like to import data based on a time interval, click the Add data interval option on the top right corner.

Important: Data interval is not mandatory. You can use this option if you want to import data incrementally. 

Select range

From - Using this option, you can import data that matches the given pattern from the specified date and time.

Note: The data will be imported based on the import configuration.

Between - Using this option, you can import data that matches the given pattern between two time intervals.

Note: The data will be imported based on the import configuration.

Import configuration

Info: The below options are applicable for cloud storage services. These import options may vary based on the data source you have selected. Refer to the import configuration tables below to know about other data sources

Import from source file

This option will import all data from the source file. This can be used if you do an import without the advanced selection method.

 Import all data     

This option will import all available data that matches the file pattern.

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage, FTP, and local file systems.

 Incremental file fetch     

You can configure how to import and fetch incremental data from your cloud storage using this option. Incremental data import is a method used to import new or modified records after the previous sync.

Use the previously imported file if no new file is available: 

When there are no new files in the source during incremental import,

If the checkbox is checked: The last fetched files will be imported again.

If the checkbox is unchecked: The import will be skipped and no files will be imported.

Which file to import? You can choose to import All files, Newest file, or Oldest file using this option.

 All files     

This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.

Note: GoogleDrive, OneDrive, SharePoint, Box has modifiedTime and createdTime.
DropBox, Amazon S3 has only modifiedTime (There is no created time)

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage, FTP, and local file system.

 Newest file     

This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Oldest file     

This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Do not import data     

The data is imported only once. The second time, the rules get applied to the same data and get exported.

6. After configuring the import configuration and data interval, click Reload data. The data reload will be triggered successfully.

7. Once the data reloads, you can apply the required transforms and add destination and export your data by doing a manual run

Important: Please make sure you do a manual run and export the reloaded data to avoid data loss if there is an ongoing schedule. You can also download the data from the Output tab in the Job summary page as a job will be created each time you perform a manual reload.

Import configuration for simple selection in cloud storage

Importing data without using a regex pattern, can be termed as a simple selection. If you import data from a cloud storage service using the simple selection method, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Cloud storage	Import source file	Not applicable
	Do not import data	Not applicable

Import configuration for advanced selection in cloud storage

Zoho DataPrep allows you to import data using the Advanced selection option. Advanced selection helps you perform dynamic file selection based on a regex pattern. If you import data using the Advanced selection method, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Cloud storage	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable

Import configuration for data sources without using Databridge

If you import data from the below sources without using Zoho Databridge, refer to the below options:

Data source	How to import data from source	Data interval applicable or not
Local files	Import source file Note: You can upload a new source file that you would like to import using the Upload File option. You can use the Parse file as option to parse unknown file formats.	Not applicable
	Do not import data	Not applicable
URL	Import source file	Not applicable
	Do not import data	Not applicable
Local database	Import all data	Not applicable
	Do not import data	Not applicable
FTP servers	Import source file	Not applicable
	Do not import data	Not applicable

Import configuration for data sources using Databridge

Zoho DataPrep allows you to import data from your local network using Zoho Databridge. If you import data from the below sources using Zoho Databridge, refer to the below options:

Data source	How to import data from source?	Data interval applicable or not
Local files	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable
Local database	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
FTP servers	Import all data	Not applicable
	Incremental file fetch All files Newest file Oldest file	Applicable
	Do not import data	Not applicable

Import configuration for Apps and cloud database

If you import data from any of the below applications or cloud database, refer to the below options:

Data source	How to import data from source?	Data interval applicable or not
Cloud database	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
Cloud database query import	Only modified and new data	Applicable
	Do not import data	Not applicable
Zoho Creator	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Zoho Analytics	Import all data	Not applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable
Zoho CRM	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Zoho Bigin	Only new data	Applicable
	Do not import data	Not applicable
	Import all data	Not applicable
	Only modified and new data	Applicable
Salesforce	Import all data	Not applicable
	Only modified and new data	Applicable
	Only new data	Applicable
	Do not import data	Not applicable
Dataiku(Beta)	Import all data	Not applicable
	Do not import data	Not applicable
Zoho Forms(Beta)	Only new data	Applicable
	Only modified and new data	Applicable
	Do not import data	Not applicable

Note: For Zoho applications, if you choose the Import all data option, the date provided in the From field will be considered and not the data interval. Data will be imported from the date specified in the From field.

  

Incremental fetch for manual reload

In an incremental file fetch, when a manual reload is configured for a pipeline, all new or modified files within the specified data interval will be fetched.

 

For instance, the Manual reload is set from August 1st to 2nd.

August 1st has 3 files, August 2nd has 4 files. The import configuration is set to fetch all files incrementally with a batch size of 5 based on the oldest created time.

During the manual reload job, the 3 files created on August 1st, the first 2 files created on August 2nd will be fetched and exported as a single file.

SEE ALSO

Import configuration for other data sources in Zoho DataPrep