You can import data from cloud storage using simple selection or advanced selection. Look into the following sections to learn about the import configuration of each selection type.

Import configuration for simple selection

If you import data from cloud storage using the simple selection method, you can use the below import configuration options. You can select the Click here link to set the import configuration.

Below is a snapshot from the schedule configuration.

 

How to import data from source? You can choose to Import source file, or Do not import data using this option.

 

Import source file

This option will import the source file during every run.

 Do not import data

The data is imported only once. The second time, the rules get applied to the same data and get exported.

  

 

If you import data from cloud storage using the advanced selection, you can configure how to import and fetch incremental data from your cloud storage using the below Import configuration options.

Below is a snapshot from the schedule configuration.

 Import all data   

This option will import all available data that matches the file pattern.

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage

 Incremental file fetch   

You can configure how to import and fetch incremental data from your cloud storage using the Import configuration option. Incremental data import is a method used to import new or modified records after the previous sync. Click here to know more.

Use the previously imported data if no new data is available: 

1. If the checkbox is checked: When there is no new data in the source, the last fetched data will be imported again.
2. If the checkbox is unchecked: When there is no new data in the source, the import will fail and no files will be imported. This will, in turn, cause the entire pipeline job to fail.

Which file to import? You can choose to import All files, Newest file, or Oldest file using this option.

 All files   

This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.

Note: Google Drive, OneDrive, SharePoint, Box has modifiedTime and createdTime.
DropBox, Amazon S3 has only modifiedTime (There is no created time)

Info: File batch size cannot be greater than 10. This option is available only for advanced selection in cloud storage.

 Newest file   

This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Oldest file   

This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Do not import data   

The data is imported only once. The second time, the rules get applied to the same data and get exported.