Note :

1. DataPrep also supports files without any extension or files in plain text format. However, binary files cannot be parsed during import; users will have to manually parse the file into one of the supported formats listed above.

2. Maximum local file size supported during import is 100MB. You can find the details on other technical limitations here.

 To start with import    

Info: You can also click the Import data  icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.

2. Choose the Files option from the Choose your data source to import the local files. You can also click the Files category from the left pane and select the Files option. 

Import incremental data 

Info: Please ensure that you have installed Zoho Databridge in your machine to import local files. Click here to know how to download and install Zoho Databridge.

4. Choose an active Databridge and provide the following details:

Folder path : The folder path where you want to search for files. Eg. D:\DataPrep\Datasets

Info: Folder path is case-sensitive.

Include subfolders : You can select this checkbox if you want to include subfolders while searching for a file.

File pattern : The pattern used to match the file names in the specified location. This supports regex type matching. You can also use the pattern, ".*" to match any file in the path specified.

Info: File pattern is case-sensitive.

Note: The file pattern match is a simple regex type match. For example, to fetch files with names such as Sales_2022.csv,Sales_2023.csv, Sales_2024.csv , you can input the pattern Sales_.* 

Similarly to fetch files such as PublicData1.csv , PublicData2.csv , PublicData3.csv , use Public.*

If you want to import a single file, then specify the pattern using the exact file name.
Eg: leads_jan_2022.*

Parse file as : Choose the required extension to parse the file. If your file format is not a commonly used one, you can use this option to parse the file into one of the following formats before importing the data into a readable format. The available formats are CSV, TSV, JSON, XML, and TXT.

Info: This option can merge a maximum of only 5 files at a time.

Note: If this checkbox is unchecked then, only 1 file will be fetched at a time.

5. Click the Import button. Once you have completed importing data, the Visual Pipeline builder opens and you can start applying transforms. You can also right-click the stage and choose the Prepare data option and prepare your data in the DataPrep Studio page. Click here to know more about the transforms. 

6. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

8. After configuring a run, a pipeline job will be created at the run time. You can view the status of a job with the granular details in the Job summary. Click here to know more about the job summary.

You can import data from local files using Zoho Databridge. Look into the following sections to learn about the import configuration.

How to import data from source? You can choose to Import source file, or Do not import data using this option.

Import source file

This option will import the source file during every run.

Note: Import configuration is not applicable when File is the data source for Schedule, Webhook, and Zoho flow. But for Backfill and Manual reload, File upload option is provided during import configuration. You can use the Parse file as option to parse unknown file formats.

Do not import data  

The data is imported only once. The second time, the rules get applied to the same data and get exported.  

Import configuration for local files using Databridge

If you import data from local network, you can configure how to import and fetch incremental data from your local network using the below Import configuration options.

You can select the Click here link to set the import configuration.

Below is a snapshot from the backfill configuration.

 

How to import data from source?

Select the way you would like to import your data from the drop-down - Import all data, Incremental file fetch, Do not import data.

 Import all data   

This option will import all available data that matches the file pattern.

 

 

File batch size: Specify the batch size when you choose to import all files. Files will be grouped by this number, based on their first modified time. These files will be imported as a single file and exported to the destination in every run.

Info: File batch size cannot be greater than 10. This option is available only when you import data from local network

 Incremental file fetch   

You can configure how to import and fetch incremental data from your local network using the Import configuration option. Incremental data import is a method used to import new or modified records after the previous data interval. 

 

 

Use the previously imported file if no new file is available: 

When there are no new files in the source during incremental import,

If the checkbox is checked: The last fetched files will be imported again.

If the checkbox is unchecked: The import will be skipped and no files will be imported.

 

Which file to import? You can choose to import All files, Newest file, or Oldest file using this option.

 All files   

This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.

Fetch based on: You can choose to select the time based on which the file must be sorted and imported - Modified time

File batch size: Specify the batch size when you choose to import all files incrementally. Files will be grouped by this number, based on their modified time in that specific data interval. These files will be imported as a single file and exported to the destination in every run.

Info: File batch size cannot be greater than 10. This option is available only when you import data from local network.

 

 Newest file   

This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 Oldest file   

This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 

 Do not import data   

The data is imported only once. The second time, the rules get applied to the same data and get exported.

 

How incremental sync works

You can configure how to import and fetch incremental data from your source using the Import configuration option. Incremental data import is a method used to import new or modified records in the specific data interval.

In incremental file fetch, when the pipeline is run, the new or modified files will be fetched. During every data interval, the file created or modified after the previous data interval is imported. If there is no new file in the local file path, no data will be imported, or the previously imported data will be fetched again according to the import configuration. In the next data interval, the file created or modified in that interval will be fetched and so on.

 

Incremental fetch for schedule run 

In incremental file fetch, when the pipeline is scheduled, the data interval of the first schedule will span from the previous one up to the current data interval. During this interval, all the new or modified files will be fetched. The second schedule will extend from the current interval to the next, with subsequent schedules following this pattern. 

 

For example, The schedule is set to 1 hour interval. The data source contains a total of 10 files, in which 5 files were uploaded an hour ago that matches the file pattern. Import configuration is set to import all files incrementally, and the batch size is set to 10.

 

When the first schedule runs, the 5 files created/modified in the last 1 hour will be fetched and exported as a single file.

 

In the second schedule, if 4 new files with the same file pattern are added in the source, only the 4 files will be fetched and exported as a single file based on the created/modified time in the source in that specific interval.
 

In the third schedule, if 13 new files are added to the source that matches the file pattern, then only the first 10 files will be fetched and exported as a single file based on the created/modified time in the source since the batch size is set as 10. The same logic will be applied for files during incremental fetch in the upcoming schedules and so on.

Click here to know about the schedule run.

Incremental fetch for backfill run

 

In an incremental file fetch, when a backfill run is configured for a pipeline, all incremental files within the specified data interval will be fetched.

For instance, the backfill is set from August 1st to 3rd with a daily job frequency. August 1st has 4 files, August 2nd has 7 files, and August 3rd has 13 files. The import configuration is set to fetch all files incrementally with a batch size of 10.

During the first backfill job, the 4 files created or modified on August 1st will be fetched and exported as a single file. In the second backfill job, the 7 files from August 2nd will be similarly processed and exported. In the third backfill job, the first 10 files from August 3rd will be fetched and exported as a single file.

Click here to know about the backfill run.

 

Incremental fetch for reload data

In an incremental file fetch, when a reload is configured for a pipeline, all new or modified files within the specified data interval will be fetched.

 

For instance, the reload is set between August 1st and 2nd.

August 1st has 3 files, August 2nd has 4 files. The import configuration is set to fetch all files incrementally with a batch size of 5 based on the oldest modified time.

During the reload job, the 3 files created on August 1st, the first 2 files created on August 2nd will be fetched and exported as a single file.

Click here to know about the reload.

SEE ALSO

How to incrementally import data from FTP servers?

How to export data to local files?