How to import data from FTP servers?

Import data from FTP servers




Zoho DataPrep allows you to fetch data from remotely located FTP servers and from local network using Zoho Databridge. With this connector, you can streamline your ETL workflows, enable smooth data movement, and simplify data integration between FTP servers and Zoho DataPrep.

To import data using an FTP connection

1. Create a pipeline or open an existing pipeline from the Home Page, Pipelines tab or Workspaces tab and click the Add data option. You can also click the Import data option under the Workspaces tab to import data.  
Info: You can also click the Import data  icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.



2. Choose the FTP option in the Choose your data source section.



3. Give a connection name. Configure the FTP connection to connect to the server and fetch the file. You can enable the Import from local network toggle to import data that match the specific . Click here to know more.



4. Select the FTP server type.

5. Enter the FTP server host. You can also enter the IP address or the FQDN (Fully Qualified Domain Name) of the server.

6. Enter the username and password if authentication is required.

Note: The connection configuration will be saved for importing data in the future. Credentials are securely encrypted and stored. 

SFTP-SSH File Transfer Protocol

7. If you select the SFTP (SSH File Transfer Protocol) FTP server type, you can login using either a Username & Password or a Key file.



8. If you choose to login using the Key file option, upload your key file.




7. Click the Connect button and provide the following details.


Folder path : The folder path where you want to search for files. Eg. /srv/ftp/filesfortest/

Info: Folder path is case-sensitive.

Include subfolders : You can select this checkbox if you want to include subfolders while searching for a file.

File pattern : The pattern used to match the file names in the specified location. This supports regex type matching. You can also use the pattern, ".*" to match any file in the path specified.

For example, to fetch files with names such as Sales_2022.csv , Sales_2023.csv , Sales_2024.csv , you can input the pattern Sales_.* 

Similarly to fetch files such as PublicData1.csv , PublicData2.csv , PublicData3.csv , use Public.*

If you want to import a single file, then specify the pattern using the exact file name.
Eg: leads_jan_2022.*

Info: File pattern is case-sensitive.

Parse file as : Choose the required extension to parse the file. If your file format is not a commonly used one, you can use this option to parse the file into one of the following formats before importing the data into a readable format. The available formats are CSV, TSV, JSON, XML, and TXT.

Merge files and import - This will merge all the files that match the pattern specified and import them as a single dataset.

You can use this option to merge files together during the import itself, without having to perform unions post import.

Info: This option can merge a maximum of only 5 files at a time.
NotesNote: If this checkbox is unchecked then, only 1 file will be fetched at a time.

Eg. If your FTP server has 10 files in a path, the first 5 will be merged into one dataset and imported. During the next reload, the remaining 5 files will be merged and imported.

Similarly, if your FTP server has 8 files in a path, the first 5 will be merged and fetched first, followed by the next 3.

8. Click Import to upload data into Zoho DataPrep from the FTP server.

9. Once all files are successfully imported, it will take you to the Pipeline builder page where you can start applying transforms to the ETL pipeline. You can also right-click the stage and choose the Prepare data option to prepare your data in the DataPrep Studio page.



10. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

NotesNote:  After adding a destination to the ETL pipeline, you can try executing your pipeline using a manual run at first. Once you make sure manual run works, you can then set up schedule to automate the pipeline and data movement. Learn about the different types of runs here.

To import data from local network

Using the Import data from local network toggle, you can perform dynamic file selection based on the regex. This can be used for getting new or incremental data from your FTP servers. Incremental fetch is a method used to import new files from a source after the previous sync. Zoho DataPrep helps you import incremental data from your local files using Zoho Databridge.

Important: This option is used not only for incremental fetch. You can also use this option for bulk import of files based on the file pattern.

1. Enable the Import from local network toggle if you want to import local files from your FTP server incrementally.

 

Info: Please ensure that you have installed Zoho Databridge in your machine to import files from FTP . Click here to know how to download and install Zoho Databridge.

 

2. Choose an active Databridge.

 

3. Configure FTP connection to connect to the server and fetch the file.

 

4. Select one of the below options in the FTP server type drop down.

  • FTP - File Transfer

  • FTPS - File Transfer Over Implicit TLS/SSL

  • FTPS - File Transfer Over Explicit TLS/SSL

  • SFTP - SSH File Transfer Protocol

 

5. Enter the FTP server host. You can also enter the IP address or the FQDN (Fully Qualified Domain Name) of the server.

 

6. Enter the Username and Password if authentication is required.


Note: The connection configuration will be saved for importing data in the future. Credentials are securely encrypted and stored.

SFTP-SSH File Transfer Protocol

7. If you select the SFTP (SSH File Transfer Protocol) server type, you can login using either a Username & Password or a Key file.



8. If you choose to login using the Key file option, upload your key file.

Important: This option will not be available if you have the older versions of Zoho Databridge. To import data from your local network using the Key file login type, make sure you have installed the latest version of Zoho Databridge. Click here to know how to download and install Zoho Databridge.



 

9. Click the Connect button and provide the following details.



Folder path : The folder path where you want to search for files. Eg. /srv/ftp/filesfortest/

Info: Folder path is case-sensitive.

Include subfolders : You can select this checkbox if you want to include subfolders while searching for a file.

File pattern : The pattern used to match the file names in the specified location. This supports regex type matching. You can also use the pattern, ".*" to match any file in the path specified.

For example, to fetch files with names such as Sales_2022.csv , Sales_2023.csv , Sales_2024.csv , you can input the pattern Sales_.* 

Similarly to fetch files such as PublicData1.csv , PublicData2.csv , PublicData3.csv , use Public.*

If you want to import a single file, then specify the pattern using the exact file name.
Eg: leads_jan_2022.*

Info: File pattern is case-sensitive.

File password : Enter the password if the file is password protected.

File type : Choose the required file format. The available formats are CSV, TSV, JSON, XLS, XLSX, XML, and TXT.

Merge files and import : This will merge all the files that match the pattern specified and import them as a single dataset.

Eg. If your FTP server has 10 files in a path, the first 5 will be merged into one dataset and imported. During the next reload, the remaining 5 files will be merged and imported.

Similarly, if your FTP server has 8 files in a path, the first 5 will be merged and fetched first, followed by the next 3.

Sheet pattern : This option is available for the XLS and XLSX formats only. The pattern used to match the sheet names in the file. This supports regex type matching. You can also use the pattern ".*" to match any sheet in the file.

The sheet pattern match is also a simple regex-based match. For example, to fetch sheets with names such as Sales_2022, Sales_2023, Sales_2024, you can input the pattern Sales_.*

Similarly, to fetch sheets such as PublicData1, PublicData2, PublicData3, use Public.*

If you want to import a single sheet, then specify the pattern using the exact sheet name.
Eg: Leads_Jan_2022.*
Info
Info: Sheet pattern is case-sensitive.
Sheet password : This option is available for XLS and XLSX formats only. Enter the password if the sheet is password protected.
Merge sheets and import : This will merge all the files that match the pattern specified and import them as a single dataset.


Idea
You can use this option to merge sheets together during the import itself, without having to perform unions post import.
NotesNote: If this checkbox is unchecked then, only 1 sheet will be fetched at a time. 

10. Click Import to upload data into Zoho DataPrep from the FTP server.


To edit the FTP connection

DataPrep saves your data connections to avoid the hassle of keying in the credentials every time you need to connect to a data source or destination. You can always edit the saved data connection and update them with new parameters or credentials using the Edit connection option. 

1. While importing a data in a pipeline, click Saved data connections from the left pane in the Choose a data source section. 



2. You can manage your saved data connections right from the data import screen. Click the ellipsis (3 dots) icon to share, edit, view the connection overview, or remove the connection. 




3. Click the Edit connection option to update the saved connection with new parameters or credentials. 



How to schedule pipelines?