How to import incremental data from FTP server?

Import incremental data from FTP servers



Incremental data fetch is a method used to import new files from a source. Zoho DataPrep helps you import incremental data from your FTP servers using Zoho Databridge. In DataPrep, you can incrementally import various file types from FTP, including CSV, TSV, JSON, XML, and TXT.

To start with import    

1. Open an existing pipeline or create a pipeline from the Home Page, Pipelines tab, or Workspaces tab and click the Add data option.

InfoInfo: You can also click the Import data  icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.
 

  

 

2. Choose the FTP option from the Choose your data source section or choose it from the Files category in the left pane.

 

 

Note: If you had already added a FTP connection earlier, you can simply select the existing connection under FTP servers in the Saved connections section and proceed with importing. To learn more about Saved connections, click here.


Import incremental data 

 

 

3. Enable the Import from local network toggle to import local files from your FTP server incrementally.

 

Info: Please ensure that you have installed Zoho Databridge in your machine to import files from FTP . Click here to know how to download and install Zoho Databridge.

 

3. Choose an active Databridge.

 

4. Configure FTP connection to connect to the server and fetch the file.

 

5. Select one of the below options in the FTP server type drop down.

  • FTP - File Transfer

  • FTPS - File Transfer Over Implicit TLS/SSL

  • FTPS - File Transfer Over Explicit TLS/SSL

  • SFTP - SSH File Transfer Protocol

 

 

6. Enter the FTP server host. You can also enter the IP address or the FQDN (Fully Qualified Domain Name) of the server.

 

7. Enter the Username and Password if authentication is required.


Note: The connection configuration will be saved for importing data in the future. Credentials are securely encrypted and stored.

 

SFTP-SSH File Transfer Protocol

7. If you select the SFTP (SSH File Transfer Protocol) server type, you can login using either a Username & Password or a Key file.




8. If you choose to login using the Key file option, upload your key file.

Important: This option will not be available if you have the older versions of Zoho Databridge. To import data from your local network using the Key file login type, make sure you have installed the latest version of Zoho Databridge. Click here to know how to download and install Zoho Databridge.



 

9. Click the Connect button and provide the following details.






Folder path : The folder path where you want to search for files. Eg. /srv/ftp/filesfortest/

Info: Folder path is case-sensitive.

Include subfolders : You can select this checkbox if you want to include subfolders while searching for a file.

File pattern : The pattern used to match the file names in the specified location. This supports regex type matching. You can also use the pattern, ".*" to match any file in the path specified.

For example, to fetch files with names such as Sales_2022.csv , Sales_2023.csv , Sales_2024.csv , you can input the pattern Sales_.* 

Similarly to fetch files such as PublicData1.csv , PublicData2.csv , PublicData3.csv , use Public.*

If you want to import a single file, then specify the pattern using the exact file name.
Eg: leads_jan_2022.*

Info: File pattern is case-sensitive.

File password : Enter the password if the file is password protected.

Merge files and import - This will merge all the files that match the pattern specified and import them as a single dataset.
Info: This option can merge a maximum of only 5 files at a time.
NotesNote: If this checkbox is unchecked then, only 1 file will be fetched at a time.

Eg. If your FTP server has 10 files in a path, the first 5 will be merged into one dataset and imported. During the next reload, the remaining 5 files will be merged and imported.

Similarly, if your FTP server has 8 files in a path, the first 5 will be merged and fetched first, followed by the next 3.

Sheet pattern : This option is available for the XLS and XLSX formats only. The pattern used to match the sheet names in the file. This supports regex type matching. You can also use the pattern ".*" to match any sheet in the file.

The sheet pattern match is also a simple regex-based match. For example, to fetch sheets with names such as Sales_2022, Sales_2023, Sales_2024, you can input the pattern Sales_.*

Similarly, to fetch sheets such as PublicData1, PublicData2, PublicData3, use Public.*

If you want to import a single sheet, then specify the pattern using the exact sheet name.
Eg: Leads_Jan_2022.*
Info
Info: Sheet pattern is case-sensitive.
Sheet password : This option is available for XLS and XLSX formats only. Enter the password if the sheet is password protected.
Merge sheets and import : This will merge all the files that match the pattern specified and import them as a single dataset.


Idea
You can use this option to merge sheets together during the import itself, without having to perform unions post import.
NotesNote: If this checkbox is unchecked then, only 1 sheet will be fetched at a time. 

10. Click the Import button. Once you have completed importing data, the Visual Pipeline builder opens and you can start applying transforms. You can also right-click the stage and choose the Prepare data option and prepare your data in the DataPrep Studio page. Click here to know more about the transforms.



11. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.


12. After adding a destination to the pipeline, you can try executing your pipeline using a manual run at first. Once you make sure manual run works, you can then set up a schedule to automate the pipeline. Learn about the different types of runs here.

While configuring the Schedule, Backfill, Manual reload, Webhooks, or Zoho Flow, the import configuration needs to be mandatorily setup for all the sources. Without setting up the import configuration, the run cannot be saved. Click here to know more about how to set up import configuration.

13. After configuring a run, a pipeline job will be created at the run time. You can view the status of a job with the granular details in the Job summary. Click here to know more about the job summary.

Import configuration for FTP servers without using Databridge

While configuring the Backfill run and Manual reload, the import configuration needs to be mandatorily setup for all the sources. Without setting up the import configuration, the run cannot be saved.

Below is a snapshot from the backfill configuration.


 

How to import data from source? You can choose to Import source file, or Do not import data using this option.

 

Import source file


This option will import the source file during every run.




NotesNote: Import configuration is not applicable when File is the data source for Schedule, Webhook, and Zoho flow. But for Backfill and Manual reload, File upload option is provided for during import configuration.

Do not import data  

The data is imported only once. The second time, the rules get applied to the same data and get exported.  


Import configuration for FTP servers using Databridge

If you import data from local network, you can configure how to import and fetch incremental data from your local network using the below Import configuration options.

You can select the Click here link to set the import configuration.

Below is a snapshot from the backfill configuration.


 


How to import data from source?

Select the way you would like to import your data from the drop-down - Import all dataIncremental file fetchDo not import data.

 Import all data   

This option will import all available data that matches the file pattern.

 


 

File batch size: Specify the batch size when you choose to import all files. Files will be grouped by this number, based on their first modified time. These files will be imported as a single file and exported to the destination in every run.

Info: File batch size cannot be greater than 10. This option is available only when you import data from local network

 Incremental file fetch   

You can configure how to import and fetch incremental data from your local network using the Import configuration option. Incremental data import is a method used to import new or modified records after the previous data interval. 


 



 

Use the previously imported file if no new file is available: 

When there are no new files in the source during incremental import,

If the checkbox is checked: The last fetched files will be imported again.

If the checkbox is unchecked: The import will be skipped and no files will be imported.

 

Which file to import? You can choose to import All filesNewest file, or Oldest file using this option.

 All files   

This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.


Fetch based on: You can choose to select the time based on which the file must be sorted and imported - Modified time.


File batch size: Specify the batch size when you choose to import all files incrementally. Files will be grouped by this number, based on their first created or modified time in that specific data interval. These files will be imported as a single file and exported to the destination in every run.

InfoInfo: File batch size cannot be greater than 10. This option is available only when you import data from local network.
 

 Newest file   

This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.



 Oldest file   

This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.

 


 Do not import data   

The data is imported only once. The second time, the rules get applied to the same data and get exported.

 


How incremental sync works

You can configure how to import and fetch incremental data from your source using the Import configuration option. Incremental data import is a method used to import new or modified records in the specific data interval.

In incremental file fetch, when the pipeline is run, the new or modified files will be fetched. During every data interval, the file created or modified after the previous data interval is imported. If there is no new file in the local file path, no data will be imported, or the previously imported data will be fetched again according to the import configuration. In the next data interval, the file created or modified in that interval will be fetched and so on.

 

 


Incremental fetch for schedule run 

In incremental file fetch, when the pipeline is scheduled, the data interval of the first schedule will span from the previous one up to the current data interval. During this interval, all the new or modified files will be fetched. The second schedule will extend from the current interval to the next, with subsequent schedules following this pattern. 
 
For example, The schedule is set to 1 hour interval. The data source contains a total of 10 files, in which 5 files were uploaded an hour ago that matches the file pattern. Import configuration is set to import all files incrementally, and the batch size is set to 10.

 

 


When the first schedule runs, the 5 files created/modified in the last 1 hour will be fetched and exported as a single file.
 
In the second schedule, if 4 new files with the same file pattern are added in the source, only the 4 files will be fetched and exported as a single file based on the created/modified time in the source in that specific interval.
 
In the third schedule, if 13 new files are added to the source that matches the file pattern, then only the first 10 files will be fetched and exported as a single file based on the created/modified time in the source since the batch size is set as 10. The same logic will be applied for files during incremental fetch in the upcoming schedules and so on.

Click here to know about the schedule run.

Incremental fetch for backfill run

 
In an incremental file fetch, when a backfill run is configured for a pipeline, all incremental files within the specified data interval will be fetched.



For instance, the backfill is set from August 1st to 3rd with a daily job frequency. August 1st has 4 files, August 2nd has 7 files, and August 3rd has 13 files. The import configuration is set to fetch all files incrementally with a batch size of 10.

During the first backfill job, the 4 files created or modified on August 1st will be fetched and exported as a single file. In the second backfill job, the 7 files from August 2nd will be similarly processed and exported. In the third backfill job, the first 10 files from August 3rd will be fetched and exported as a single file.

Click here to know about the backfill run.
 

Incremental fetch for reload data

In an incremental file fetch, when a reload is configured for a pipeline, all new or modified files within the specified data interval will be fetched.


 
For instance, the reload is set between August 1st and 2nd.
August 1st has 3 files, August 2nd has 4 files. The import configuration is set to fetch all files incrementally with a batch size of 5 based on the oldest modified time.
During the reload job, the 3 files created on August 1st, the first 2 files created on August 2nd will be fetched and exported as a single file.

Click here to know about the reload.

SEE ALSO

How to incrementally import data from local files?

How to export data to FTP servers?

      Create. Review. Publish.

      Write, edit, collaborate on, and publish documents to different content management platforms.

      Get Started Now


        Access your files securely from anywhere

          Zoho CRM Training Programs

          Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

          Zoho CRM Training
            Redefine the way you work
            with Zoho Workplace

              Zoho DataPrep Personalized Demo

              If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

              Zoho CRM Training

                Create, share, and deliver

                beautiful slides from anywhere.

                Get Started Now


                  Zoho Sign now offers specialized one-on-one training for both administrators and developers.

                  BOOK A SESSION







                              Quick LinksWorkflow AutomationData Collection
                              Web FormsRetailOnline Data Collection Tool
                              Embeddable FormsBankingBegin Data Collection
                              Interactive FormsWorkplaceData Collection App
                              CRM FormsCustomer ServiceForms for Solopreneurs
                              Digital FormsMarketingForms for Small Business
                              HTML FormsEducationForms for Enterprise
                              Contact FormsE-commerceForms for any business
                              Lead Generation FormsHealthcareForms for Startups
                              Wordpress FormsCustomer onboardingForms for Small Business
                              No Code FormsConstructionRSVP tool for holidays
                              Free FormsTravelFeatures for Order Forms
                              Prefill FormsNon-Profit
                              Forms for Government
                              Intake FormsLegal
                              Mobile App
                              Form DesignerHR
                              Mobile Forms
                              Card FormsFoodOffline Forms
                              Assign FormsPhotographyMobile Forms Features
                              Translate FormsReal EstateKiosk in Mobile Forms
                              Electronic FormsInsurance
                              Drag & drop form builder

                              Notification Emails for FormsAlternativesSecurity & Compliance
                              Holiday FormsGoogle Forms alternative GDPR
                              Form to PDFJotform alternativeHIPAA Forms
                              Email FormsWufoo alternativeEncrypted Forms
                              Accessible FormsTypeform alternativeSecure Forms

                              WCAG

                                          Create. Review. Publish.

                                          Write, edit, collaborate on, and publish documents to different content management platforms.

                                          Get Started Now






                                                            You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                                                Manage your brands on social media

                                                                  Use cases

                                                                  Make the most of Zoho Desk with the use cases.

                                                                   
                                                                    

                                                                  eBooks

                                                                  Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho Desk.

                                                                   
                                                                    

                                                                  Videos

                                                                  Watch comprehensive videos on features and other important topics that will help you master Zoho Desk.

                                                                   
                                                                    

                                                                  Webinar

                                                                  Sign up for our webinars and learn the Zoho Desk basics, from customization to automation and more

                                                                   
                                                                    
                                                                  • Desk Community Learning Series


                                                                  • Meetups


                                                                  • Ask the Experts


                                                                  • Kbase


                                                                  • Resources


                                                                  • Glossary


                                                                  • Desk Marketplace


                                                                  • MVP Corner



                                                                    Zoho Sheet Resources

                                                                     

                                                                        Zoho Forms Resources


                                                                          Secure your business
                                                                          communication with Zoho Mail


                                                                          Mail on the move with
                                                                          Zoho Mail mobile application

                                                                            Stay on top of your schedule
                                                                            at all times


                                                                            Carry your calendar with you
                                                                            Anytime, anywhere




                                                                                  Zoho Sign Resources

                                                                                    Sign, Paperless!

                                                                                    Sign and send business documents on the go!

                                                                                    Get Started Now




                                                                                            Zoho TeamInbox Resources





                                                                                                      Zoho DataPrep Demo

                                                                                                      Get a personalized demo or POC

                                                                                                      REGISTER NOW


                                                                                                        Design. Discuss. Deliver.

                                                                                                        Create visually engaging stories with Zoho Show.

                                                                                                        Get Started Now









                                                                                                                            • Related Articles

                                                                                                                            • Import configuration for FTP servers

                                                                                                                              You can import data from FTP s using simple selection or advanced selection. Look into the following sections to learn about the import configuration of each selection type. How to import data from FTP servers without using Databridge? How to import ...
                                                                                                                            • Export data to FTP servers

                                                                                                                              You can export files to FTP servers using the following steps. To export data to FTP servers 1. Open an existing pipeline or create a pipeline from the Home Page, Pipelines tab or Workspaces tab. You can bring your data from 50+sources. 2. On the ...
                                                                                                                            • Import incremental data from local files

                                                                                                                              Incremental data fetch is a method used to import new files or modified files from a source. Zoho DataPrep helps you import incremental data from your local files using Zoho Databridge. In DataPrep, you can incrementally import various file types, ...
                                                                                                                            • Import data from FTP servers

                                                                                                                              Zoho DataPrep allows you to fetch data from remotely located FTP servers and from local network using Zoho Databridge. With this connector, you can streamline your ETL workflows, enable smooth data movement, and simplify data integration between FTP ...
                                                                                                                            • Import data from URLs

                                                                                                                              You can enter the URL to a CSV, TSV, JSON, HTML, XLS, XLSX or XML file. DataPrep also supports URL imports that require authentication. With this connector, you can streamline your ETL workflows, enable smooth data movement, and simplify data ...
                                                                                                                              Wherever you are is as good as
                                                                                                                              your workplace

                                                                                                                                Resources

                                                                                                                                Videos

                                                                                                                                Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                                                eBooks

                                                                                                                                Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                                                Webinars

                                                                                                                                Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                                                CRM Tips

                                                                                                                                Make the most of Zoho CRM with these useful tips.



                                                                                                                                  Zoho Show Resources