Import incremental data from local files
Incremental data fetch is a method used to import new files or modified files from a source. Zoho DataPrep helps you import incremental data from your local files using Zoho Databridge.
Note :
1. DataPrep also supports files without any extension or files in plain text format. However, binary files cannot be parsed during import; users will have to manually parse the file into one of the supported formats listed above.
2. Maximum local file size supported during import is 100MB. You can find the details on other technical limitations here.
To start with import
Info: You can also click the Import data
2. Choose the Files option from the Choose your data source to import the local files. You can also click the Files category from the left pane and select the Files option.
Import incremental data
4. Choose an active Databridge and provide the following details:
Folder path : The folder path where you want to search for files. Eg. D:\DataPrep\Datasets
Include subfolders : You can select this checkbox if you want to include subfolders while searching for a file.
File pattern : The pattern used to match the file names in the specified location. This supports regex type matching. You can also use the pattern, ".*" to match any file in the path specified.
Note: The file pattern match is a simple regex type match. For example, to fetch files with names such as Sales_2022.csv,Sales_2023.csv, Sales_2024.csv , you can input the pattern Sales_.*
Similarly to fetch files such as PublicData1.csv , PublicData2.csv , PublicData3.csv , use Public.*
If you want to import a single file, then specify the pattern using the exact file name.
Eg: leads_jan_2022.*
File password : Enter the password if the file is password protected.
Note: If this checkbox is unchecked then, only 1 file will be fetched at a time.
Note: If this checkbox is unchecked then, only 1 sheet will be fetched at a time.
6. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.
While configuring the Schedule, Backfill, Manual reload, Webhooks, or Zoho Flow, the import configuration needs to be mandatorily setup for all the sources. Without setting up the import configuration, the run cannot be saved. Click here to know more about how to set up import configuration.
8. After configuring a run, a pipeline job will be created at the run time. You can view the status of a job with the granular details in the Job summary. Click here to know more about the job summary.
Import configuration for local files without using Databridge
You can import data from local files using Zoho Databridge. Look into the following sections to learn about the import configuration.
How to import data from source? You can choose to Import source file, or Do not import data using this option.
Import source file
This option will import the source file during every run.
Note: Import configuration is not applicable when File is the data source for Schedule, Webhook, and Zoho flow. But for Backfill and Manual reload, File upload Do not import data
The data is imported only once. The second time, the rules get applied to the same data and get exported.
Import configuration for local files using Databridge
If you import data from local network, you can configure how to import and fetch incremental data from your local network using the below Import configuration options.
Import all data
This option will import all available data that matches the file pattern.
Incremental file fetch
You can configure how to import and fetch incremental data from your local network using the Import configuration option. Incremental data import is a method used to import new or modified records after the previous data interval.
Use the previously imported data if no new data is available:
During incremental import,
- If the checkbox is checked: When there is no new data in the source, the last fetched data will be imported again.
- If the checkbox is unchecked: When there is no new data in the source, the import will fail and no files will be imported. This will, in turn, cause the entire pipeline job to fail.
Which file to import? You can choose to import All files, Newest file, or Oldest file using this option.
All files
This option will import all the files that match the given file pattern only in the specific data interval based on the Fetch based on field.
Info: File batch size cannot be greater than 10. This option is available only when you import data from local network.Newest file
This option will import the latest file that matches the given file pattern in the specific data interval based on the Fetch based on field.
Oldest file
This option will import the oldest file that matches the given file pattern in the specific data interval based on the Fetch based on field.
Do not import data
The data is imported only once. The second time, the rules get applied to the same data and get exported.
How incremental sync works
Incremental fetch for schedule run
Click here to know about the schedule run.
Incremental fetch for backfill run
Click here to know about the backfill run.
Incremental fetch for reload data
Click here to know about the reload.
Access your files securely from anywhere
Zoho CRM Training Programs
Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.
Zoho DataPrep Personalized Demo
If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.
- Zoho Workerly Home
- Forums
- Connect With Us:
- Zoho Recruit Home
- Forums
- Connect With Us:
Create. Review. Publish.
You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.
Zoho Sheet Resources
Zoho Forms Resources
Zoho Sign Resources
Sign, Paperless!
Zoho TeamInbox Resources
Design. Discuss. Deliver.
Related Articles
Import configuration for local files
You can import data from local files using Zoho Databridge. Look into the following sections to learn about the import configuration of each selection type. How to import data from local files without using Databridge? How to import data from local ...Import incremental data from FTP servers
Incremental data fetch is a method used to import new files from a source. Zoho DataPrep helps you import incremental data from your FTP servers using Zoho Databridge. In DataPrep, you can incrementally import various file types from FTP, including ...Import data from YugaByte
You can import data from the YugaByte database into Zoho DataPrep using Zoho Databridge. Click here to know about Zoho Databridge. Before looking into how to import data, let's take a look at the pre-requisites to connect to the Denodo database. ...Import data from Salesforce
Zoho DataPrep allows you to seamlessly bring in your sales data from Salesforce connector. You can import your sales data from Salesforce to perform data cleanup, data migration, and data backup in more than one way. With this connector, you can ...Import data from local databases
You can import data from the following local databases into DataPrep using Zoho Databridge: MySQL MS SQL Server Oracle PostgreSQL Maria DB Pervasive SQL Sybase DB2 Exasol Sqlite Actian Vector Greenplum Denodo Progress OpenEdge YugabyteDB Microsoft ...