How to import data from cloud storage services?

Import data from cloud storage services




DataPrep supports import from the following cloud storage services.
  1. Google Drive
  2. One Drive
  3. Box Storage
  4. Dropbox
  5. Zoho WorkDrive
  6. Amazon S3
With these connectors, you can streamline your ETL workflows, enable smooth data movement, and simplify data integration between various cloud storage services and Zoho DataPrep. 

To import data from a cloud storage service

1. Open an existing pipeline or create a pipeline from the Home Page, Pipelines tab, or Workspaces tab and click the Add data option.
InfoInfo: You can also click the Import data  icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.



2. Choose the Cloud storage category from the left pane and click the required cloud storage service. You can also search cloud storage services in the search box. 
      


NotesNote: If you have already added a connection earlier, click the Saved connections category from the left pane and proceed to import. To learn more about Saved connections, click here.

2. Select an account from the saved connections or connect a new account using the Add new option.



3. Authenticate your Google account. You will need to authorize DataPrep to access your files when you do this for the first time. 
Notes
Note: The connection configuration will be saved for importing from the cloud storage in the future. Credentials are securely encrypted and stored.

4. Select the files you want to import.  You can also select files from the Shared with me tab.

5. Click the Import button. You will be taken to the Pipeline builder page right away, and the import will run in the background.


You can incrementally import the files using the Advanced selection link. You can also use this option for bulk import of files based on the file pattern. Click here to know more about advanced selection from GoogleDrive and incremental fetch.

For other cloud storage options, authenticate the cloud service and authorize DataPrep to access the files. 
Notes
Note: In the case of Zoho WorkDrive, all folders and files will be directly fetched from your account using which you have logged into Zoho DataPrep. Here's a snapshot below.



7. If any files require additional input, an Action Required indicator will appear in the pipeline builder header. Click Action Required indicator to open the Import panel, which lists each file that needs your attention. For each file:
  • If the file is password protected, enter the password in the field provided.
  • If the file contains multiple sheets, select the sheets you want to import and click Import.



Notes
Note: Supported file formats are CSV, TSV, JSON, HTML, XLS, XLSX, and XML.
InfoIf the file is HTML, XLS, or XLSX, you can click the Preview option to view a sample of the data.

8. Once the import is complete, you can start applying transforms to the ETL pipeline. You can also right-click the stage and choose the Prepare data option to prepare your data in the DataPrep Studio page. Click here to learn more about the transforms.



9. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

NotesNote:  After adding a destination to the ETL pipeline, you can try executing your pipeline using a manual run at first. Once you make sure the manual run works, you can then set up schedule to automate the pipeline and data movement. Learn about the different types of runs here.
Info
You can also import files from shared folders in Google Drive, Dropbox, Box, OneDrive, and Zoho WorkDrive.
Alert
Important notes on Amazon S3 
Prior to connecting your Amazon S3 data with Zoho DataPrep, you need to enable the permissions below from your Amazon S3 account:
1. ListAllMyBuckets
2. Get BucketLocation
3. ListBucket
4. GetObject
Please refer to Amazon S3 help pages to provide these permissions.