Pipeline builder is a place that allows the users to create data flows in DataPrep, with multiple data stages and various flow level transforms. Whenever a new pipeline is created, the pipeline builder page opens into view. The pipeline builder page lists all the relevant information about the pipeline. It includes details on the pipeline, data source details, dataset details, destination details, and options like undo and redo the pipeline, import data, version history, schedule, run, and pipeline options.

 

Let us take a look in detail.

Pipeline options 

You can use the following options associated with your pipeline from the ellipses icon.

 

 Overview 

Go to the ellipses icon from the pipeline builder, and click on the Overview menu. You can view the pipeline overview for details such as the pipeline name, created by, created on, last modified date and time, last modified user, workspace name, stages, data sources, destinations, shared users, schedule time, last job run, and next job run. You can also rename, delete, and share with support from the pipeline overview.

 

 Version history 

Version history helps you keep track of all the actions done in a pipeline. You can view the list of the published older versions of the pipeline and view which version is currently live.

Note: If you restore a version, modify it, and then switch to another version without making your changes live, the initial version will be lost and won’t appear in the version history. Drafts are not saved in the version history unless they are made live.

 

The history view by default displays all actions of a pipeline in chronological order. You can use filters to view the history by version types, user, or date as needed.

 
Draft 

 

 Job history 

 

Go to the ellipses icon from the pipeline builder, and click on the Job history option to view all the jobs of a particular pipeline.

Click on a job in the Jobs page to navigate to the Job summary. The Job summary shows the details of a job in three different tabs: Overview, Stages, and Output. Click here to know more.

 

 Personal data overview 

 

 

 Share pipeline

Enter the user email or group name to share with users or groups.

You can choose the Send email notification to users checkbox to notify the user through email.

 

You can also Pick users/groups from your previously shared contact list.

Click Share to share your pipelines with the users or groups.




Click the button if you want to stop sharing your pipeline with the user.

 

To create a new group

Click here to know more about sharing your pipeline.

Save as template

2. Click the Save as template option and choose Save as pipeline template. The pipeline will be saved as a pipeline template in the templates library.

You can also download the pipeline template and use it using the Download pipeline as .zdpt option.

3. In the Workspace tab, navigate to the Pipeline template and click Use now to create a new pipeline using the saved template.

Share with support

 

Click the Share with support option. When you share the pipeline with our support team:

• You grant full access to our support team to view and modify all the data in your pipeline, including columns with personal data. If you want to proceed, then click Share option.
  
  
  
  
  

• Our support team will automatically lose access after 2 weeks. If you want to withdraw the access before 2 weeks, you can manually revoke access using this menu option again.
  
  

 

Reload data

Move to trash

Use this option to delete and move your pipeline to trash. The pipelines that are deleted can be located under the trash tab in the Workspace page. You can choose to restore the entities or permanently delete them from the trash section. Items in the trash will be permanently deleted after 180 days.

Zoom in, Zoom out - You can zoom in and out of the pipeline using this option. The range is between 60% - 100%

 

Scroll back to content - If your pipeline moves out of view, click this to scroll back to the top content.

 

Undo and Redo 

Import data 

You can click on the Import data  icon at the top of the pipeline builder to bring data from multiple sources into the pipeline.

 

Elements in a pipeline

Change source

 1. Right click the source icon and choose Change source option. You will be redirected to the import page from where you can import data from any source as required.

Datasource details

 

2. Right click the data source icon and choose the Datasource details option or click the data source icon once to view the data source details.

Note: You can also rename the name of the file imported.

3. Click the icon or right click the stage linked to your data source and choose,

 

Add Stage - To add a new stage using the imported data. Stages are nodes created for processing data while applying data flow transforms. Every data imported from your data source will have a stage created by default.

 

Remove from pipeline - You can use this option to remove the imported data and the flow from the pipeline.

 

Dataset details

 

4. Click the stage icon once to view the dataset details - Ruleset, overview.

 

Overview of dataset: The overview shows the list of the name of the dataset created by, created on, last modified by, last modified, columns, rule count, data quality, and data preview.





You can also choose to Prepare Data to open the DataPrep Studio page and explore more data transforms.

Ruleset: Each transform applied on the dataset is stored in DataPrep as a rule, in order of their execution. The ordered list of these rules is called a Ruleset. You can access the Ruleset from the Ruleset pane, where you can add, edit, or remove rules that have been applied on the stage.

 

Stage options

Prepare data

 

Join

 

Append 

 

Pivot

 

Unpivot

 

Add stage

 

Add destination

Data destination is a place where you want to export your data to. It can be a cloud database, business applications like Zoho CRM, Zoho Analytics, etc. You can choose your preferred destination out of 50+ data destinations in DataPrep to export your prepared data to.

 

Overview

 

View ruleset

 

 

Remove from pipeline

 

Destination details

 

6. Click the destination once to view the details such as Connection details, Folder path, File export option, File name, File format, Delimiter, Text qualifier, Row Separator, File encoding, Compress as a .zip file, Personal data etc.

 

Right click the destination and you'll find the following option,

 

Edit destination - You can change the destination of your pipeline using this option. Click the Edit destination option and you will redirected to the Choose your destination dialog box, from where you can choose your preferred destination out of 50+ data destinations in DataPrep.

Remove destination - You can use this option to remove the destination added to your pipeline.