Pipelines are fundamental entities that allows users to create data flows in DataPrep, with multiple data stages and various flow level transforms. Template library is a place which contains pre-defined templates and all the saved pipeline templates. You can create a new pipeline from scratch using the pre-defined template or saved templates from the template library. The option to create a pipeline from a template is available in multiple places:

• Home page

• Pipelines page

• Workspaces page

  To create a pipeline using templates library in the home page   

1. You can create a new pipeline from the Home page. Click on the Create New drop-down icon and choose From templates library.

 To create a pipeline using Pre-defined templates 

b. Select one of the templates to view the details of the template, including a description. From here, you can click Use Template.

c. You can choose to create this pipeline in the existing workspace or new workspace. You can also choose to load template with a sample data by selecting the Load template with sample data and explore the rules and transforms. Click Next.

d. A pipeline gets created in the chosen workspace with the template. The Pipeline builder will open into view.

Note: If you choose to load sample data, you can just configure your destination and start exploring how to apply transforms.

 To create a pipeline using Saved template 

a. Click the Saved templates tab.



b. The saved pipeline templates are listed here. Click the required template.

Note: You can also rename, share and move template to trash using the  icon.

c. You can choose to create this pipeline in the existing workspace or new workspace and click Next.

d. A pipeline gets created in the chosen workspace with the template. The Pipeline builder will open into view.

2. Authenticate the data sources and destinations in your pipeline.

You can also choose a different source and destination. For example, if Files is the source in the template and you would like to change the source to OneDrive, then you can select OneDrive when you authenticate your source. But please make sure the schema of the new data you bring into matches with the rules applied in your pipeline template.

If the data model mismatches after connecting the data from a different source, then it can be corrected by going into the ruleset and changing around the rules.

Note: You can also click on the Import data  icon at the top of the pipeline builder to bring data from multiple sources.

After data is imported, you will be redirected to the pipeline builder where you can see your data source and a stage linked to it.

Stages are nodes created for processing data while applying data flow transforms. Every data imported from your data source will have a stage created by default.

4. You can right-click the stage and apply the data flow transforms.

5. You can also choose to Prepare Data to open the DataPrep Studio page and explore more data transforms. Click here to know the various elements in the pipeline builder.

Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

6. Data destination is a place where you want to export your data to. It can be a cloud database, business applications like Zoho CRM, Zoho Analytics, etc. You can choose your preferred destination out of 50+ data destinations in DataPrep to export your prepared data to.

7. Now that you have added a destination, you may want to try executing your pipeline using the Run button.

8. Each run is saved as a job. When a pipeline run is executed, the data fetched from your data sources will be prepared using the series of transforms you have applied in each of the stages, and then data will be exported to your destination. This complete process is captured in the job history. Once you make sure manual run works, you can then set up a schedule to automate the pipeline. Learn about the different types of runs here.

9. Go to the  ellipses icon, and click on the Job history menu to check your job status. All runs captured as jobs will be saved under the Jobs page.

Info: You can also view the overall jobs status of all the pipelines under the Jobs tab.

 
To create a pipeline using templates library in the pipeline page    

 You can create a new pipeline from the Pipelines tab. Click on the Create New drop-down icon and choose From templates library.



Other steps to build and execute the pipeline can be found here.

 To create a pipeline using template library within a Workspace

  
1. In the Workspaces page, you can create a new pipeline from the template library within an existing workspace.

2. Click on the Create New drop-down icon and choose From templates library.

Other steps to build and execute the pipeline can be found here.

 To save a pipeline template 

1. Open the required pipeline and click the ellipses icon in the Pipeline canvas page.

2. Click the Save as template option and choose Save as pipeline template. The pipeline will be saved as a pipeline template in the templates library.

3. In the Workspace tab, navigate to the Pipeline template and click Use now to create a new pipeline using the saved template.

How to create a new pipeline from scratch?