Incremental data fetch

Only modified and new data

To import the modified and new data incrementally from the last imported time, select Only modified and new data option from the drop-down.

Info: The modified and new data will be fetched using the Modified_Time and Added_Time columns from the last imported time.

Use existing data if new data is not available: 

When there are no new data in the source during incremental import,

If the checkbox is checked: The last fetched data will be imported again.

If the checkbox is unchecked: The import will be skipped and no data will be imported.

Only new data

To import the new data incrementally from the last imported time, select Only new data option from the drop-down.

Info: The new data will be fetched using the Added_Time columns from the last imported time.

Settings

Stop export if data has invalid values: Enabling this will stop the export when prepared data still has invalid values.

Order exports

Note: This option will be visible only if you have added more than one destination in your pipeline.

To rearrange the order of your export destinations

1) Click the Order exports toggle

Note: Click the Edit order link if you want to rearrange the order again.

11. After you configure the schedule or any other run, click Save to execute the run. This will start the pipeline.

Each run is saved as a job. When a pipeline is scheduled, the data will be fetched from your data sources, prepared using the series of transforms you have applied in each of the stages, and then data will be exported to your destination at regular intervals. This complete process is captured in the job history.
12. To go to the jobs list of a particular pipeline, go to the  ellipses icon in the pipeline builder, and click on the Job history menu to check the job status of your pipeline.

13. Click the required job ID in the Jobs history page to navigate to the Job summary of a particular job.

The Job summary shows the history of a job executed in a pipeline flow. Click here to know more.

Info: You can also view the status of your run later on the Jobs page.

Note: If you make any further changes to the pipeline, the changes are saved as a draft version. Choose the Draft option and mark your pipeline as ready for the changes to reflect in the run.

After you set your schedule or any other run, you can choose to Pause schedule or Resume schedule, Edit schedule and Remove schedule using the Schedule Active option in the pipeline builder.