Iterative development and debugging with Azure Data Factory

Azure Data Factory lets you iteratively develop and debug Data Factory pipelines.

For an eight-minute introduction and demonstration of this feature, watch the following video:

Iterative debugging features

Create pipelines and do test runs using the Debug capability in the pipeline canvas without writing a single line of code.

Debug capability on the pipeline canvas

View the results of your test runs in the Output window of the pipeline canvas.

Output window of the pipeline canvas

After a test run succeeds, add more activities to your pipeline and continue debugging in an iterative manner. You can also Cancel a test run while it is in progress.

Cancel a test run

When you do test runs, you don't have to publish your changes to the data factory before you select Debug. This is helpful in scenarios where you want to make sure that the changes work as expected before you update the data factory workflow.

More info about debugging

  1. The test runs initiated with the Debug capability are not available in the list on the Monitor tab. You can only see runs triggered with Trigger Now, Schedule, or Tumbling Window triggers in the Monitor tab. You can see the last test run initiated with the Debug capability in the Output window of the pipeline canvas.

  2. Selecting Debug actually runs the pipeline. So, for example, if the pipeline contains copy activity, the test run copies data from source to destination. As a result, we recommend that you use test folders in your copy activities and other activities when debugging. After you've debugged the pipeline, switch to the actual folders that you want to use in normal operations.

Setting breakpoints for debugging

Data Factory also lets you debug until you reach a particular activity on the pipeline canvas. Just put a breakpoint on the activity until which you want to test, and select Debug. Data Factory ensures that the test runs only until the breakpoint activity on the pipeline canvas. This Debug Until feature is useful when you don't want to test the entire pipeline, but only a subset of activities inside the pipeline.

Breakpoints on the pipeline canvas

To set a breakpoint, select an element on the pipeline canvas. A Debug Until option appears as an empty red circle at the upper right corner of the element.

Before setting a breakpoint on the selected element

After you select the Debug Until option, it changes to a filled red circle to indicate the the breakpoint is enabled.

After setting a breakpoint on the selected element

Next steps

Continuous integration and deployment in Azure Data Factory