Load Data into a Power BI Streaming Dataset and Build a Dataflows Monitoring Report with Power BI
In this step-by-step tutorial, we will show you how to set up your own monitoring dashboard for all of your dataflows:
First, you will create a new streaming dataset in Power BI. This dataset will collect all the metadata from the dataflow run, and for every refresh of a dataflow, a record is added to this dataset. You can run multiple dataflows all to the same dataset. Lastly, you can build a Power BI report on the data to visualize the metadata and start monitoring the dataflows.
You can use this dashboard to monitor your dataflows' refresh duration and failure count. With this dashboard, you can track any issues with your dataflows performance and share the data with others.
Create a new streaming dataset in Power BI
Navigate to Power BI.
Follow these instructions to create a new streaming dataset in Power BI. Ensure you create a streaming dataset based on the API and add the historical data opt in.
In the right pane, enter the following values, and then select Create.
- Dataset Name: "Dataflow Monitoring".
- Value: "Dataflow Name", Data type: Text.
- Value: "Dataflow ID", Data type: Text.
- Value: "Refresh Status", Data type: Text.
- Value: "Refresh Type", Data type: Text.
- Value: "Start Time", Data type: Date and Time.
- Value: "End Time", Data type: Date and Time.
Create a dataflow
Create a Flow in Power Automate
- Navigate to Power Automate,
- Search for the template "When a dataflow refresh completes, output status into Power BI Streaming Dataset". If you encounter difficulty, see these instructions.
Customize the flow. Actions that require input from you will automatically be expanded.
The Dataflow Refresh trigger is expanded because you need to enter information on your dataflow:
- Group Type: Select Environment when connecting to Power Apps and Workspace when connecting to Power BI.
- Group: Select the Power Apps environment or the Power BI workspace your dataflow is in.
- Dataflow: Select your dataflow by name.
The Add Rows (Streaming Dataset) action is expanded because you need to enter your Power BI Workspace and Dataset:
- Workspace ID: Select the Power BI workspace which contains your streaming dataset.
- Dataset: Select the streaming dataset "Dataflows Monitoring" you previously created.
- Table: Select "RealTimeData"
Add dynamic values to the required fields.
For every required field, you need to add a dynamic value. This value is the output of the metadata of the dataflow run.
- Select the field next to Dataflow ID and then select the lightning button.
- Repeat this process for all required fields.
Save the flow.
Create a Power BI Report
- Navigate to Power BI.
- Navigate to the streaming dataset.
- Create your own report on top of this data.