Hello,
I have a a pipeline with 1 dataflow. It takes between 2 and 3 minutes to run it and I do not understand why it takes so much time.
For instance, a run took 2 minutes 05.
The data flow inside it took 02 minutes 04.
There are 5 steps in my dataflow:
1) Take files from an azure storage container
2) Flatten the files
3) I do a derived column
4) I set the upsert for the database
5) I insert data to cosmos db
If I take a look to the detailed metrics, each step of the dataflow took 10s (with a cluster startup time to 1s 479ms for the dataflow).
I am already using an integration runtime with a TTL.
So do you have an idea on how my pipeline/dataflow can be so long while the detailed dataflow metrics are so short?