Hi there
I need to look for a way to reduce the time that the pipeline is consuming. I've got a simple Pipeline, that has two data flow activities, that is taking a lot of time for processing only one register:
1) Sorce to Staging: Overall time 5m 10 sec - Processing time 11s 803ms
2) Staging to DWH: Overall time 5m 2 sec - Processing time 2s 757ms
I read that this is because every Data Flow will require between 5-7 min for the cluster startup time and that it is necessary to modify the TTL of the Azure IR.
These are my questions. If the Azure IR is modified, how will this affect those Pipelines that only have one Data Flow activity, are the going to experience any decrease in the execution time?. In my example pipeline, it is possible to modify the second Data Flow to implement a Stored Procedure, by doing this, what should happen with the execution time of the entire pipeline? And finally, what is the price of modify the TTL?
I do not have permission for setting these values or modify the ETL, that is why I'm asking, I would like to be sure about this before making any proposal.
Regards,