Hello,
I am using Azure Synapse Analytics and I have set up 1 single Spark Pool.
I created 4 notebooks in Synapse and would like to schedule them to run in one single ADF pipeline.
The problem is that it takes about 4-5 minutes to run each notebook in ADF totaling about 20 minutes.
When I run the notebooks through synapse studio their run time is 30-60 seconds each.
The reason why the notebooks take so long to run in ADF is because the Spark Cluster restarts in each of the ADF Notebook activities.
Is there a way in ADF to start up the Synapse spark pool once and then run each note book on the already started spark pool?
Thanks,
Courtney

