Is there a way to speed up a pipeline that is executing a notebook on spark pool?
The pipeline itself executes for 30-35 seconds, but cluster start takes around 3 minutes, which is totally inefficient.
For example, in Databricks jobs you can execute a job using cluster which is already running and you dont need to wait for the cluster to start. How to achieve the same using Synapse?
Thanks in advance,
Yaro