question

GopinathRajee-8127 avatar image
0 Votes"
GopinathRajee-8127 asked seanma commented

Azure Databricks - Running Jobs in Ineractive Cluster vs JobsCluster

All,

Other than cost being one of the factors, Why would one want to use an Interactive Cluster (High Concurrency) to run nightly jobs instead of running them using JobClusters via Pools.

Thanks,
grajee

azure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered seanma commented

Hello @GopinathRajee-8127,

Thanks for the question and using MS Q&A platform.

Interactive clusters are used to analyze data collaboratively with interactive notebooks. Job clusters are used to run fast and robust automated workflows using the UI or API.

So, while in development phase, you will mostly use interactive cluster. On the other hand once you move to production where things are more in a automated way, job clusters comes in picture.

For more details, refer to Azure Databricks - Clusters.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If Interactive Cluster set as always running, nightly job will run a little faster than job cluster with pool, otherwise there will not have any differences but Interactive Cluster is more expensive than job cluster with pool


0 Votes 0 ·