question

AdminBinWang-9659 avatar image
0 Votes"
AdminBinWang-9659 asked PRADEEPCHEEKATLA-MSFT edited

Run databricks notebook on a job cluster which is created from a docker image, in Azure data factory

In databricks workspace, I can create an interactive cluster based on a docker image from container registry.

However when running my ADF pipeline which executes ADB notebook, I want to use job cluster, based on the same docker image.

How do I do that?

Thank you.
Best Regards,
Bin Wang

azure-data-factoryazure-databricksazure-container-registry
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AdminBinWang-9659,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @AdminBinWang-9659,

Welcome to the Microsoft Q&A platform.

You can choose existing interactive cluster to use the cluster created from a docket image in Azure Data Factory.

From Azure Data Factory you can submit the pipelines to three types of clusters.

New Job Cluster – Automated cluster to run fast and robust automated jobs. (You can select the cluster version, node type, python version and worker options) – which is similar to creating a new cluster in Azure Databricks workspace.

132253-image.png

Existing interactive cluster – You can use the existing interactive cluster from your Azure Databricks workspace by choosing from existing clusters.

132254-image.png

Existing instance pool – You can use the existing instance pool from your Azure Databricks workspace by choosing from existing pools.

132203-image.png

For more details, refer to Run a Databricks notebook with the Databricks Notebook Activity in Azure Data Factory.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


image.png (140.5 KiB)
image.png (143.1 KiB)
image.png (141.4 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AdminBinWang-9659,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·