Run databricks notebook on a job cluster which is created from a docker image, in Azure data factory

Admin-Bin Wang 1 Reputation point
2021-09-14T04:43:31.663+00:00

In databricks workspace, I can create an interactive cluster based on a docker image from container registry.

However when running my ADF pipeline which executes ADB notebook, I want to use job cluster, based on the same docker image.

How do I do that?

Thank you.
Best Regards,
Bin Wang

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,624 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee
    2021-09-15T04:15:47.387+00:00

    Hello @Admin-Bin Wang ,

    Welcome to the Microsoft Q&A platform.

    You can choose existing interactive cluster to use the cluster created from a docket image in Azure Data Factory.

    From Azure Data Factory you can submit the pipelines to three types of clusters.

    New Job Cluster – Automated cluster to run fast and robust automated jobs. (You can select the cluster version, node type, python version and worker options) – which is similar to creating a new cluster in Azure Databricks workspace.

    132253-image.png

    Existing interactive cluster – You can use the existing interactive cluster from your Azure Databricks workspace by choosing from existing clusters.

    132254-image.png

    Existing instance pool – You can use the existing instance pool from your Azure Databricks workspace by choosing from existing pools.

    132203-image.png

    For more details, refer to Run a Databricks notebook with the Databricks Notebook Activity in Azure Data Factory.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
  2. Vigneshwaran K 0 Reputation points
    2023-08-03T17:02:14.8133333+00:00

    @PRADEEPCHEEKATLA-MSFT

    If the interactive is in stopped state because of auto terminate , will it get started automatically by "Notebook activity" ?

    0 comments