Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute

Krzysztof Przysowa 20

I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute.

Serverless compute does not support cluster level (compute scoped) libraries.
In databricks workflows, it is being done as "environments and libraries"
https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs#configure-environments-and-dependencies-for-non-notebook-tasks
I would like to be able to point to databricks workspace file location or volume file location for required library.

Please add this functionality.

PRADEEPCHEEKATLA-MSFT 79,376 Reputation points Microsoft Employee

2024-05-02T02:34:43.2433333+00:00

@Krzysztof Przysowa - Thanks for the question and using MS Q&A platform.

As per the official documentation: Configure environments and dependencies for non-notebook tasks - You can install Python libraries from workspace files, Unity Catalog volumes, or public package repositories.

Please do follow these steps to add a library when you create or edit a task:

Hope this helps. Do let us know if you any further queries.
PRADEEPCHEEKATLA-MSFT 79,376 Reputation points Microsoft Employee

2024-05-06T05:51:32.2633333+00:00

@Krzysztof Przysowa - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Krzysztof Przysowa 20 Reputation points

2024-05-07T17:06:59.4733333+00:00

@phemanth, apologies for the delay in my answer,

yes, the guide you have provided is for databricks workflows only (aka jobs created within the databricks workspace)
I am after jobs that are created from Azure Data Factory, how to specify the wheel file path in Environment and Libraries settings in ADF databricks activity?
I have tried to just add the path like /Volume/catalog/schema/volume/path/ to a library box of the ADF databricks activity but it is treated as cluster level library, which is restrictive especially in the view of using serverless compute in the future.

Do you have any view how to overcome this problem?
Is it a lack of support for new databricks functionality? Can we raise a but or feature request for it?
PRADEEPCHEEKATLA-MSFT 79,376 Reputation points Microsoft Employee

2024-05-08T06:46:04.15+00:00

@Krzysztof Przysowa - You are correct that the guide I provided is for Databricks workflows only.

Regarding your question, to specify the wheel file path in Environment and Libraries settings in ADF Databricks activity, you can use the "Python file" field to specify the path to the Python file that contains the code you want to execute.

You can then use the "Python script arguments" field to pass any arguments to the Python script. If your Python script requires any custom libraries, you can include them in the same directory as the Python file and reference them in your code.

If you need to use a custom library that is not in the same directory as the Python file, you can use the "init_scripts" field to specify a shell script that installs the required libraries.

Regarding your question about lack of support for new Databricks functionality, you can raise a feature request with Microsoft Azure support. They will be able to provide you with more information on whether this functionality is planned for future releases.

OR

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Do let us know if you any further queries.
PRADEEPCHEEKATLA-MSFT 79,376 Reputation points Microsoft Employee

2024-05-20T05:56:04.02+00:00

@Krzysztof Przysowa - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute