question

LuukvanVliet-7731 avatar image
0 Votes"
LuukvanVliet-7731 asked HimanshuSinha-MSFT commented

Azure runbooks or Azure batch for using Python 3 as part of data pipelines in Azure data factory?

Hello,

We have some scenario's where we need to include Python 3 in our ADF pipelines. Workloads are small (under 1 GB of data).

Now I have a few requirements.

  • From within the Python script I must be able to access my azure sql database (so do either support odbc (pyodbc) ? )

  • Packages (and dependencies) should be managed easily. (So far I tried adding packages to an azure runbook (python 3 preview) and I ended up downloading them manually and adding them one by one which is a poor solution.


What would you advise to do? Solutions beyond runbooks and batch service are also very welcome as long as they are supported in azure data factory.




azure-data-factoryazure-automationazure-batch
5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

Hello @LuukvanVliet-7731 ,
Thanks for the ask and using the forum .
I think in your pipeline you want to execute some Python script and if thats the intend I think you can use Azure databricks . You can create a cluster and install the packages and have a notebook where you can have the python code . You can initiliaze the cluster from ADF . I have tried this and it worked for me .

Going back to the point which you made , i think working with Azure batch should also click .

Thanks
Himanshu

· 2
5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image HimanshuSinha-MSFT ·

Hello @LuukvanVliet-7731 ,
We have not heard back from you on this and was just following up if you have any further queries .
Thanks
Himanshu

0 Votes 0 ·
LuukvanVliet-7731 avatar image LuukvanVliet-7731 HimanshuSinha-MSFT ·

Hello @HimanshuSinha-MSFT ,

I have implemented both solutions and both of them work fine!
THank you!

0 Votes 0 ·