We have some scenario's where we need to include Python 3 in our ADF pipelines. Workloads are small (under 1 GB of data).
Now I have a few requirements.
From within the Python script I must be able to access my azure sql database (so do either support odbc (pyodbc) ? )
Packages (and dependencies) should be managed easily. (So far I tried adding packages to an azure runbook (python 3 preview) and I ended up downloading them manually and adding them one by one which is a poor solution.
What would you advise to do? Solutions beyond runbooks and batch service are also very welcome as long as they are supported in azure data factory.