Hello,
We have a scenario that we cannot solve with Azure Data Factory, therefore we need to schedule a python script that pulls data from multiple REST apis, transforms the data and inserts it into azure sql database.
What is a best practise to insert a pandas dataframe into azure sql? Currently I am using sqlalchemy (after installing ODBC drivers in azure databricks) but is very slow. I have heard of SparkSQL and Pyspark, but are they really speeding up insertion of records in azure sql database? Any references to a solution that fits my needs?