Very slow and long running simple querry

Morpheuss 1 Reputation point
2021-08-30T08:44:44.14+00:00

Running this :

from azureml.opendatasets import NycTlcGreen

data = NycTlcGreen()
df = data.to_spark_dataframe()

Display 10 rows

display(df.limit(10))

Azure Synaps : 8 vcpu/64 go (3 nodes ) run for over 30 min / 40 min with no error, no results : had to stop it as the same querry on a box of ( 4 vcpu /16 Go got the results in less than 15 min).

Any idea about what I'm missing here ?

Thank for your help.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,422 questions
{count} votes