Very slow and long running simple querry
Morpheuss
1
Reputation point
Running this :
from azureml.opendatasets import NycTlcGreen
data = NycTlcGreen()
df = data.to_spark_dataframe()
Display 10 rows
display(df.limit(10))
Azure Synaps : 8 vcpu/64 go (3 nodes ) run for over 30 min / 40 min with no error, no results : had to stop it as the same querry on a box of ( 4 vcpu /16 Go got the results in less than 15 min).
Any idea about what I'm missing here ?
Thank for your help.