question

Morpheuss-3347 avatar image
0 Votes"
Morpheuss-3347 asked PRADEEPCHEEKATLA-MSFT commented

Very slow and long running simple querry

from azureml.opendatasets import NycTlcGreen

data = NycTlcGreen()
df = data.to_spark_dataframe()

Display 10 rows

display(df.limit(10))

run for over 40 min without ever ending : Conf : (8 vcpu /64 GO 3nodes).

Any help would be appreciated. Nothing in the Queue, no previous job, spark pool basic config.

Many thanks for any hint.

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @Morpheuss-3347,

Welcome to the Microsoft Q&A platform.

We haven't experienced the above behaviour using Synapse Apache Spark pools till date.

This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.

As per the test from my end on Synapse Apache Spark Pool: Medium (8 vCores/64 GB).

On a new cluster it took nearly 3mins 20 secs.

127833-image.png

On a running cluster which took just 15 secs.

127851-image.png

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


image.png (100.3 KiB)
image.png (104.1 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Morpheuss-3347,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @Morpheuss-3347,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·