question

GrantJosh-5060 avatar image
0 Votes"
GrantJosh-5060 asked SaurabhSharma-msft commented

Unexpected behaviour with %%time magic command in Synapse spark

The Synapse documentation states that the %%time magic command can be used in Spark notebooks in Synapse. This does not seem to be the case when calling those notebooks from a pipeline. Here's what I'm experiencing, using PySpark notebooks in Synapse pipelines:

  1. If a notebook cell starts with %%time, it works in the notebook editor but the entire cell is bypassed when run from a pipeline. Bypassed, as in the output indicates that the prior cell runs, then the following cell, but the cell with %%time is just missing from the output!

  2. If a notebook cell contains %time (just a single %), the notebook editor highlights it as incorrect syntax, but it still runs OK. When run from a pipeline, the cell generates an error.

MS, is this a bug, or is the Synapse documentation wrong and %%time is not supposed to work?



azure-synapse-analytics
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@grantjosh-5060 Thanks for using Microsoft Q&A !!
I am able to reproduce this issue and checking internally with the products team and I believe you are talking about the below where cell having magic command is being excluded from the result -
- Notebook code -
78032-image.png

77970-image.png

Command executed successfully with %
78051-image.png


0 Votes 0 ·
image.png (30.4 KiB)
image.png (24.6 KiB)
image.png (10.5 KiB)

@SaurabhSharma-msft That's correct. This notebook shows that the cell is not just missing from the output, but it was never executed at all:

78374-image.png


78368-image.png


0 Votes 0 ·
image.png (18.9 KiB)
image.png (20.4 KiB)

... and changing the magic command from %%time to %time also fails in the pipeline, with a different error:

78394-image.png


0 Votes 0 ·
image.png (21.2 KiB)

@grantjosh-5060 Thanks for confirming. Sure, I have already started an internal discussion with the products team and I will share the updates as soon as I hear back.

0 Votes 0 ·

1 Answer

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

@grantjosh-5060 I have received confirmation from the products team that -

  • We currently do not support %%time and %time in synapse pipeline yet. Products team is working to update the documentations to make it more clear for all users.

  • Also, %time is a cell magic, the correct syntax is to put the cell statement in the same line with %time, like below:
    %time 2*4

    79250-image.png


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.


image.png (8.4 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@grantjosh-5060 Following up as I have not heard back from you. Did my answer solve your issue? If so, please mark as accepted answer. If not, please let me know how I may better assist.

0 Votes 0 ·

@grantjosh-5060 I have not heard back from you. Did my answer solve your issue? If so, please mark as accepted answer. If not, please let me know how I may better assist.

0 Votes 0 ·