question

RyanAbbey-0701 avatar image
0 Votes"
RyanAbbey-0701 asked PRADEEPCHEEKATLA-MSFT commented

Synapse Spark - withColumn results in zero rows...

Can someone explain how this is happening? I have a dataframe with 800+ rows... I add a new column based on an existing column and the resulting dataframe has 0 rows... this is existing logic that was working fine the previous plentiful runs and now doing this... I am missing the logic behind how it is happening, anyone with ideas?

126251-image.png



Does Microsoft actively modify functionality in existing Spark environments? I'm using 2.4... and I'm struggling to put a reason behind why such odd behaviour keeps popping up!

azure-synapse-analytics
image.png (14.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hi @RyanAbbey-0701,

Thanks for the question and using MS Q&A platform.

As per my repro, I'm able to see the expected output by running the above query.

126306-image.png

Could you please verify the result (no of rows) by using df = df.withColumn('SenderId', F.col('Provider')).show()?

126383-image.png



image.png (30.1 KiB)
image.png (111.4 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes, I would expect that to happen each and every time... so the question is, why wasn't it?

I restarted my spark cluster and it worked correctly thereafter which suggests an issue within the cluster... this isn't the first issue either, we're experiencing a variety of oddities which is pushing us to the point of reverting to DataBricks

0 Votes 0 ·

Hi @RyanAbbey-0701,

This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.

0 Votes 0 ·