Can someone explain how this is happening? I have a dataframe with 800+ rows... I add a new column based on an existing column and the resulting dataframe has 0 rows... this is existing logic that was working fine the previous plentiful runs and now doing this... I am missing the logic behind how it is happening, anyone with ideas?

Does Microsoft actively modify functionality in existing Spark environments? I'm using 2.4... and I'm struggling to put a reason behind why such odd behaviour keeps popping up!

