question

Jothi-6650 avatar image
0 Votes"
Jothi-6650 asked ·

Azure Databricks - Split column based on special characters in Databricks

I have a column in my csv file that possibly has value in below formats.
"Q1_1_Value-10_counts"
"Value_10_counts"
"Q1_1
1_value_yes"

This has to be split as below respectively
"Value_-_10_counts"
"Value_10_counts"
"value_yes"

When I try to split based on , i get error if second format arrives. So, i put an 'IF' condition as below.
Below is the python code I used.
if '
' in df['column'].values:
df[['column','newcolumn']] = df['column'].str.split('__', expand=True)
else:
df['newcolumn'] = df['column']

code enters else condition even column has ''
tried below also - getting result as 'False' only
exists = '
' in df.column
print(exists)

found = df[df['column'].str.contains('__')]
display(found)

Kindly help, how to handle this? any of you faced similar situation
Or, let me know if it will work if split by Q first, and then check for first alphabet (which will ignore initial numbers and __)

azure-databricks
· 2
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Jothi-6650 ,


I want to make sure that I really have an understanding of what you’re telling me.


You mentioned that " When I try to split based on , " , can you please let me know know where is they "," in the row which you mentioned ?


One of the desired output is mentioned "Value_-_10_counts",but the output input i think is "Q1_1_Value-10_counts" , Just wanted to check if the desired output should be "Value-_10_counts" ?


If you can share on row as you see in the .csv ( with dummy date ) that will be a big help /


Thanks Himanshu


0 Votes 0 ·

Hello @Jothi-6650,


We have not heard back from you on this . Can you please some test data as requested below .


Also if case if you have a resolution please do share that with the community as it will help other community members .


Thanks & stay safe


Himanshu




0 Votes 0 ·

0 Answers