Troubleshooting CSV file operations in Synapse Spark with R script

Question

I'm encountering an issue I can't solve on my own. I'm running an R script in Synapse Spark, and I want to save the results of that script into a CSV file in Azure Data Lake Storage (ADLS).

I've attempted to read a CSV file and then write it back into a separate CSV file, but it's not working as expected. Here's how I'm trying to read the CSV file:

dfcsv1 <- read.df('abfs://[my-container]@[my-storage-account].dfs.core.windows.net/test/testdata2.csv', 'csv', header="true")

Then, I convert it to a data frame:

df <- as.data.frame(dfcsv1)

And here's how I'm attempting to write it back:

df.write.mode("overwrite").csv("abfs://[my-container]@[my-storage-account].dfs.core.windows.net/test//testdata3.csv")

But I'm getting this error message:

Error: [1] "Error in base::parse(...): :1:27: unexpected symbol\n1: df.write.mode("overwrite").csv\n ^"

What should I do to resolve this?

Answer 1

@Pravalika-randstad - Thanks for the question and using MS Q&A platform.

It looks like the error message is indicating that there is a syntax error in your code. Specifically, it seems that there is an issue with the line where you are attempting to write the data frame to a CSV file.

The error message suggests that there is an unexpected symbol in the code. One possible issue is that you have an extra forward slash in the file path. Try removing the extra forward slash in the file path and see if that resolves the issue.

Here's the corrected code:

df.write.mode("overwrite").csv("abfs://[my-container]@[my-storage-account].dfs.core.windows.net/test/testdata3.csv")

This should write the data frame to a CSV file in the specified location. If you continue to encounter issues, please let me know and I can help you troubleshoot further.

Hope this helps. Do let us know if you any further queries.