question

MikeBrewer-2545 avatar image
0 Votes"
MikeBrewer-2545 asked PRADEEPCHEEKATLA-MSFT commented

Trouble with spark code in Notebook, 'str' object has no attribute 'option

I can't debug this. I copied it from a Databricks video, so maybe it does not transfer over????

 import sys
 from pyspark import SparkContext
 from pyspark.sql import SparkSession
 from pyspark.sql.types import *
    
 spark = SparkSession \
     .builder \
     .appName("Validate") \
     .getOrCreate
    
 custumSchema = StructType([
             StructField("A#", IntegerType(), True),
             StructField("FirstName", StringType(), True),\
             StructField("LastName", StringType(), True),\
             StructField("DOB",DateType(), True),\
             StructField("Gender", StringType(), True ),\
             StructField("corrupt_record", StringType(), True )\
             ]) 
    
    
    
 df = spark.read\
     .format='csv' \
     .option("badRecordsPath", 'abfssXXXXXXCSV/BadCSV/*.csv')\
     .option("mode", "PERMISSIVE")\
     .options(header='true', delimiter=',',) \
     .option("columnNameOfCorruptRecord", "corrupt_record") \
     .load('abfss://synapXXXXXXXCSV/*.csv', schema = custumSchema)
    
 df.show()


Here's my error



AttributeError Traceback (most recent call last)
/tmp/ipykernel_7707/2196056541.py in <module>
21
22 df = spark.read\
---> 23 .format='csv' \
24 .option("badRecordsPath", 'abfss://synapseqadatalakegen2fs@synapseqadatalakegen2.dfs.core.windows.net/DataLakehouse/CSV/BadCSV/*.csv')\
25 .option("mode", "PERMISSIVE")\

AttributeError: 'str' object has no attribute 'option'



I'm stumped on this one. Have tried many versions to fix...any help appreciated,

Thanks, Mike






azure-synapse-analyticsazure-databricksdotnet-ml-big-data
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @MikeBrewer-2545,

Just checking in to see if the below answer provided by @DillonJS helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

DillonJS avatar image
0 Votes"
DillonJS answered DillonJS edited

Have you tried .format("csv")? I tried your method and got the same error, and when I changed to .format("csv") in databricks it worked.



If this helps please mark as correct answer.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.