question

AbdulSattar-7128 avatar image
0 Votes"
AbdulSattar-7128 asked HimanshuSinha-MSFT commented

pyspark data frame for json

hello all ,
i am using MS-Azure databricks platform for python+spark ,

cluster = spark 3.1.1

kindly tell me how can i import json file in it

syntax that i have used now:


df = spark.read.json("FileStore/tables/sample1.json").show(truncate = False)

and it is giving me the error :

IllegalArgumentException: Path must be absolute: FileStore/tables/sample1.json

azure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AbdulSattar-7128,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @AbdulSattar-7128,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @AbdulSattar-7128 ,

Thanks for the ask and using the Microsoft Q&A platform .

I think you forgot to mount the container .

I have tested the below code and it is working . I am using the access key from the blob .

   dbutils.fs.mount(
   source = "wasbs://yourcontainer@ yourstorageaccount.blob.core.windows.net/",
   mount_point = "/mnt",
   extra_configs = {"fs.azure.account.key.yourstorageaccount.blob.core.windows.net":"acccess key from the portal"})
 df = spark.read.json("/mnt/HimanshuTest.json").show(truncate = False)
 dbutils.fs.unmount("/mnt")

Output
86990-image.png


Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members



image.png (111.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.