question

GoelAkanksha-9319 avatar image
0 Votes"
GoelAkanksha-9319 asked PRADEEPCHEEKATLA-MSFT commented

FileNotFoundException when using abfss to list files in Azure Databricks!

Hi team,

I am trying to connect to ADLS2 using hadoop configurations:
15557-image.png


But when I am trying to use FS commands to list all the files on the path, i am getting File not found exception:

import org.apache.hadoop.fs.{FileSystem, Path}
val fs = FileSystem.get(sparkSession.sparkContext.hadoopConfiguration)
val listOfFiles = fs.listFiles(new Path("abfss://conhw@saditdhprocessed.dfs.core.windows.net/clickstream/driverenum/temp"), false)

I am getting this error:
15558-image.png


azure-databricksazure-data-lake-storage
image.png (159.5 KiB)
image.png (109.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @GoelAkanksha-9319,

Welcome to the Microsoft Q&A platform.

From the error message, java.lang.Exception: Could not find File/xxxxxxxx/clickstream/driverenum/temp", looks like you are passing incorrect path.

Make sure the path exists and try the steps mentioned below and do let us know the status.

OR

You can use this method for access ADLS Gen2 files directly using the RDD API as shown below.

15851-image.png

Hope this helps. Do let us know if you any further queries.


Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.



image.png (131.2 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @PRADEEPCHEEKATLA-MSFT,

Yes, the path is correct.


As, I am using IntelliJ as an IDE, Can you help me with the maven dependency required to use "dbutils" commands in spark scala code?

I am not able to run dbutils commands in Azure.

0 Votes 0 ·

Hey @GoelAkanksha-9319,

Unfortunately, you cannot use Databricks Utilities commands in IntelliJ IDE.

For more details, checkout Azure Databricks Connect - limitations.


0 Votes 0 ·