SAS authentication does not work with ADSL gen2 with pyspark

SHYAMALA GOWRI 70

I tried creating SAS Token and tried using in psyspark code (Service Principal mode of authentication worked ) but it failed with following for SAS .

24/04/03 14:05:02 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: abfss://pyspark@sparkadlsiae.dfs.core.windows.net/people.csv.Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not found.

I had executed Pyspark as

pyspark --jars hadoop-azure-3.3.3.jar,hadoop-azure-datalake-3.3.3.jar,hadoop-common-3.3.3.jar

And my connection property were as follows:

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "SAS")

spark.conf.set("fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")

spark.conf.set("fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net", "<token>")

spark.read.csv("abfss://<CONTAINER>@<STORAGE ACCOUNT>.dfs.core.windows.net/<PATH>/<FILE>.csv")

Should i be adding more jars in the classpath to get this working

Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-04T06:10:55.1133333+00:00
Hi @SHYAMALA GOWRI

Thank you for reaching out to the community forum with your query.

As I understand you're facing an error message you provided, it seems that the SAS token provider class is not found. This could be due to a missing jar file in the classpath.

You have already added the hadoop-azure-3.3.3.jar, hadoop-azure-datalake-3.3.3.jar, and hadoop-common-3.3.3.jar jars to the classpath. However, I think you may need to add the azure-storage-8.6.6.jar and azure-storage-common-8.6.6.jar jars to the classpath as well.

You can download these jars from the Azure Maven repository or add them as dependencies in your project's pom.xml file if you are using Maven.

Here's an example of how to add the azure-storage and azure-storage-common dependencies in your pom.xml file:

<dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-storage</artifactId> <version>8.6.6</version> </dependency> <dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-storage-common</artifactId> <version>8.6.6</version> </dependency>

Once you have added these jars to the classpath, try running your PySpark code again and see if the issue is resolved.

I hope this helps! Let me know if you have any further questions.

SHYAMALA GOWRI 70

Many thanks for helping on this, i tried as per ur suggestion

pyspark --jars hadoop-azure-3.3.3.jar,hadoop-azure-datalake-3.3.3.jar,hadoop-common-3.3.3.jar,azure-storage-8.6.6.jar,azure-storage-common-12.24.1.jar

This is the code i am trying to execute

from pyspark.sql import SparkSession
#Get spark session
spark = SparkSession.builder.appName("Test Spark App").getOrCreate()
#Set config params described above
spark.conf.set("fs.azure.account.auth.type.sparkadlsiae.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.sparkadlsiae.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.sparkadlsiae.dfs.core.windows.net", "sv=2022-11-02&ss=bfqt&srt=co&sp=rwdlacupyx&se=2024-04-03T14:57:13Z&st=2024-04-03T06:57:13Z&spr=https&sig=A%B%C%3D")
spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "false")
#params
storage_account_name= "sparkadlsiae"
container_name = "pyspark"
file = "people.csv"
#Read from the adls location
path = "abfss://"+container_name+"@"+storage_account_name+".dfs.core.windows.net/"+file
spark.read.format("csv").load(path).show()

and it again fails with the same error, i was trying to locate the package for FixedSASTokenProvider ..I unzipped these jars and checked,but i dint find this class in the package

Smaran Thoomu 9,685 Microsoft Vendor

Hi @SHYAMALA GOWRI

It seems that the FixedSASTokenProvider class is not available in the hadoop-azure-datalake-3.3.3.jar file.

Instead, you can try using the org.apache.hadoop.fs.azurebfs.SASTokenProvider class as the SAS token provider type. You can modify your code as follows:

from pyspark.sql import SparkSession

# Get spark session
spark = SparkSession.builder.appName("Test Spark App").getOrCreate()

# Set config params described above
spark.conf.set("fs.azure.account.auth.type.sparkadlsiae.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.sparkadlsiae.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.SASTokenProvider")
spark.conf.set("fs.azure.sas.token.provider.sparkadlsiae.dfs.core.windows.net", "<SAS token>")
spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "false")

# Params
storage_account_name = "sparkadlsiae"
container_name = "pyspark"
file = "people.csv"

# Read from the adls location
path = "abfss://" + container_name + "@" + storage_account_name + ".dfs.core.windows.net/" + file
spark.read.format("csv").load(path).show()

Please replace <SAS token> with your actual SAS token.

I hope this helps! Let me know if you have any further questions.

SHYAMALA GOWRI 70

Thanks for the quick response , i tried with the following

from pyspark.sql import SparkSession

# Get spark session
spark = SparkSession.builder.appName("Test Spark App").getOrCreate()
# Set config params described above
spark.conf.set("fs.azure.account.auth.type.sparkadlsiae.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.sparkadlsiae.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.SASTokenProvider")
spark.conf.set("fs.azure.sas.token.provider.sparkadlsiae.dfs.core.windows.net", "sv=2022-11-02&ss=bfqt&srt=co&sp=rwdlacupyx&se=2024-04-03T14:57:13Z&st=2024-04-03T06:57:13Z&spr=https&sig=A%B%C%3D")
spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "false")
# Params
storage_account_name = "sparkadlsiae"
container_name = "pyspark"
file = "people.csv"
# Read from the adls location
path = "abfss://" + container_name + "@" + storage_account_name + ".dfs.core.windows.net/" + file
spark.read.format("csv").load(path).show()

and executed this with

pyspark --jars hadoop-azure-3.3.3.jar,hadoop-azure-datalake-3.3.3.jar,hadoop-common-3.3.3.jar,azure-storage-7.0.1.jar,azure-storage-common-12.24.1.jar

But i hit the similar issue again :

24/04/05 15:43:55 WARN FileSystem: Failed to initialize fileystem abfss://pyspark@sparkadlsiae.dfs.core.windows.net/people.csv: Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SASTokenProvider not found
24/04/05 15:43:55 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: abfss://pyspark@sparkadlsiae.dfs.core.windows.net/people.csv.
Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SASTokenProvider not found
	at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getSASTokenProvider(AbfsConfiguration.java:905)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1563)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:236)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:183)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)

Would be of great help if you can share the jar's required to execute the above code.

Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-05T11:02:32.6666667+00:00
Hi @SHYAMALA GOWRI

Thanks for following up and providing more details. It seems the org.apache.hadoop.fs.azurebfs.SASTokenProvider class is still not being found despite adding the suggested jars. Here are a couple of things to check:

Jar Version Compatibility:

The azure-storage-common jar version you're using (12.24.1) might not be compatible with the hadoop-azure-datalake version (3.3.3). Try using a version of azure-storage-common closer to 3.3.x. Refer to the official documentation for compatible versions: https://hadoop.apache.org/docs/stable/hadoop-azure-datalake/index.html

Conflicting Jar Versions:

There's a possibility of conflicts with other jars in your classpath. Try running your code with just the essential jars: hadoop-azure-3.3.3.jar, hadoop-azure-datalake-3.3.3.jar, and azure-storage-common (compatible version). Gradually add other jars if needed.

Class Availability:

Ensure the JARs are indeed accessible by PySpark. Verify the classpath configuration (--jars argument) and double-check that the JAR files exist in those locations.

Double-check the SAS token format for any errors. Ensure you're using the correct fs.azure.sas.token.provider.type property based on the available class (either FixedSASTokenProvider or SASTokenProvider).

I hope this helps! Let me know if you have any further questions.
SHYAMALA GOWRI 70 Reputation points

2024-04-05T11:37:57.5233333+00:00

i am getting the same error with

pyspark --jars hadoop-azure-3.3.3.jar,hadoop-azure-datalake-3.3.3.jar,azure-storage-7.0.1.jar

if these jar's are not there while pyspark loads up it shows that it could not load the jar. But it doesnt show those errors in this case.

Is there any other way to check if my SAS token is not having format errors ?
SHYAMALA GOWRI 70 Reputation points

2024-04-05T12:08:41.08+00:00

with same set of jars , the following oauth code works..I am having an issue only with SAS tokens

from pyspark.sql import SparkSession #Get spark session spark = SparkSession.builder.appName("Test Spark App").getOrCreate() #Set config params described above spark.conf.set("fs.azure.account.auth.type.sparkadlsiae.dfs.core.windows.net", "OAuth") spark.conf.set("fs.azure.account.oauth.provider.type.sparkadlsiae.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set("fs.azure.account.oauth2.client.id.sparkadlsiae.dfs.core.windows.net", "a2584bd0-4cb4-482a-a407-c475a40c394f") spark.conf.set("fs.azure.account.oauth2.client.secret.sparkadlsiae.dfs.core.windows.net","1") spark.conf.set("fs.azure.account.oauth2.client.endpoint.sparkadlsiae.dfs.core.windows.net", "https://login.microsoftonline.com/0d909434-4142-4034-8b85-691ead3660b6/oauth2/token") spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "false") #params storage_account_name= "sparkadlsiae" container_name = "pyspark" file = "people.csv" #Read from the adls location path = "abfss://"+container_name+"@"+storage_account_name+".dfs.core.windows.net/"+file spark.read.format("csv").load(path).show()
Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-08T03:34:22.4566667+00:00

@SHYAMALA GOWRI Regarding your question about checking if your SAS token has format errors, you can try generating a new SAS token and using it in your code to see if it resolves the issue. You can also try using the SAS token in other Azure services to see if it works.
SHYAMALA GOWRI 70 Reputation points

2024-04-22T16:23:00.7466667+00:00

I have validated all the above mentioned pointers yet facing the same issue. I believe the

TokenProvider class needs to be implemented and then added the spark class path? Is there any TokenProvider implementation that comes with Azure that we can use for

fs.azure.sas.token.provider.type
Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-23T11:36:57.8+00:00

@SHYAMALA GOWRI I agree that this issue looks strange, and I wasn't able to reproduce this issue. If you have a support plan, could you, please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan, please let us know here.
SHYAMALA GOWRI 70 Reputation points

2024-04-23T17:10:09.5133333+00:00

@Smaran Thoomu i am able to get this working after i built jar that contains an implementation of org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider and added it to the classpath.
Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-23T17:20:46.0166667+00:00

@SHYAMALA GOWRI Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.
Aleksandrovic Palma 5 Reputation points

2024-04-23T22:57:04.8033333+00:00

Hi @SHYAMALA GOWRI , I'm facing the same issues. Could you share with us your org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider implementation please?
Thanks in advance!

SHYAMALA GOWRI 70

@Aleksandrovic Palma i created a jar with the following code and added it to the classpath

package org.apache.hadoop.fs.azurebfs.sas;
import java.io.IOException;
import java.util.Objects;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
import org.apache.hadoop.security.AccessControlException;
class FixedSASTokenProvider implements SASTokenProvider {
    private Configuration configuration;
    @Override
    public String getSASToken(String accountName,
                              String fileSystem,
                              String path,
                              String operation)
        throws IOException, AccessControlException {
        Objects.requireNonNull(accountName, "accountName must not be null");
        String fixedTokenConfig = new StringBuilder("fs.azure.sas.fixed.token.").append(accountName)
                .append(".dfs.core.windows.net").toString();
        String fixedToken = configuration.get(fixedTokenConfig);
        if (fixedToken == null) {
            throw new AccessControlException("SAS token isn't defined.");
        }
        return fixedToken;
    }
    @Override
    public void initialize(Configuration configuration, String accountName)
        throws IOException {
        Objects.requireNonNull(configuration, "configuration must not be null");
        this.configuration = configuration;
    }
}

Aleksandrovic Palma 5 Reputation points

2024-04-24T18:59:50.7466667+00:00

Thanks @SHYAMALA GOWRI !
This solved the issue I had. I'm able to connect to our delta storage from my local spark instance.
Cheers!

Accepted answer

Smaran Thoomu 9,685 Reputation points Microsoft Vendor

2024-04-23T17:19:52.6933333+00:00

Hi @SHYAMALA GOWRI

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Issue: I tried creating SAS Token and tried using in psyspark code (Service Principal mode of authentication worked ) but it failed with following for SAS .

24/04/03 14:05:02 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: abfss://pyspark@sparkadlsiae.dfs.core.windows.net/people.csv.Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not found.

I had executed Pyspark as

pyspark --jars hadoop-azure-3.3.3.jar,hadoop-azure-datalake-3.3.3.jar,hadoop-common-3.3.3.jar

And my connection property were as follows:

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "SAS")

spark.conf.set("fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")

spark.conf.set("fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net", "<token>")

spark.read.csv("abfss://<CONTAINER>@<STORAGE ACCOUNT>.dfs.core.windows.net/<PATH>/<FILE>.csv")

Should i be adding more jars in the classpath to get this working.

Solution: I am able to get this working after i built jar that contains an implementation of org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider and added it to the classpath.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

I hope this helps!

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

SAS authentication does not work with ADSL gen2 with pyspark

0 additional answers