question

KeatOoi-0760 avatar image
0 Votes"
KeatOoi-0760 asked SaurabhSharma-msft commented

Azure databricks s3 SQS - Unable to load AWS credentials from environment variable

Hi Experts,

I am getting an error when trying to read the event message from the AWS S3 SQS in Azure Databricks.

here is my code

%python
schema_message = "message string"
df = spark.readStream \
.format("s3-sqs") \
.option("queueUrl", SQSQueue) \
.option("region",awsRegion) \
.option("fileFormat", "json") \
.option("awsaccessKeyId ", awsAccessKey) \
.option("awssecretKey", awsSecretKey) \
.schema(schema_message) \
.load()

Error Message :

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider: profile file cannot be null, WebIdentityTokenCredentialsProvider:

Notes : The codes works fine if i set the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY variable values in the environment variables of the databricks cluster.


Any idea why the above codes is not working?





azure-databricks
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @KeatOoi-0760 ,
Thanks for the ask and using the Microsoft Q&A platform .
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.
Thanks
Himanshu

0 Votes 0 ·

Hello @KeatOoi-0760 ,

Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu

0 Votes 0 ·

Hi @KeatOoi-0760 ,
We have not heard back from you. Can you please let us know in case you need help creating a support ticket.

Thanks
Saurabh

0 Votes 0 ·

Hi @KeatOoi-0760 ,
Please let me know if you are still facing issues. In case, you have already found a solution, would you please share it here with the community ?

Thanks
Saurabh

0 Votes 0 ·

0 Answers