question

AnantFuria-7056 avatar image
0 Votes"
AnantFuria-7056 asked AnantFuria-7056 edited

Creating a Hive external table with Azure Blob Storage credentials in Spark-sql

Hi, we are using our own Hadoop/Spark cluster where we are running some spark-sql queries.
My question is how can we set azure storage credential for a external table? instead of storing the credential in core-site.xml

For e.g.

CREATE EXTERNAL TABLE test_my_table ( t1 string, t2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE LOCATION
'wasbs://test-blob@storageAcccount.blob.core.windows.net/example/data'

This would give me an error
Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: org.apache.hadoop.fs.azure.AzureException org.apache.hadoop.fs.azure.AzureException: No credentials found for account
in the configuration, and its container test-blob is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.);

So how do I provide azure account key in create table statement? or in spark-sql ?


azure-storage-accountsazure-blob-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

deherman-MSFT avatar image
0 Votes"
deherman-MSFT answered

@AnantFuria-7056
Please check the threads here and here to see if it resolves your issues. There are also instructions available for doing this in HDInsight which also might be helpful to you.

Hope this helps. Let us know if you have further questions or issues.



Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AnantFuria-7056 avatar image
0 Votes"
AnantFuria-7056 answered AnantFuria-7056 edited

@deherman-MSFT - I did look through it, they all talk about environment level keys. What I am interested in is
for e.g.
I have 3 tables, and all 3 of them actually belong to different 3 storage account. That means the key needs to be table specific.

Table 1 --> wasbs://test-blob1@account1.blob.core.windows.net/example/data
Table 2 --> wasbs://test-blob2@account2.blob.core.windows.net/example/data
Table 3 --> wasbs://test-blob3@account3.blob.core.windows.net/example/data

And keys for account1 or 2 or 3 are different.


Sorry, I know this sounds confusing, but feel free to let me know if you need more details.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.