Azure DataBricks ADF connectivity issue Delta Tables

Rish Shah 126

Hi,

I am trying to pass a simple Select SQL query to query a DeltaTable in DataBricks using copy activity. My sink is SQL table.

I am able to preview the results of my query and am happy with that. However I am getting error when running the ADF Pipeline.. Please help:

Error code: 2200
Failure type: User configuration issue
Details: ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configuration.

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-05-11T18:54:49.303+00:00

Hello @Rish Shah ,
Thanks for using Q&A ,
While trying to repro i got the below error ,Can you please let me confirm you are seeing something similar or different error at your end ?

ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container YYY in account XXXX.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration. Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container YYY in account XXXX.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration..

Thanks
Himanshu
Rish Shah 126 Reputation points

2021-05-11T18:57:54.927+00:00

No... The error message is as per my screenshot above.
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-05-11T23:28:38.587+00:00

Thanks @Rish Shah
Just wanted you to point out to the prerequisites here . Can you please check and let me know if the stesps are done at your end ?
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-05-12T18:36:07.17+00:00

Hello @Rish Shah ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
Rish Shah 126 Reputation points

2021-05-12T18:41:27.87+00:00

We are using adls gen 2 and are adhering to the prerequisite. Still we are getting same error message
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-05-18T22:18:41.653+00:00

Hello @Rish Shah ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu

Accepted answer

Rish Shah 126 Reputation points

2021-05-21T10:00:52.37+00:00

My colleague Siva was able to find a solution today. I am updating this in a hope that the solution will help someone:

Basically, we need to enable staging to copy data from Deltalake. We need to update the spark config (go in databricks, spark cluster, edit, advanced. Leave whatever is in there as is, plus add the below) with the staging storage account key as per the comment below:

Spark Config: Edit the Spark Config by entering the connection information for your Azure Storage account .
This will allow your cluster to access the lab files. Enter the following:

spark.hadoop.fs.azure.account.key.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net <ACCESS_KEY>, where <STORAGE_ACCOUNT_NAME> is your Azure Storage account name, and <ACCESS_KEY> is your storage access key.

Example: spark.hadoop.fs.azure.account.key.bigdatalabstore.blob.core.windows.net HD+91Y77b+TezEu1lh9QXXU2Va6Cjg9bu0RRpb/KtBj8lWQa6jwyA0OGTDmSNVFr8iSlkytIFONEHLdl67Fgxg==
Please sign in to rate this answer.

1 person found this answer helpful.
Julian Breunung 22 Reputation points

2021-09-15T16:08:24.147+00:00

Just to add to this information: My findings have been that the Spark cluster needs to be configured to have access to both the source AND the target storage account if you are using a copy activity. In my case we are loading data from a databricks view and store it as parquet on another data lake.
After setting up the Spark Config to be able to access both unterlying storage accounts (we are using Service Principals) the error message went away.
Of course the Databricks cluster must also be able to connect with the target storage account from a networking point of view...

Mehdi Aziz Sidi Boumedine 1 Reputation point

2021-11-10T11:13:49.117+00:00

But we're using key vaults to avoid exposing the keys, isn't that risky to put it like that in the settings?

Julian Breunung 22 Reputation points

2021-11-10T11:32:38.347+00:00

You can create a KeyVault backed secret scope (https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes) and access the values via the following syntax in your spark config: {{secrets/<secret-scope-name>/<secret-name>}}

Mehdi Aziz Sidi Boumedine 1 Reputation point

2021-11-10T12:08:44.96+00:00

Hey, Julian. Secret Scope was already there and was working fine before trying to use the delta lake copy.

Julian Breunung 22 Reputation points

2021-11-10T13:09:30.987+00:00

Of course you can use an existing secret scope as well...
Sign in to comment

1 additional answer

brajesh jaishwal 26 Reputation points

2021-05-17T21:03:42.117+00:00

Any solution to this, I am facing the same issue
Please sign in to rate this answer.
Rish Shah 126 Reputation points

2021-05-17T21:11:31.823+00:00

None... Please do let me know if you find any resolution... Thanks

Pranay Katta 21 Reputation points

2021-06-04T01:37:42.857+00:00

I am facing the same issue. Were you able to solve the problem ?

Thanks
Sign in to comment