question

MarkoOja-9481 avatar image
1 Vote"
MarkoOja-9481 asked LucasZylbersztejnPolak-0885 commented

ADF Delta table in copy activity fails

Hi.

I need help debugging my error.
I have ADF copy activity from Delta lake dataset to CSV (and other pipeline other way around).
Delta lake dataset works fine in design mode. Meaning that schem import and data preview both works. However when I run either of my pipelines I get this error:

ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key Caused by: Invalid configuration value detected for fs.azure.account.key.

I have located both my delta tables and CSV's / folders in same storage account (gen2 with hierarchical namespace enabled ) in same container.
From databricks cluster I can access all locations and files used. My cluster does have my storage accounts primary key in Spark-configs. All out of ideas why I am getting this error.

Please help :)

azure-data-factoryazure-databricksazure-data-lake-storage
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @MarkoOja-9481,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @MarkoOja-9481,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

@HimanshuSinha-MSFT .. I am also getting the same error even after adding the appid in the databricks

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered LucasZylbersztejnPolak-0885 commented

Hello @,
Thanks for the ask and using Microsoft Q&A platform .
You will have to create a Application id and then pass the details in the config as shown below

144295-image.png



fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account-name>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account-name>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account-name>.dfs.core.windows.net <service-credential>
fs.azure.account.oauth2.client.endpoint.<storage-account-name>.dfs.core.windows.net https://login.microsoftonline.com/<tenant-id>/oauth2/token

Please do let me know how it goes .
Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (69.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks @HimanshuSinha-MSFT

There are two more things that are not clear for me.

  1. Which Storage Account should we use? Because in my case, the table that I'm trying to read from the Delta Lake was directly uploaded in Databricks, and is not pointing to any Storage Account.

  2. The details should be sent like the image attached?

176561-screenshot-2022-02-21-192128.png


0 Votes 0 ·