question

FelipeRegisESilva-0968 avatar image
0 Votes"
FelipeRegisESilva-0968 asked PRADEEPCHEEKATLA-MSFT commented

Not able to mount on ADLS Gen2 when using a high concurrency cluster in Azure Databricks

Hello all,

I'm experiencing this issue Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel is not whitelisted when I'm trying to mount a ADLS container in Databricks notebook.

It follows the script I'm running:

%python
if not any(mount.mountPoint == "containerA" for mount in dbutils.fs.mounts()):
dbutils.fs.mount(
source = "abfss://" + containerNameTemp + "@" + storageAccountName + ".dfs.core.windows.net/",
mount_point = "containerA",
extra_configs = configs)

The cluster is a high concurrency cluster. Credential Passthrough is not enabled and Table Access Control is enabled.

It works fine when using a standard cluster.

Can someone explain me what is going on?

Thanks in advance,

Regis











azure-databricksazure-data-lake-storage
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @FelipeRegisESilva-0968,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @FelipeRegisESilva-0968,

Thanks for the question and using MS Q&A platform.

This error shows up with some library methods when using High Concurrency cluster with credential pass through enabled. If that is your scenario a work around that may be an option is to use a different cluster mode.

196887-image.png

For more details, refer to Access Azure Data Lake Storage using Azure Active Directory credential passthrough.

Below are the three workarounds to resolve this issue:

  • Update spark.databricks.pyspark.enableProcessIsolation to false

  • Update spark.databricks.pyspark.enablePy4JSecurity to false

  • Use standard clusters.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


image.png (16.5 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello, @PRADEEPCHEEKATLA-MSFT. Thanks for your help. I guess it worked partially for the first 02 commands you provided. Now, we are able to mount but dataframe load from ADLS Gen2 is still failing.

We need to have table access control feature enabled in this high concurrency cluster. For this reason we can't use a standard cluster.

Team is trying to load dataframe from .avro file within a container folder in ADLS Gen2 and they are getting the following error:

"java.lang.SecurityException: User does not have permission SELECT on any file."

198297-image.png

I've already checked ACL and dbricks principal is already granted permission to access the folder and file as specified in the screenshot.

I didn't know that there were a SELECT permission on file to be configured besides on to workspace objects in databricks.

Thanks in advance for your help and attention.


0 Votes 0 ·
image.png (25.5 KiB)

Hello @FelipeRegisESilva-0968,

The scope of your question is different compared to the original scope of the question asked.

I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.

0 Votes 0 ·