Not able to mount on ADLS Gen2 when using a high concurrency cluster in Azure Databricks

Question

Hello all,

I'm experiencing this issue Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel is not whitelisted when I'm trying to mount a ADLS container in Databricks notebook.

It follows the script I'm running:

%python
if not any(mount.mountPoint == "containerA" for mount in dbutils.fs.mounts()):
dbutils.fs.mount(
source = "abfss://" + containerNameTemp + "@" + storageAccountName + ".dfs.core.windows.net/",
mount_point = "containerA",
extra_configs = configs)

The cluster is a high concurrency cluster. Credential Passthrough is not enabled and Table Access Control is enabled.

It works fine when using a standard cluster.

Can someone explain me what is going on?

Thanks in advance,

Regis

Answer 1

Hello @Felipe Regis E Silva ,

Thanks for the question and using MS Q&A platform.

This error shows up with some library methods when using High Concurrency cluster with credential pass through enabled. If that is your scenario a work around that may be an option is to use a different cluster mode.

For more details, refer to Access Azure Data Lake Storage using Azure Active Directory credential passthrough.

Below are the three workarounds to resolve this issue:

• Update spark.databricks.pyspark.enableProcessIsolation to false
• Update spark.databricks.pyspark.enablePy4JSecurity to false
• Use standard clusters.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators