Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,970 questions
py4j.security.Py4JSecurityException
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 16%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Ahuja, Rachit
0
Reputation points
Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS.
I am getting the following error:
Py4JError: An error occurred while calling o547.resourceProfileManager. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.resource.ResourceProfileManager org.apache.spark.SparkContext.resourceProfileManager() is not whitelisted on class class org.apache.spark.SparkContext at py4j.security.WhitelistingPy4JSecurityManager.checkCall(WhitelistingPy4JSecurityManager.java:473) at py4j.Gateway.invoke(Gateway.java:305) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)
Here is my cluster configuration JSON: { "cluster_id": "0412-193940-roa64zgt", "creator_user_name": "b90f2c51-c49c-4f97-95a9-d58c6c1e8fc8", "spark_context_id": 5504105953918341000, "driver_healthy": true, "cluster_name": "DsShared", "spark_version": "14.3.x-cpu-ml-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true" }, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1 }, "node_type_id": "Standard_D4s_v5", "driver_node_type_id": "Standard_D4s_v5", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 20, "enable_elastic_disk": true, "disk_spec": {}, "cluster_source": "UI", "enable_local_disk_encryption": false, "instance_source": { "node_type_id": "Standard_D4s_v5" }, "driver_instance_source": { "node_type_id": "Standard_D4s_v5" }, "data_security_mode": "LEGACY_PASSTHROUGH", "runtime_engine": "STANDARD", "effective_spark_version": "14.3.x-cpu-ml-scala2.12", "autoscale": { "min_workers": 1, "max_workers": 3, "target_workers": 1 }, "default_tags": { "Vendor": "Databricks", "Creator": "b90f2c51-c49c-4f97-95a9-d58c6c1e8fc8", "ClusterName": "DsShared", "ClusterId": "0412-193940-roa64zgt", }, "termination_reason": { "code": "INACTIVITY", "type": "SUCCESS", "parameters": { "inactivity_duration_min": "20" } }, "pinned_by_user_name": "4460559388477034", "init_scripts_safe_mode": false, "spec": { "cluster_name": "DsShared", "spark_version": "14.3.x-cpu-ml-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true" }, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1 }, "node_type_id": "Standard_D4s_v5", "driver_node_type_id": "Standard_D4s_v5", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 20, "enable_elastic_disk": true, "cluster_source": "UI", "data_security_mode": "LEGACY_PASSTHROUGH", "runtime_engine": "STANDARD", "autoscale": { "min_workers": 1, "max_workers": 3 } } }
Any help would be highly appreciated.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT 27,566 Reputation points Microsoft Employee2024-05-06T17:59:40.56+00:00 Hello Ahuja, Rachit,
Welcome to the Microsoft Q&A forum.
Can you please let us know the notebook commands that you are using here?
-
BhargavaGunnam-MSFT 27,566 Reputation points Microsoft Employee
2024-05-10T23:05:39.4066667+00:00 Hello Ahuja, Rachit,
I am checking to see if you had a chance to look into my above response.
-
PRADEEPCHEEKATLA-MSFT 79,536 Reputation points Microsoft Employee2024-05-20T08:05:29.2833333+00:00 @Ahuja, Rachit - It seems like the error is related to Py4JSecurityException. This error occurs when a method is not whitelisted on the class. In your case, the method "public org.apache.spark.resource.ResourceProfileManager org.apache.spark.SparkContext.resourceProfileManager()" is not whitelisted on class org.apache.spark.SparkContext.
To resolve this issue, you can try adding the following configuration to your cluster:
JavaOptions": "-Dspark.driver.extraJavaOptions=-Dspark.security.credentials.whitelist=org.apache.spark.resource.ResourceProfileManager", "spark.executor.extraJavaOptions": "-Dspark.executor.extraJavaOptions=-Dspark.security.credentials.whitelist=org.apache.spark.resource.ResourceProfileManager"This configuration will whitelist the ResourceProfileManager class and should resolve the Py4JSecurityException error.
Let me know if this helps.
-
BhargavaGunnam-MSFT 27,566 Reputation points Microsoft Employee
2024-05-21T21:04:06.7066667+00:00 Thanks PRADEEPCHEEKATLA-MSFT
Hello Ahuja, Rachit,
Please check the above response and let us know if you have any further questions.
Sign in to comment