Azure Databricks

py4j.security.Py4JSecurityException

Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…

Spark_Ambiguous_Executor_MaxExecutorFailures

Hi I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error DAG = { "activities": [ { "name": "Notebook", "path":…

Why create compute is taking long time?

I am trying create a compute for my workspaces i tried every combination still it is not working

[Databricks] Clusters are failing to launch. Cluster launch will be retried.

Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…

Databricks support redirects to azure support: unexpected internal error when spinning up a Databricks all-purpose cluster

Hello, What do we do when we get this error, when spinning up a Databricks all-purpose cluster? {   "reason": {     "code": "CONTAINER_LAUNCH_FAILURE",     "type": "SERVICE_FAULT",    …

How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?

|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …

No Previews option in Azure Databricks user menu

I want to enable serverless compute in Azure Databricks which is in public preview, my workspace is eligible based on the details in the docs here and I am the workspace admin but I don't see a Previews option in my user menu. Is there another way to…

Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks

Cannot read excel file which is in using load_workbook of openpyxl but can read if copied to dbfs

Cannot See Index tagging in while uploading Blob in ADLS gen2

Indexing a Pyspark dataframe

Hey guys, I am having a very large dataset as multiple parquets (like around 20,000 small files) which I am reading into a pyspark dataframe. I want to add an index column in this dataframe and then do some data profiling and data quality check…

How to ship Azure Databricks artifacts from Dev->QA->Prod through Azure Devops Pipelines?

We have a Azure Databricks workspace and Dev/QA/Prod environments. Everytime the Data engineers have to ship the artifacts from nonprod -> prod (e.g. python notebooks, config modules, etc) they have to copy the artifacts manually over to the next…

How to reduce unnecessary high memory usage in a Databricks cluster?

We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…

Error while provisioning Databricks

Hi All I am receiving the below error while provisioning Databricks The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, The resource write operation…

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

PowerBI / Databrick can we edit data in report

When we create reports in PowerBi or in Databricks. can we edit the data in report and if it can updated in backend datasource. Please let me know if this possible

How do I figure out what public IP ranges my Databricks workspace clusters are coming from?

Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to…

Error with Create Table USING DELTA LOCATION in training exercise

In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…

Azure Databricks

Azure Databricks

An Apache Spark-based analytics platform optimized for Azure.

1,947 questions

Sign in to follow

Azure Training

Azure Training

Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.

977 questions

Sign in to follow

Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute

I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…

DatabricksSQL Logs and correlate with query history

Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…

SAP latency data

Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…