How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?
|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Indexing a Pyspark dataframe
Hey guys, I am having a very large dataset as multiple parquets (like around 20,000 small files) which I am reading into a pyspark dataframe. I want to add an index column in this dataframe and then do some data profiling and data quality check…
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to…
Clusters are failing to launch. Cluster launch will be retried.
Hi, I am a newbie. Can someone show me how I can fix the below please? Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores…
[Databricks] Clusters are failing to launch. Cluster launch will be retried.
Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks
Cannot read excel file which is in using load_workbook of openpyxl but can read if copied to dbfs
Rest API call towards Azure Storage account (ADLSgen2) from Databricks (write to Delta) fails when Authentication is via ACL and SP
In the nutshell. I work in very sensitive and security intense environment. We have decided to use ACL (instead of RBAC/ABAC) for authorization to achieve finer control over Storage account. For our Databricks service, we use only job cluster and job is…
py4j.security.Py4JSecurityException
Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…
Spark_Ambiguous_Executor_MaxExecutorFailures
Hi I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error DAG = { "activities": [ { "name": "Notebook", "path":…
Why create compute is taking long time?
I am trying create a compute for my workspaces i tried every combination still it is not working
Databricks support redirects to azure support: unexpected internal error when spinning up a Databricks all-purpose cluster
Hello, What do we do when we get this error, when spinning up a Databricks all-purpose cluster? { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type": "SERVICE_FAULT", …
No Previews option in Azure Databricks user menu
I want to enable serverless compute in Azure Databricks which is in public preview, my workspace is eligible based on the details in the docs here and I am the workspace admin but I don't see a Previews option in my user menu. Is there another way to…
How to ship Azure Databricks artifacts from Dev->QA->Prod through Azure Devops Pipelines?
We have a Azure Databricks workspace and Dev/QA/Prod environments. Everytime the Data engineers have to ship the artifacts from nonprod -> prod (e.g. python notebooks, config modules, etc) they have to copy the artifacts manually over to the next…
Error while provisioning Databricks
Hi All I am receiving the below error while provisioning Databricks The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, The resource write operation…
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
PowerBI / Databrick can we edit data in report
When we create reports in PowerBi or in Databricks. can we edit the data in report and if it can updated in backend datasource. Please let me know if this possible
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…