Raising the limit in Azure Databricks identities
Context: The Azure Databricks documentation states that "You can have a maximum of 10,000 combined users and service principals and 5,000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and…
How to use AWS Databricks note book in ADF..?
Hi Team, I want to create Linked service in ADF with AWS Databricks notebook. Can you guys help me with list of steps that i need follow. Regards, Naveen.
Custom text single label classification - Model API Consumption within Databricks
Hello together, i trained a model within Azure Language Studio , Custom text single label classification - and i want to consume Model API within Databricks Notebook. I get always below given error, kindly asking for your help. Thanks. Error: HTTP…
How to fix timeout error when clreating compute cluster on azure databricks.
Azure error message: [id: InstanceId(e8b82690e17a4624a152afb20dce5339), status: INSTANCE_LAUNCHING, workerEnvId:WorkerEnvId(workerenv-3367469989580530), lastStatusChangeTime: 1713836219115, groupIdOpt Some(0),requestIdOpt…
How to ingest CDC data from SQL DATA SYNC to Event hub?
Hi we have a scenario to implement real-time CDC from SQL DATA SYNC to Azure SQL DB. Which tools are good for near real-time processes and prices as well. By using data bricks we can merge stream data into delta lake. But How to ingest real-time…
How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken
I converted the data from parquet form to data format. Now I want to write the data to blob storage - datalakegen2. But facing below error while writing. I useed the below command to write my data: output_path =…
LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: d2425fce-9179-49eb-829c-a7b2a0d963ed.
Hi Everyone, I am getting the Below Error Code while running my cells in Synapse Notebook: > LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID:…
Azure Databricks fail to install Geospark libraries from Maven
Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . However , I am getting below error Library installation attempted on the driver node of cluster…
How to recover a accidently deleted databricks instance in a free trial account
How to recover a accidently deleted databricks instance in a free trial account.
Azure Delta Lake to Snowflake
Hi Team, I am creating Delta lake in Azure data lake from ADF using Dataflow Sink - inline dataset as Delta and also through Databricks. Have created External Table in Databricks which is pointing to Mounted Azure datalake location. Now, I want to load…
FileAlreadyExistsException: Failed to rename temp file dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/__tmp_path_dir/.2.zip.52d0723f-b803-4a8a-9533-9d6e67813641.tmp to dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/2.zip because file exists
I have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We encountered a rare issue (could not replicate it). Below is the exception Message: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to…
Failing to connect to metastore when using dbx to launch ephimeral cluster in databricks
Dear all, I am using dbx to deploy and launch jobs on ephemeral clusters on databricks. I have initialized the the cicd-sample-project and connected to a fresh empty Databricks Free trial environment and everything works. But when I try to do…
Databricks Cluster Size
Hi, We have a below use case. We are developing ML Development using azure data bricks cluster service's. Our data bricks will receive 20 GB size of dataset records and we are running our logic/algorithms against this dataset's. We have to run every…
How to calculate the price of a databricks job with Photon engine?
It's trivial to calculate a databricks job cost using Azure pricing calculator. You add up Azure VM price and Azure Databricks DBU consumption price and you would get your answer. However when it comes to Databricks with Photon runtime, the pricing is…
ADF pipeline to read the data from UC table to adls gen2 account
Hello Team, We have a requirement to create Azure Datafactory pipeline to read the data from UC table, access on the table is provided ( to Azure Datafactory Managed Identity) and copy the data into adls gen2. Is there a way or article to implement this?…
Enabling Azure PIM Disables user within DataBricks
We have successfully stood up Azure Databricks in our tenant. We are leveraging SCIM and User Provisioning to grant our users SSO access into DataBricks. We are trying to layer in an additional layer of security to meet our current security standards by…
How to Fix Error Configuring VPC Peering in Azure Databricks? Failed to add virtual network peering 'Peering' to 'workers-vnet'. Error: The client "" with object ID "" has permission to perform action
I'm facing an issue while attempting to configure VPC peering in Azure Databricks. When trying to establish VPC peering between the Azure Databricks workspace's VNET ("workers-vnet") and an external network, I encountered the following error…
Can we customize the index url to fetch Python package from a protected PyPi source
Recently Databricks provided in public preview the possibility to add library in the compute policy. We will need to install a library located in a protected PyPi repo from Azure DevOps: pkgs.dev.azure.com/{further_path}/pypi/packages. We used to have…
How to add SSL certificate for using it on Databricks cluster with PySpark
Hi, We would like to use functions written in PySpark for calling an external service that requires SSL certificate on the cluster. Currently we are using an init script similar to explained in the documentation -…
How can you use Databricks Vector Search with images?
I have been working with Databricks Vector Search and feel comfortable using it on any kind of textual data. However I am running into problems with using it for image searching, and there are no examples available in the documentation to provide more…