Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,955 questions
1 answer
One of the answers was accepted by the question author.
Azure Databricks Lakehouse Monitoring queries
Hi Team, I was exploring on Azure Databricks Lakehouse monitoring. I have few queries on this: When I am running a "refresh metrics" irrespective of an automated schedule or manual refresh, which compute does it run? There is no mechanism to…
asked 2024-04-28T11:50:10.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 35%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sudipta Goswami
20
Reputation points
commented 2024-05-18T11:31:49.4066667+00:00
Sudipta Goswami
20
Reputation points
0 answers
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
asked 2024-05-02T15:44:57.8633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
edited a comment 2024-05-17T11:54:17.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,475
Reputation points Microsoft Vendor
1 answer
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
asked 2024-05-16T08:34:02.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 75%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Manoj
0
Reputation points
answered 2024-05-17T06:20:35.8533333+00:00
Wilko van de Velde
2,226
Reputation points
2 answers
Reading data from Sharepoint Using Databricks
I am trying to read the SharePoint data from Databricks. But, not able to access it. Could you please let me know if it is possible to read the SharePoint data using Databricks. If possible then could you please share some code or Link where i can find…
asked 2021-11-28T03:10:18.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 38%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
Jitendra . Kumar
1
Reputation point
commented 2024-05-16T16:41:37.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 21%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Cuirot Maxime
0
Reputation points
0 answers
Unable to downgrade Databricks workspace
When downgrading the Databricks Workspace, I receive the following message; However, none of the Enhanced Security options are currently enabled; Could you please help me identify the cause of the error?
asked 2024-05-16T09:02:23.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 73%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Naomi Mostert | Codex
0
Reputation points
edited the question 2024-05-16T11:16:14.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 64%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EON%3C/text%3E%3C/svg%3E)
OMMI NAVEEN KUMAR
195
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Special character handling for file processing
Hello, I have some CSV files as feeds into datalake storage, this file will contain data /records with some special characters eg '/' We need to process the file from one container to another and we will need to remove some of the these special…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
0 answers
Is dynamic SQL Queries supported on Azur Databricks SQL Cluster?
Hello, I'm planning to implement Dynamic SQL function to query data on Databricks table. Tables and access for the users are governed by a custom access matrix using the Unity catalog. The problem is that in a custom matrix, there are two types of users:…
asked 2024-05-15T19:01:11.4766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 0%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJP%3C/text%3E%3C/svg%3E)
Jayesh Potwade
0
Reputation points
edited a comment 2024-05-16T06:02:52.4+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
Error accessing Azure sql from Azure databricks using jdbc authentication=ActiveDirectoryInteractive
Getting below error while accessing Azure sql using jdbc from Azure databricks notebook, com.microsoft.sqlserver.jdbc.SQLServerException: Failed to authenticate the user p***** in Active Directory (Authentication=ActiveDirectoryInteractive). Unable to…
asked 2024-05-14T11:35:08.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Pankaj Mendi
0
Reputation points
commented 2024-05-15T13:53:58.31+00:00
Alberto Morillo
32,981
Reputation points MVP
2 answers
One of the answers was accepted by the question author.
Azure Databricks test cases and Git commit
Hi friends, we have testing our test cases in a testing environment, and these are many tests, and want to test them as per test uses- cases and before committing into Git. Since there are two many of them I do not want to do it manually for each…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 15%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
asked 2024-04-26T21:07:19.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 38%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBJ%3C/text%3E%3C/svg%3E)
Bradley Jamrozik
0
Reputation points
commented 2024-05-15T10:05:36.5+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
2 answers
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
asked 2024-05-01T12:12:06.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-15T07:56:16.4166667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
0
Reputation points
edited an answer 2024-05-15T04:39:12.8666667+00:00
Sumarigo-MSFT
43,911
Reputation points Microsoft Employee
2 answers
Clusters are failing to launch. Cluster launch will be retried.
Hi, I am a newbie. Can someone show me how I can fix the below please? Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores…
asked 2024-05-13T21:53:30.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 18%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBC%3C/text%3E%3C/svg%3E)
Billy Cheng
0
Reputation points
commented 2024-05-14T22:50:40.3833333+00:00
TP
78,426
Reputation points
1 answer
One of the answers was accepted by the question author.
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
1 answer
How to change databricks location
i want to change the existing location(region) for databricks and along with other apps as well
asked 2022-01-11T09:09:10.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 13%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Venkatesulu Kalla
21
Reputation points
commented 2024-05-14T09:55:09.6133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 59%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sadashiv Gupta
0
Reputation points Microsoft Employee
1 answer
How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?
|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …
asked 2024-04-30T17:39:32.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 18%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Parris Sikorski (ALLEGIS GROUP HOLDINGS INC)
0
Reputation points Microsoft Vendor
commented 2024-05-14T04:14:02.63+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
asked 2024-05-08T08:58:46.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Senad Hadzikic
20
Reputation points
commented 2024-05-14T04:09:06.0466667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
Indexing a Pyspark dataframe
Hey guys, I am having a very large dataset as multiple parquets (like around 20,000 small files) which I am reading into a pyspark dataframe. I want to add an index column in this dataframe and then do some data profiling and data quality check…
asked 2024-05-09T07:29:38.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 81%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Varun S Kumar
50
Reputation points
edited a comment 2024-05-14T04:06:06.94+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to…
asked 2024-05-08T22:13:53.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 45%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
McDonald, Matthew
101
Reputation points
commented 2024-05-14T03:54:20.4066667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
[Databricks] Clusters are failing to launch. Cluster launch will be retried.
Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…
asked 2024-05-08T22:05:05.41+00:00
Billy Cheng
0
Reputation points
commented 2024-05-14T03:45:51.6633333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee