Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,343 questions
0 answers
How to read nested xlsx files in Azure Data Factory
How we can read these types of nested xlsx files in ADF
asked 2024-04-27T03:51:29.27+00:00
Abhishek Tripathi
0
Reputation points
edited the question 2024-04-27T03:53:15.94+00:00
Abhishek Tripathi
0
Reputation points
0 answers
Databricks OutOfMemory error on code that previously worked without issue
I have a notebook in Azure Databricks that does some transformations on a bronze tier table and inserts the transformed data into a silver tier table. This notebook is used to do an initial load of the data from our existing system into our new datalake.…
asked 2024-04-26T22:23:15.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 98%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Shane McGarry
0
Reputation points
asked 2024-04-26T22:23:15.07+00:00
Shane McGarry
0
Reputation points
0 answers
Databricks OutOfMemory error on code that previously worked without issue
I have a notebook in Azure Databricks that converts a list of columns in a bronze tier table into individual child rows in a silver tier table. This notebook was previously running (for weeks) without issue. Suddenly, I am now consistently receiving an…
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
945 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,921 questions
asked 2024-04-26T22:03:44.67+00:00
Shane McGarry
0
Reputation points
edited the question 2024-04-26T22:05:06.3166667+00:00
Shane McGarry
0
Reputation points
2 answers
I am getting error while displaying the dataframe in azure databricks
df.show() is working but display(df) is not working df=spark.read.csv("abfss://demo@anshadls2001.dfs.core.windows.net/data/",header= "True") display(df) (1) Spark Jobs Failed to store the result. Try rerunning the command.Failed to…
asked 2024-01-20T04:50:28.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 87%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pawan Singh
0
Reputation points
answered 2024-04-26T21:25:36.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 76%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Mojtaba Taslimitehrani
0
Reputation points
0 answers
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
asked 2024-04-26T21:07:19.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 38%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBJ%3C/text%3E%3C/svg%3E)
Bradley Jamrozik
0
Reputation points
asked 2024-04-26T21:07:19.1333333+00:00
Bradley Jamrozik
0
Reputation points
1 answer
Sync the tables from one azure databricks workspace to other databricks workspace or to adls gen2
Hello Team, We have two UC enabled databricks workspace. And we have to sync tables created in one Azure databricks workspace to other databricks workspace using PAT/any other reliable way or to adls gen2 account. Request you to let me know is there a…
asked 2024-04-11T18:50:09.1566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
65
Reputation points
commented 2024-04-26T18:25:52.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,136
Reputation points Microsoft Employee
2 answers
Databricks jobs are failing with the insufficient privileges error
When running the jobs in databrick user is getting the insufficient privilege error, but user do have all the privileges like select, read_metadata, modify, create and owner as well on the respective database.
asked 2024-04-24T19:28:13.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 24%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chethana B S
0
Reputation points
answered 2024-04-26T14:47:14.9333333+00:00
Chethana B S
0
Reputation points
2 answers
execution plan in databricks
How to check execution plan in data bricks select * from cte
1 answer
One of the answers was accepted by the question author.
Raising the limit in Azure Databricks identities
Context: The Azure Databricks documentation states that "You can have a maximum of 10,000 combined users and service principals and 5,000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and…
asked 2024-04-25T10:41:16.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 28.999999999999996%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kim Stig Hansen
0
Reputation points
accepted 2024-04-26T07:31:55.23+00:00
Kim Stig Hansen
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to use AWS Databricks note book in ADF..?
Hi Team, I want to create Linked service in ADF with AWS Databricks notebook. Can you guys help me with list of steps that i need follow. Regards, Naveen.
asked 2023-10-30T03:44:41.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 77%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Kumar Rachapalle (EXT/DIGITAL)
25
Reputation points
commented 2024-04-25T10:09:37.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 66%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Syed Ahamed S
0
Reputation points
1 answer
Custom text single label classification - Model API Consumption within Databricks
Hello together, i trained a model within Azure Language Studio , Custom text single label classification - and i want to consume Model API within Databricks Notebook. I get always below given error, kindly asking for your help. Thanks. Error: HTTP…
asked 2024-04-18T11:00:16.9066667+00:00
Aziz Öztürk
20
Reputation points
commented 2024-04-25T05:47:24.94+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
1 answer
How can I delete my Azure Databricks workspace when the workspace resource has already been removed?
As the title says I have an Azure Databricks Workspace where I have deleted the Workspace resource in Azure. However, the workspace still appears in the Account Console. From here I can log into the workspace but not create clusters etc. When I click on…
asked 2024-04-18T11:20:48.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 10%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kim Reinhart Jensen (KJEN)
0
Reputation points
commented 2024-04-25T05:46:32.7666667+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
1 answer
How to fix timeout error when clreating compute cluster on azure databricks.
Azure error message: [id: InstanceId(e8b82690e17a4624a152afb20dce5339), status: INSTANCE_LAUNCHING, workerEnvId:WorkerEnvId(workerenv-3367469989580530), lastStatusChangeTime: 1713836219115, groupIdOpt Some(0),requestIdOpt…
asked 2024-04-23T02:31:21.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 21%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
GEOFREY GETUBA
0
Reputation points
commented 2024-04-25T05:39:28.3766667+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
1 answer
How create a unity catalogue for existing Azure Databricks workspaces with all permissions as identity governance administrator?
Trying to create a unity catalogue for easy flow of machine learning solution on Databricks which has 3 separate workspaces - development, staging and production.
asked 2024-04-24T03:22:24.2566667+00:00
Mohd Zubair Humza
0
Reputation points
commented 2024-04-25T05:37:09.69+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
1 answer
An error occurred when using odbc and managed identity to link azure databricks in the. net core
Environmental description: The app service has already been configured with managed identity, and the relevant permissions for this managed identity have been configured in Azure databricks. When the app service uses code to access Azure databricks, the…
asked 2024-04-23T05:55:31.6166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 79%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
ZhiHong Zhang (BEYONDSOFT CONSULTING INC)
0
Reputation points
commented 2024-04-25T05:18:33.11+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
2 answers
coalesce and broadcast join
HI, what exactly happen between coalesce and broadcast join in backend on databricks level
3 answers
partition in db
HI , what hppend in databricks backend when partion is applied to table level
asked 2024-04-19T09:05:58.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
145
Reputation points
commented 2024-04-24T13:17:39.69+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to ingest CDC data from SQL DATA SYNC to Event hub?
Hi we have a scenario to implement real-time CDC from SQL DATA SYNC to Azure SQL DB. Which tools are good for near real-time processes and prices as well. By using data bricks we can merge stream data into delta lake. But How to ingest real-time…
asked 2020-11-10T12:28:43.067+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
edited an answer 2024-04-24T06:51:50.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 91%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Shane Blake
0
Reputation points
1 answer
How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken
I converted the data from parquet form to data format. Now I want to write the data to blob storage - datalakegen2. But facing below error while writing. I useed the below command to write my data: output_path =…
asked 2024-04-09T23:29:12.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 55.00000000000001%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sai Sunny Kothuri
0
Reputation points
commented 2024-04-24T05:00:35.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 98%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KarishmaTiwari-MSFT
18,447
Reputation points Microsoft Employee
0 answers
DatabricksSQL Logs and correlate with query history
Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…
asked 2024-04-22T11:09:19.4+00:00
Julio Avellaneda
0
Reputation points
commented 2024-04-24T02:51:51.83+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee