PowerBI / Databrick can we edit data in report
When we create reports in PowerBi or in Databricks. can we edit the data in report and if it can updated in backend datasource. Please let me know if this possible
How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse
I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…
DatabricksSQL Logs and correlate with query history
Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…
py4j.security.Py4JSecurityException
Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Spark_Ambiguous_Executor_MaxExecutorFailures
Hi I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error DAG = { "activities": [ { "name": "Notebook", "path":…
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
Databricks jobs are failing with the insufficient privileges error
When running the jobs in databrick user is getting the insufficient privilege error, but user do have all the privileges like select, read_metadata, modify, create and owner as well on the respective database.
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
Azure Databricks workflow job failure
We have a stream workflow job that run 24*7 and loads the data in delta table for say: raw.deltaTableA Now, the problem is in case we are trying to optimize this delta (optimize raw.deltaTableA) table while the table is getting loaded we get frequent…
Data size of databricks delta tables
It has been observed that the size of delta tables are much less as compared to when checked the underlying delta files in the storage account. Suppose a databricks delta table raw.deltaTableA has size of 2MB if we check the size of underlying delta…
Why create compute is taking long time?
I am trying create a compute for my workspaces i tried every combination still it is not working
Rest API call towards Azure Storage account (ADLSgen2) from Databricks (write to Delta) fails when Authentication is via ACL and SP
In the nutshell. I work in very sensitive and security intense environment. We have decided to use ACL (instead of RBAC/ABAC) for authorization to achieve finer control over Storage account. For our Databricks service, we use only job cluster and job is…
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
Databricks to Table storage Data load
Hi Team, Currently, I have data bricks spark jobs running which load data from Blob Storage and then process it using Databricks and then dump the clean data into another blob storage. Now, I would like to dump the cleaned file to Table storage…
Databricks support redirects to azure support: unexpected internal error when spinning up a Databricks all-purpose cluster
Hello, What do we do when we get this error, when spinning up a Databricks all-purpose cluster? { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type": "SERVICE_FAULT", …
Do you know how to install the 'ODBC Driver 17 for PostgreSQL' on a Azure Databricks cluster?
I am attempting to run postgreSQL stored procedures , through Azure Databricks notebook. We have stored procedure written in Azure Database for PostgreSQL and we want to run postgreSQL stored procedures through Azure Databricks Notebook (using…