PowerBI / Databrick can we edit data in report
When we create reports in PowerBi or in Databricks. can we edit the data in report and if it can updated in backend datasource. Please let me know if this possible
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Error while provisioning Databricks
Hi All I am receiving the below error while provisioning Databricks The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, The resource write operation…
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
No Previews option in Azure Databricks user menu
I want to enable serverless compute in Azure Databricks which is in public preview, my workspace is eligible based on the details in the docs here and I am the workspace admin but I don't see a Previews option in my user menu. Is there another way to…
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse
I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…
DatabricksSQL Logs and correlate with query history
Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…
py4j.security.Py4JSecurityException
Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…
Spark_Ambiguous_Executor_MaxExecutorFailures
Hi I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error DAG = { "activities": [ { "name": "Notebook", "path":…
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
Databricks jobs are failing with the insufficient privileges error
When running the jobs in databrick user is getting the insufficient privilege error, but user do have all the privileges like select, read_metadata, modify, create and owner as well on the respective database.
Azure Databricks workflow job failure
We have a stream workflow job that run 24*7 and loads the data in delta table for say: raw.deltaTableA Now, the problem is in case we are trying to optimize this delta (optimize raw.deltaTableA) table while the table is getting loaded we get frequent…
Data size of databricks delta tables
It has been observed that the size of delta tables are much less as compared to when checked the underlying delta files in the storage account. Suppose a databricks delta table raw.deltaTableA has size of 2MB if we check the size of underlying delta…
Why create compute is taking long time?
I am trying create a compute for my workspaces i tried every combination still it is not working
Rest API call towards Azure Storage account (ADLSgen2) from Databricks (write to Delta) fails when Authentication is via ACL and SP
In the nutshell. I work in very sensitive and security intense environment. We have decided to use ACL (instead of RBAC/ABAC) for authorization to achieve finer control over Storage account. For our Databricks service, we use only job cluster and job is…
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…