ADF Data Flows Flatten nested json array values are being populated as null
Hi All I am building a data transamination with ADF data flow using a nested json array of objects , but after parse and flatten the json node itOffer.item.LeadOfer.zdeal.item[].dealNumber I am seeing that the column values are populated as null . I…
Issue with Copy Dataverse data into Azure SQL using Synapse Link Template
I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…
how to peer vnet with synapse managed vent
We have a VNet on which AKS is running, we want to peer this VNet with Synapse Managed VNet so that we can connect to the api hosted in the aks vnet from spark cluster
source files problem from internal server to Gen2 container
Hello, I am trying to connect my interal files system to the gen2 container sink, following this instructions https://arulmouzhi.wordpress.com/2021/04/12/bringing-folder-structure-via-azure-data-factory/ The first one in Allfiles set, I can load…
Running specific pipelines from an Orchestration pipeline
How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.
Should Serverless Spark pool be able to process less than 5 million rows quickly?
We are downloading about 10K records daily and the trying to merge them into a Deltalake file and noticing that it takes about 35-45 minutes to merge. Is that expected?
Azure Synapse Analytics
Hi, I have a question. Can I access the Database on SQL Database pane from apache sp
Updating Oracle on-premise table from a Synapse table
Hi, I need to update an Oracle on-premise table from a Synapse table by joining the Oracle table with the Synapse table avoiding to build a staging table on Oracle platform to copy the Synapse data. It is a problem to obtain the authorization from my…
We are not getting data in azure data lake Service v2 storage's container.
Hello, We are getting data in the Azure event hub sink through the POSTGRES SQL database but we are not getting files in the container which we have configured in an ADLS GEN 2 and while configuring we have an encounter that we are using a free…
How can i integrate hubspot rest api linked service in azure synapse.
When i directly integrated husbpot it's not giving me all the data. But when i try to link that with rest api my test connection is failied. I checked the hubspot api endpoint and from postman i can see all the data tables i want but it is not working in…
How to reset password for Azure synapse dedicated SQL Pool "when Reset SQL Admin password" is disabled?
Hello, When I'm creating a linked service to fetch the data from Dedicated SQL Pool within the Azure Synapse Analytics workspace, it ask for the SQL authentication. Since I hadn't created the username and password at that time, I want to create or reset…
Seeking Expertise in Spark SQL CTE Recursive Queries in Azure Databricks
I'm currently diving deep into Spark SQL and its capabilities, and I'm facing an interesting challenge. I'm eager to learn how to write CTE recursive queries in Spark SQL, but after thorough research, it seems that Spark doesn't natively support…
How to ignore the records by applying an auditing filed column condition using ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
azure synapse RBAC not working
I have just created a synapse analytics workspace, on opening it I get an error message similar to the below. From my research, I have gr anted storage blob contributor role to the synapse application as well as the user assigned managed identity. Going…
Using SMI in Synapse Spark Job
Hello We have Synapse Pipeline with Spark Job Definition We currently use SPN to read data from ADLS2 and also to write to kusto with Spark Kusto Connector (Getting a token from SPN) We have saved SPN credentials into AKV. We have urgent requirement to…
how can i solve for this error HTTP header that's mandatory for this request is not specified.", 400 in azure synapse pyspark
hey guys how can solve for this error i am using synapse to read data from a blob with Hierarchical namespace Enabled but when i try to write back i get : Operation failed: "An HTTP header that's mandatory for this request is not specified.",…
Failed to convert the value in 'xxxx' property to 'System.String' type in Data Lake Connector
Environment: Azure Synapse I am developing a pipeline that has a lookup to SQL that is passing the values to GetMetaData. When using @dataset.name of parameter in the dataset, it fails with "Failed to convert the value in name of parameter…
How to flatten dynamic Json in ADF/Synapse
I have a issue where I have dynamic json and enabled schema drift in my source, To flatten the json I am coding it as array(byName('Columnname')) in unroll by dynamic content section. Here columnname is the array which need to be unrolled. I am getting…
From Synapse notebook, trouble connecting to Azure SQL using system managed identity
I can create a Linked Service to my Database using my Synapse system-managed identity. This works fine, and the "test connection" shows success. But for the life of me, I can't figure out how to do it in a Python notebook. Here is the…
Assert error output not writing to blob
Hello. I have built a pipeline which gathers data from a .csv file and then goes through a couple of assert activities to check the validity of the data. Valid rows are supposed to be input into the table, while assert-failure rows are setup to be…