Azure Data Factory

fisikxnyba

https://github.com/orgs/dbeaver/discussions/32289

ADF Copy Activity failed with ORA-00928: missing SELECT keyword

ADF Copy activity is failing with following error. Its happening after copying certain amount of data. So don't think its issue with pipeline. Also same pipeline works in another environment(PROD/DEV) so mostly its data issue. How to identify what is…

Connection issue with Oracle Autonomous Cloud (ADW) and Azure Data Factory

Hi, I am trying to connect Oracle Autonomous Database (ADW) with Azure Data Factory using self hosted integration runtime. After setting up Oracle DataDirect wire protocol 8.0 driver still i am unable to connect. i am getting this error below ERROR…

Error from ADF pipeline loading data into Dynamics 365 table

I have an Azure pipeline that is copying data from a table in an azure sql db to a table in Dynamics 365 CE. The D365 has MFA attached so I am using an azure service principal in the linked service connection for the pipeline. The Service principal is…

Error while passing parameters in Rest Api Azure Data Factory

Hi Folks. Hope you Are doing well. I am passing parameters from one data factory to another to trigger a copy activity that would copy data from an intermediate Location to a final Target which can be an SFTP, Mshare or ADLS location. I am…

ADF dataflow activity fails with large record sets

I have dataflow activity that retrieves data from an Azure MySQL database and invokes an address API using external call transformation to update the database with standardized addresses. However, when dealing with a large number of records…

Hi folks , Getting this error while running copy data activity in azure data factory " Http Url Address 'https://github.com/https://github.com/cloudboxacademy/covid19/blob/main/ecdc_data/cases_deaths.csv' is invalid." Even the url is accessible.

Hi folks , Getting this error while running copy data activity in azure data factory. Consider this is source url " Http Url Address 'https://github.com/https://github.com/cloudboxacademy/covid19/blob/main/ecdc_data/cases_deaths.csv' is…

In ADF, All Linked Services not showing up in Git Mode from Live Mode

There are total 4 linked services created in my ADF and its on live mode. I make a git configuration and set the ADF to new branch along with providing publish branch. But only 2 linked services showed up leaving behind a LS with setup on azure sql db…

How to create service principal, link to the azure sql database in the azure portal.

I want to know how to create service principal to connect to azure SQL server database. (I tried by creating user assigned identity. But this also not succeeded.) The user assigned manage identity is configured by, *create user assigned manage identity…

Azure Data Factory - Pipeline always fails at the second connection to On-Premise data source

I have SSIS Package that pulls data from On-premise data Source and push data to Azure Database. In that package, I have 3 data flow tasks, each of them connect to one db on On-premise data Source. SSIS Package has been deployed to Azure SQL DB (SSISDB)…

Azure Data Factory - Copy data activity failed: Failure happened on 'Source' side

Operation on target Copy data1 failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path:…

How to update and delete a row using Azure Data Factory Change Data Capture

I am exploring Azure Data Factory CDC feature, and I am trying to perform the CDC from one SQL table to another SQL table, also the SQL table in the source has a primary key. Whenever a new row is added to the source table, it gets added to the…

Unable to copy share point files through ADF

I am facing issue now! I followed all the steps https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory and tried to read pdf files from sharepoint through ADF. source: Sharepoint(PDF files)(I selected…

How to copy files from sharepoint drive that has huge heirarcy of folders and subfolders

We want to get all files from the sharepoint into our ADLS container. In sharepoint drive there are arround 60k files but through our existing pipeline we are getting only 600 files. We have a web activity with below…

Can you assist with a Synapse issue involving memory allocation in SQL DW Copy Command in Data Factory, without solely resorting to increasing DWU?

I'm encountering an issue while using the Copy Activity in Data Factory with the Polybase copy option. The error message I receive is as follows: “…..SQL DW Copy Command operation failed with error 'Unable to allocate 257707 KB for columnstore…

How to keep ARM Template in sync when modifying Azure Data Factory?

I'm unclear on the process for maintaining the ARM Template for deployments when modifying an Azure Data Factory (ADF). I see two options: Modify the ADF in the browser-based drag-n-drop editor and use the generated Json ARM Template as is. Manually…

Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks

Cannot read excel file which is in using load_workbook of openpyxl but can read if copied to dbfs

How to Get REST API Data when return rows are more than 30,000 and store in one single csv

How to Get REST API Data when return rows are more than 30,000 and store in one single csv. I tried pagination rule with AbsoluteUrl to $.Response. How to get copy activity go through each record with offset and limit automatically? How to write all…

Hi all. When i am trying to free trail account for azure its showing like card holder name info is incorrect and at last it is showing like "went with something wrong , we are investigating on it ".Could anyone please help at this situation as i have work

How to fix self -hosted integration runtime is unavailable staus,How to bring running staus

How to fix existing self -hosted integration runtime is unavailable status,How to bring running status