9,805 questions with Azure Data Factory tags

Sort by: Updated
1 answer One of the answers was accepted by the question author.

How can I find help validating a certificate?

Indeed, I am a student in cloud computing in Morocco and I do not have enough means to obtain the AZ 900 certification and my family also does not have enough financial means. Could someone help me validate this certificate please?

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,380 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,503 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,476 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T15:54:26.2366667+00:00
HAMIDOU YOUSSOUF 20 Reputation points
accepted 2024-05-31T14:55:42.77+00:00
HAMIDOU YOUSSOUF 20 Reputation points
1 answer

how to call Oracle procedure in ADF lookup activity

We need call Oracle Procedure from the Azure Data Factory. It seems that in lookup activity only Select is supported. How can we call the Oracle Procedure without the modification on Oracle side?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-01-04T13:38:07.0533333+00:00
Shi, Kangwen 0 Reputation points
edited a comment 2024-05-31T14:55:32.31+00:00
Maltais, Nancy 0 Reputation points
0 answers

Azure Data Factory simple copy file and rename

Hello: I have a simple requirement. I have a couple of files in an Azure Blob folder (receiving folder) and I need to copy them, to another fold (Azure Blob folder) and rename them. This posting is really close to what I need to do: …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-31T14:28:26.9866667+00:00
KBD 0 Reputation points
commented 2024-05-31T14:40:39.7333333+00:00
KBD 0 Reputation points
2 answers

how to check who created the log analytics workspace in azure but it created more then 90n days ago we can't able to check in activity log how to we check that?

how to check who created the log analytics workspace in azure ? it created more then 90n days ago we can't able to check in activity log how to we check in another way? kindly please share me how can we check that

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,883 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,319 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,068 questions
asked 2024-05-31T11:00:29.9166667+00:00
Tamil Selvan M 0 Reputation points
edited an answer 2024-05-31T13:04:39.7066667+00:00
Vidya Viraktamath 0 Reputation points Microsoft Employee
2 answers
asked 2024-05-29T14:14:58.2933333+00:00
NK 0 Reputation points
commented 2024-05-31T12:28:37.3866667+00:00
Smaran Thoomu 11,045 Reputation points Microsoft Vendor
2 answers

couldnt able to download the files completely by using copy activity download from website always downloading partially

whenever i try to download the files from website by using copy activity its downloaded partially 124.kb instead if 240MB - 350MB ..

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-25T00:50:10.38+00:00
Anilkumar Balakumaran (z) 0 Reputation points
commented 2024-05-31T09:41:55.29+00:00
Anilkumar Balakumaran 0 Reputation points
1 answer

Running specific pipelines from an Orchestration pipeline

How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,380 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,781 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,509 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,503 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T21:38:39.0933333+00:00
ADITYA SINGH 0 Reputation points
commented 2024-05-31T09:41:01.8633333+00:00
Sina Salam 4,546 Reputation points
2 answers One of the answers was accepted by the question author.

How to apply Wildcard REGEX filename from SFTP using ADF ?

I have a list of files on an SFTP server that I need to copy and move to a blob storage container. The filenames should match the following wildcard pattern: (FullName)[0-9](-)[0-9](-)[0-9]*(.)(CSV). Here are the steps I've taken: Created a new…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-31T08:39:33.4166667+00:00
WeirdMan 180 Reputation points
accepted 2024-05-31T09:05:11.5366667+00:00
WeirdMan 180 Reputation points
4 answers

Azure Synapse Analytics Failed to setup debug session

Right now I'm not able to run a data flow task on our Azure Synapse environment. When I try to start a debug session I get the message Failed to setup debug session. When I trigger the pipeline instead the task fails with "Operation on target…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,503 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2022-09-20T12:40:49.163+00:00
Anonymous
answered 2024-05-31T09:04:57.9933333+00:00
srivally chintakindi 0 Reputation points
2 answers One of the answers was accepted by the question author.

Understand costs AZ Databricks job

I have an Azure Data Factory pipeline that executes multiple Databricks Notebooks using job clusters. I need to track the cost of these job clusters, including both the Databricks and the underlying VM costs, specifically for this set of jobs. Currently,…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,985 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-31T07:58:49.75+00:00
WeirdMan 180 Reputation points
answered 2024-05-31T08:55:07.06+00:00
Vinodh247-1375 11,721 Reputation points
0 answers

ADF Data Flows Flatten nested json array values are being populated as null

Hi All I am building a data transamination with ADF data flow using a nested json array of objects , but after parse and flatten the json node itOffer.item.LeadOfer.zdeal.item[].dealNumber I am seeing that the column values are populated as null . I…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,503 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,985 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T11:06:56.13+00:00
venkat rao 45 Reputation points
edited a comment 2024-05-31T08:49:38.1266667+00:00
Harishga 4,485 Reputation points Microsoft Vendor
1 answer

Call API with dynamic URL and store JSON results in DataLake Storage

I am trying to make implement the following scenario using Data Factory: I am making multiple API calls, relying on a dyanmic URL (e.g. "url.com/api/{ID}" for a list of IDs). The resulting JSON from each of the API calls should be stored as a…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,380 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T11:55:42.17+00:00
Lucas Kluge 0 Reputation points
answered 2024-05-31T08:47:11.1566667+00:00
Amira Bedhiafi 16,951 Reputation points
4 answers

Script Activity not supporting multi-instruction with new Snowflake connector

I have a pipeline, in Azure Data Factory that has an "Activity Script" that executes 3 instructions against Snowflake. The script was working fine until I have updated the Snowflake connection in order to remove the legacy Snowflake connection…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-03-21T08:23:45.0033333+00:00
DavidR 20 Reputation points
answered 2024-05-31T08:44:24.4333333+00:00
Laan, Niels 0 Reputation points
1 answer

Choosing an Approach for Incremental Loading with Watermark in Azure Data Factory: Efficiency and Cost Considerations

Hi all, I'm working on implementing an Azure Data Factory pipeline for incremental data loading using a watermark table approach. I have identified two different approaches but am unsure which one is considered the best practice in terms of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-01-13T01:55:43.14+00:00
Lucas Medina 20 Reputation points
edited a comment 2024-05-31T08:16:33.87+00:00
AbhishekKrishna-4295 0 Reputation points
1 answer

How to compare two dates in ADF to see which is greater?

Hi there, I am using Azure Data Factory to compare two dates (one is stored in a string variable and one its utcnow) and I need to know when the utcnow is greater than the date in the stored string variable. Both are just dates no timestamps I looked at…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T12:03:14.8966667+00:00
Mike Langlois 5 Reputation points
commented 2024-05-31T08:02:21.2066667+00:00
Smaran Thoomu 11,045 Reputation points Microsoft Vendor
1 answer

Data flow failing at Sink Step when pipeline is scheduled

I have Pipeline with Data flow debug where data is sinked into Data set. When run manually it runs fine and the parquet file is created. However when scheduled in Pipeline it gives following error. Pipeline runs fine only gives error on schedule…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,503 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T07:21:04.75+00:00
Nazish Zahra 0 Reputation points
commented 2024-05-31T07:47:28.8633333+00:00
Amira Bedhiafi 16,951 Reputation points
1 answer

How to convert type of any block blob to append blob inside adls gen2? Is there any process or activity which will convert the type of my entire blob without changing the content?

I have a json file inside a data lake gen2 storage and that json file blob type is a block blob. I want to convert that blob type to append blob through any services of azure. Need to know the process in detail.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,380 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,421 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,509 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,985 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-31T06:30:48.3333333+00:00
Chakraborty, Nabarun (External) 0 Reputation points
answered 2024-05-31T07:45:10.9433333+00:00
Vinodh247-1375 11,721 Reputation points
0 answers

Except Unique Assert Transformation in Dataflow

I am trying to use Asset transformation in dataflow to check the dups in source. I am not able to get the correct result when using parameters that will have single or multiple columns as primary keys (example shared below) when using below expression. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-26T08:02:22.9033333+00:00
Vaibhav 105 Reputation points
commented 2024-05-31T07:39:10.81+00:00
Smaran Thoomu 11,045 Reputation points Microsoft Vendor
1 answer

Unable to create a cluster in Databricks getting quota exceeded error while everything is new

Azure Quota Exceeded Exception: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDADSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location:…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,985 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2024-05-30T21:08:45.6433333+00:00
Sibbinath, Akhil 0 Reputation points
answered 2024-05-31T05:35:34.79+00:00
Smaran Thoomu 11,045 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

Azure HTTP Linked service ADF Username only no password

I am trying to use Azure HTTP Linked services to connect an API which uses just a username and no password for basic auth. The password input is mandatory in Azure. Anything i can do?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
asked 2023-05-22T16:38:33.0066667+00:00
TeriTech03 26 Reputation points
answered 2024-05-30T21:05:19.4166667+00:00
Dmitriy Ryabin 1 Reputation point