Azure Datafactory (copy data activity) : filter rows data before ingesting into datawarehouse

Khadam 41 Reputation points
2020-06-11T12:27:45.447+00:00

Hi all,

I'm using a copydata activity inside datafactory to connect to azure datalake and insert a range of csv files into Azure data warehouse.
My question is : what is the best way to filter a csv file before inserting it into data base.

What is the drawbacks of insertion of all file's rows and then use a stores procedure to come with a filtering action.

PS: I don't privilege data flow

Thanks for your help

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,338 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
0 comments
Accepted answer
  1. Vaibhav Chaudhari 38,576 Reputation points
    2020-06-13T15:33:38.367+00:00

    I think, approach to dump CSV data to some temp table and then use SP to insert required records to final table should be good. I don't see any drawback here.

    If the response helped, do "Accept Answer" and upvote it - Vaibhav

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jainav Surana 1 Reputation point
    2021-09-08T05:05:20.217+00:00

    @Khadam @Vaibhav Chaudhari
    Did you insert all records in one go or one by one. I am looking some logic to validate all the records before insertion into multiple tables on-premises SQL database

    0 comments