question

IvanNel-1835 avatar image
0 Votes"
IvanNel-1835 asked MartinJaffer-MSFT commented

ADF Copy Task - Source Varying Column count

ADF reading multiple txt files, there are two types of txt files, 1 type has 18 cols, the other has 24.
The first 18 columns are the same in both types of files, Type 2 only has an additional 6.

If i create my sink mapping with a 24 col file, the pipeline fails when it tries to read an 18 col file

Column 'Prop_19' specified in column mapping cannot be found in source data.

My sink table has 24 columns.

azure-data-factoryazure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@IvanNel-1835 if @NandanHegde-7720 's response solved your issue, please mark as accepted answer. If not, please share additional info.

0 Votes 0 ·

1 Answer

NandanHegde-7720 avatar image
2 Votes"
NandanHegde-7720 answered

Hey,
You can use getmetadata activity to identify the structure of header and the column count in the txt files.
Based on the column count, you can create seperate copy activity via IF activity/Switch activity.

One can also leverage a single copy activity and create dynamic mappings at run time based on column count:
https://sqlitybi.com/dynamically-set-copy-activity-mappings-in-azure-data-factory-v2/


Similar thread : https://stackoverflow.com/questions/68800592/adf-if-condition-count-source-columns/68800648#68800648

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.