question

AnkitKumar-6861 avatar image
0 Votes"
AnkitKumar-6861 asked KranthiPakala-MSFT commented

How to append a string value in the column value in Copy Activity in ADF

In below copy activity screen shot , i am trying to create a new column(PartitionKey) whose value is same as DocType column. I would also like to insert one string value along with Doctype value. something like concat("recoding",$.Doctype) as value of PartitionKey. I am unable to do so and would like to know if this is possible to do.

I want this because I get error for some items where DocType key is not present. and I would like to avoid that.

The error message i get is below and I am not sure for which item this error occurs as there are thousands of items getting copied.

ErrorCode=DuplicateColumnNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=,Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column name '$.DocType' to be duplicated is not found in the source data. Please check the source column name

115403-image.png



azure-data-factory
image.png (22.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @AnkitKumar-6861,

Thanks for using this forum and posting your query.

  1. The reason for this error is because you are using an incorrect syntax $.DocType, it should be just the column name as DocType . This would avoid the error and copy the values of DocType column to your newly added column PartitionKey.

    115647-image.png

  2. For the additional requirement of concatenating a string value to this additional column - I don't think this is possible within copy activity and Copy activity is not intended for this purpose. If you would to do a data transformation then I would recommend you to please use Mapping Data flow in ADF which is better suited for transformations. Here is a list of supported transformations by Mapping dataflow : Mapping data flow transformation overview
    You can use a Derived column transformation which would help to achieve your requirement.

Hope this info helps. Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.







image.png (54.2 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I used derived column in mapping data flow and it worked fine, but with mapping data flow the issue is I don't get all the values of my document in sink. its probably because data flow takes only the first 10 documents as sample and deduce the schema from there and if any new columns comes after these 10 documents then those are not taken.

0 Votes 0 ·

Hi @AnkitKumar-6861,

Sorry for the delayed response. It shouldn't be that behavior. If you are trying to preview the data then it is limited to a sample dataset/records as you mentioned, but when you Debug run/Trigger now the pipeline then all the data from source to be copied to sink. If this is not the behavior you have noticed, then I would recommend you to please open a new thread with as much as information possible (like sample source data without sensitive info, related screenshots of the issue and output file with the limited data you are seeing). This would help us understand the root cause and assist you better.

Please let us know how it goes.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

0 Votes 0 ·

You are wrong here. Please check Troubleshoot mapping data flow documentation from Microsoft
https://docs.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-connector-format#azure-cosmos-db-and-json-format


0 Votes 0 ·
Show more comments