question

BalwinderKaur-5937 avatar image
0 Votes"
BalwinderKaur-5937 asked SathyamoorthyVijayakumar-MSFT commented

Append to an existing xml file in ADLS

I have created a pipeline in synapse that has two web activities the first one fetches the data from a REST API and other one stores that data in ADLS. is there any activity or option that I can use to append the data in the storage account instead of overwriting the existing data. ![137239-image.png][1] ![137284-image.png][2] [1]: /answers/storage/attachments/137239-image.png [2]: /answers/storage/attachments/137284-image.png

azure-data-factoryazure-synapse-analyticsazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SathyamoorthyVijayakumar-MSFT avatar image
0 Votes"
SathyamoorthyVijayakumar-MSFT answered SathyamoorthyVijayakumar-MSFT commented

Hello anonymous user,

Thanks for the question and using MS Q&A platform.

From your question I'm understanding that you would like to append the XML Data to a single file rather than overwriting.

You could make use of Append Blobs in place of Block Blobs.

137726-image.png
Referring this link

Step 1 :

To create AppendBlob - You will have to invoke a PUT request with the header x-ms-blob-type = AppendBlob with content length = 0 . This is a one time process.

Note : You could either create it manually or dynamically check whether the blob exist, if not create a AppendBlob with 0 content length.

Step 2:

Once the blob has been created you can append the data- by making use of the below endpoint. (Reference)

https://storagename.blob.core.windows.net/container/directory/filename.xml?comp=appendblock

Again, you will have to pass the header x-ms-blob-type = AppendBlob in addition to other headers(like x-ms-version etc) .


Output :

At my end, I tried appending the same XML data twice.

137785-image.png


Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (20.4 KiB)
image.png (13.3 KiB)
· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello Vijaya,
Thank you for your reply. Actually I read about this process on Microsoft docs too. The issue I am experiencing now is the creation of the Append blob. How did you create the append blob. I used a web activity to invoke the put request and in the settings of the web activity I entered

url : https://mystorageacc.blob.core.windows.net/containername/filename.xml
headers : x-ms-blob-type = AppendBlob
content length = 0
entered "{}" in the body section

getting this authentication header error. (Chose managed identity Authentication method.)
"<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.<AuthenticationErrorDetail>Authentication scheme Bearer is not supported in this version."

tried creating Authorization header with value [sharedkey accountname : keyvalue] , didn't work please advise.


Thank you, appreciate your help.


0 Votes 0 ·

Hello anonymous user,

I was also using the Web Activity invoke and was able to successfully create an append blob.

138190-image.png


I had the below configuration for the headers

138233-image.png

URL in the below format :

  https://<accountname>.blob.core.windows.net/container4/Test01/XML6.xml

Method was set to PUT

Headers :

 x-ms-blob-type               AppendBlob                                                                              
 x-ms-version                   2017-04-17                                                                              
 Authorization                  SharedKey <accountname>:<SIGNEDSTRING>
 x-ms-date                      Wed, 06 Oct 2021 16:02:40 GMT   

Can you pls use the above configuration at your end replacing with your URL, account name and Signed string & check ?



0 Votes 0 ·
image.png (3.3 KiB)
image.png (40.5 KiB)
BalwinderKaur-5937 avatar image BalwinderKaur-5937 SathyamoorthyVijayakumar-MSFT ·

Hello Vijaya,

I did try with all those headers and values but still getting the error for the Authorization header format.
For the authorization header where it says signedstring I entered Access key value. is that right?

0 Votes 0 ·
Show more comments

Hello anonymous user,

Glad, the issue was resolved. Please consider upvoting or accepting solution as an answer if the information helps you. Original posters help the community find answers faster by identifying the correct answer

0 Votes 0 ·