Append to an existing xml file in ADLS

BalwinderKaur-5937 41

I have created a pipeline in synapse that has two web activities the first one fetches the data from a REST API and other one stores that data in ADLS. is there any activity or option that I can use to append the data in the storage account instead of overwriting the existing data. ![137239-image.png][1] ![137284-image.png][2] [1]: /api/attachments/137239-image.png?platform=QnA [2]: /api/attachments/137284-image.png?platform=QnA

1 answer

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-05T12:58:33.907+00:00
Hello anonymous user,

Thanks for the question and using MS Q&A platform.

From your question I'm understanding that you would like to append the XML Data to a single file rather than overwriting.

You could make use of Append Blobs in place of Block Blobs.

Referring this link

Step 1 :

To create AppendBlob - You will have to invoke a PUT request with the header x-ms-blob-type = AppendBlob with content length = 0 . This is a one time process.

Note : You could either create it manually or dynamically check whether the blob exist, if not create a AppendBlob with 0 content length.

**Step 2: **

Once the blob has been created you can append the data- by making use of the below endpoint. (Reference)

https://storagename.blob.core.windows.net/container/directory/filename.xml?comp=appendblock

Again, you will have to pass the header x-ms-blob-type = AppendBlob in addition to other headers(like x-ms-version etc) .

Output :

At my end, I tried appending the same XML data twice.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
BalwinderKaur-5937 41 Reputation points

2021-10-05T21:51:55.61+00:00

Hello Vijaya,
Thank you for your reply. Actually I read about this process on Microsoft docs too. The issue I am experiencing now is the creation of the Append blob. How did you create the append blob. I used a web activity to invoke the put request and in the settings of the web activity I entered

url : https://mystorageacc.blob.core.windows.net/containername/filename.xml
headers : x-ms-blob-type = AppendBlob
content length = 0
entered "{}" in the body section

getting this authentication header error. (Chose managed identity Authentication method.)
"<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.<AuthenticationErrorDetail>Authentication scheme Bearer is not supported in this version."

tried creating Authorization header with value [sharedkey accountname : keyvalue] , didn't work please advise.

Thank you, appreciate your help.

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-06T16:34:46.96+00:00

Hello anonymous user,

I was also using the Web Activity invoke and was able to successfully create an append blob.

I had the below configuration for the headers

URL in the below format :

https://<accountname>.blob.core.windows.net/container4/Test01/XML6.xml

Method was set to PUT

Headers :

x-ms-blob-type AppendBlob x-ms-version 2017-04-17 Authorization SharedKey <accountname>:<SIGNEDSTRING> x-ms-date Wed, 06 Oct 2021 16:02:40 GMT

Can you pls use the above configuration at your end replacing with your URL, account name and Signed string & check ?

BalwinderKaur-5937 41 Reputation points

2021-10-06T18:23:24.033+00:00

Hello Vijaya,

I did try with all those headers and values but still getting the error for the Authorization header format.
For the authorization header where it says signedstring I entered Access key value. is that right?

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-08T05:56:13.97+00:00

Hello anonymous user,

from your question I understood - you're authorizing with the SharedKey ( [sharedkey accountname : keyvalue])

Reference : https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key

Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"

Per the doc :

The string to sign that I had used is below :

PUT x-ms-blob-type:AppendBlob x-ms-date:Fri, 08 Oct 2021 05:38:21 GMT x-ms-version:2017-04-17 /<storageaccount>/<container>/<folder>/filename.xml

The below is the sample PS Script to generate the Signature ( Signed String) - you would need the account key from the storage account and $stringToSign with the above value.

$hmacsha = New-Object System.Security.Cryptography.HMACSHA256 $hmacsha.key = [Convert]::FromBase64String($accountkey) $signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign)) $signature = [Convert]::ToBase64String($signature)

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-11T08:04:52.417+00:00

Hello anonymous user,

Just checking in if you have had a chance to see the previous response. Please do let us know if you have any query.

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-12T14:01:31.257+00:00

Hello anonymous user,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-18T15:21:09.06+00:00

@bkaur - There is a separate request for creating append blob and adding the data (Step 2) in the answer. Please let me know if you have any questions.

BalwinderKaur-5937 41 Reputation points

2021-10-20T17:01:41.28+00:00

Hello Vijaya, I used the sas authorization method and was able to create the append blob and add data to it.
Thank you for your response.

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-10-25T03:06:59.297+00:00

Hello anonymous user,

Glad, the issue was resolved. Please consider upvoting or accepting solution as an answer if the information helps you. Original posters help the community find answers faster by identifying the correct answer
Sign in to comment