question

CourtneyHaedke-0265 avatar image
0 Votes"
CourtneyHaedke-0265 asked MartinJaffer-MSFT commented

Azure Data Factory Web Activity save output

32964-webactivityexample.jpgIs there a way to save the output of an Azure Data Factory Web Activity into a dataset?

Here is my current use case:

I have to dynamically build a JSON post request
The API I'm trying to reach requires a SSL certificate so I have to use the Web Activity Authentication Client Certificate option.
The API also requires basic authentication so I input the Content -Type and authorization guid in the header section of the Web Activity.
Once I get the JSON response from my post request I need to save the response into a blob storage some where.
I tried using the Copy Data Set HTTPS or Rest API as a data set source but both only allow one type of authentication certificate or Basic authentication.

Is there a way I can configure the Rest API or HTTPS data set source handle both types of authentications (SSL and Basic Authorization) or capture all the Web Activity output into a blob storage?

Thank you all for your help! I'm desperate at the moment lol..

Here is what my Web Activity looks like (sorry I had hide part of the credentials for security purposes: Web Activity


azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @CourtneyHaedke-0265 , welcome to Microsoft Q&A, and think you for your question. I see you difficulty.

I would like to break this down into 2 parts:

  1. How can I record the output of my web activity which is currently working?

  2. Can I make this work with REST dataset or HTTPS dataset instead?


I have some experience with 1. and will try to address below.
I will need to do a little digging on 2.

Web Activity takes the data away from the realm of "Copy Activity". Therefore we need to find something else to write the data. I have several ways.
In the past I have sent the result of Web Activity to Azure Function App which wrote to blob.
I have also sent the output of Web Activity as input to body of another Web Activity which called the blob REST API, and wrote directly using that.


0 Votes 0 ·

Hi Martin,

Thank you for getting back to me. Are there examples on how to sent the output of the web activity using another web activity?
Do you have an examples? I'm trying to figure out how to configure the second web activity.

0 Votes 0 ·

For placing the output of the first activity in the body of the second, the expression generally looks like:
@activity('Web1').output
or
@activity('Web1').output.data
or something similar depending upon what the output of the first activity looks like. Data Factory adds some properties to the output, such as headers, so your case will need a little customization. Do a debug run, and look at the output of the first web activity.

If you do not see the body section, check what http verb you are using. GET does not have a body, but PUT or POST do.
For example, I target a web activity at https://reqres.in/api/users/2

33413-image.png
Since I want to pass the "data" and not the headers, I use @activity('Web1').output.data
33384-image.png


1 Vote 1 ·
image.png (81.1 KiB)
image.png (54.8 KiB)

I'm reading the article you sent me. Do I need to create an authorization header in the blob storage? Is there a simpler way to generate the header? A script available some where?

0 Votes 0 ·
Show more comments
MartinJaffer-MSFT avatar image
2 Votes"
MartinJaffer-MSFT answered MartinJaffer-MSFT commented

@CourtneyHaedke-0265 I found an easier way to deal with the authorization. Use Managed Service Identity. This has 2 parts.

  1. Go to your storage account. Go to the IAM / RBAC. Give your Data Factory the Storage Blob Data Contributor role.
    Important: "Storage Blob Data Contributor" is not the same as "Contributor"

  2. Go to the web activity. In the bottom under 'advanced' select "MSI". The resource should be https://storage.azure.com/
    You may need to set the x-ms-version header to 2017-11-09 or higher.

This should effectively handle the authentication for you.

33471-image.png



image.png (124.7 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

This solution worked! Thank you for the great support! I'm excited to continue using Data Factory for my project.

0 Votes 0 ·

Thank you for the feedback @CourtneyHaedke-0265 ! I am converting some of the comments to answer to make them more visible to community. Could you please mark the appropriate one as accepted answer, or let me know if I need to convert other comments to answer please? Organizing and markign as accepted answer will improve visibility of this thread and help others benefit.

0 Votes 0 ·
CourtneyHaedke-0265 avatar image
0 Votes"
CourtneyHaedke-0265 answered

Thanks Martin! I understand that piece. What are the requirements to for the header to complete a put request to an azure blob storage. 33461-bloblstorage.jpg



bloblstorage.jpg (40.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.