Web activity in Azure Data Factory

Web Activity can be used to call a custom REST endpoint from a Data Factory pipeline. You can pass datasets and linked services to be consumed and accessed by the activity.

Note

This article applies to version 2 of Data Factory, which is currently in preview. If you are using version 1 of the Data Factory service, which is generally available (GA), see documentation for Data Factory version 1.

Syntax

{  
   "name":"MyWebActivity",
   "type":"WebActivity",
   "typeProperties":{  
      "method":"Post",
      "url":"<URLEndpoint>",
      "headers":{  
         "Content-Type":"application/json"
      },
      "authentication":{  
         "type":"ClientCertificate",  
         "pfx":"****",
         "password":"****"
      },
      "datasets":[  
         {  
            "referenceName":"<ConsumedDatasetName>",
            "type":"DatasetReference",
            "parameters":{  
               ...
            }
         }
      ],
      "linkedServices":[  
         {  
            "referenceName":"<ConsumedLinkedServiceName>",
            "type":"LinkedServiceReference"
         }
      ]
   }
}

Type properties

Property Description Allowed values Required
name Name of the web activity String Yes
type Must be set to WebActivity. String Yes
method Rest API method for the target endpoint. String.

Supported Types: "GET", "POST", "PUT"
Yes
url Target endpoint and path String (or expression with resultType of string). The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint. Yes
headers Headers that are sent to the request. For example, to set the language and type on a request: "headers" : { "Accept-Language": "en-us", "Content-Type": "application/json" }. String (or expression with resultType of string) Yes, Content-type header is required. "headers":{ "Content-Type":"application/json"}
body Represents the payload that is sent to the endpoint. Required for POST/PUT methods. String (or expression with resultType of string).

See the schema of the request payload in Request payload schema section.
No
authentication Authentication method used for calling the endpoint. Supported Types are "Basic, or ClientCertificate." For more information, see Authentication section. If authentication is not required, exclude this property. String (or expression with resultType of string) No
datasets List of datasets passed to the endpoint. Array of dataset references. Can be an empty array. Yes
linkedServices List of linked services passed to endpoint. Array of linked service references. Can be an empty array. Yes

Note

REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint.

The following table shows the requirements for JSON content:

Value type Request body Response body
JSON object Supported Supported
JSON array Supported
(At present, JSON arrays don't work as a result of a bug. A fix is in progress.)
Unsupported
JSON value Supported Unsupported
Non-JSON type Unsupported Unsupported

Authentication

None

If authentication is not required, do not include the "authentication" property.

Basic

Specify user name and password to use with the basic authentication.

"authentication":{  
   "type":"Basic,
   "username":"****",
   "password":"****"
}

Client certificate

Specify base64-encoded contents of a PFX file and the password.

"authentication":{  
   "type":"ClientCertificate",
   "pfx":"****",   
   "password":"****"
}

Request payload schema

When you use the POST/PUT method, the body property represents the payload that is sent to the endpoint. You can pass linked services and datasets as part of the payload. Here is the schema for the payload:

{
    "body": {
        "myMessage": "Sample",
        "datasets": [{
            "name": "MyDataset1",
            "properties": {
                ...
            }
        }],
        "linkedServices": [{
            "name": "MyStorageLinkedService1",
            "properties": {
                ...
            }
        }]
    }
} 

Example

In this example, the web activity in the pipeline calls a REST end point. It passes an Azure SQL linked service and an Azure SQL dataset to the endpoint. The REST end point uses the Azure SQL connection string to connect to the Azure SQL server and returns the name of the instance of SQL server.

Pipeline definition

{
    "name": "<MyWebActivityPipeline>",
    "properties": {
        "activities": [
            {
                "name": "<MyWebActivity>",
                "type": "WebActivity",
                "typeProperties": {
                    "method": "Post",
                    "url": "@pipeline().parameters.url",
                    "headers": {
                        "Content-Type": "application/json"
                    },
                    "authentication": {
                        "type": "ClientCertificate",
                        "pfx": "*****",
                        "password": "*****"
                    },
                    "datasets": [
                        {
                            "referenceName": "MySQLDataset",
                            "type": "DatasetReference",
                            "parameters": {
                                "SqlTableName": "@pipeline().parameters.sqlTableName"
                            }
                        }
                    ],
                    "linkedServices": [
                        {
                            "referenceName": "SqlLinkedService",
                            "type": "LinkedServiceReference"
                        }
                    ]
                }
            }
        ],
        "parameters": {
            "sqlTableName": {
                "type": "String"
            },
            "url": {
                "type": "String"
            }
        }
    }
}

Pipeline parameter values

{
    "sqlTableName": "department",
    "url": "https://adftes.azurewebsites.net/api/execute/running"
}

Web service endpoint code


[HttpPost]
public HttpResponseMessage Execute(JObject payload)
{
    Trace.TraceInformation("Start Execute");

    JObject result = new JObject();
    result.Add("status", "complete");

    JArray datasets = payload.GetValue("datasets") as JArray;
    result.Add("sinktable", datasets[0]["properties"]["typeProperties"]["tableName"].ToString());

    JArray linkedServices = payload.GetValue("linkedServices") as JArray;
    string connString = linkedServices[0]["properties"]["typeProperties"]["connectionString"].ToString();

    System.Data.SqlClient.SqlConnection sqlConn = new System.Data.SqlClient.SqlConnection(connString);

    result.Add("sinkServer", sqlConn.DataSource);

    Trace.TraceInformation("Stop Execute");

    return this.Request.CreateResponse(HttpStatusCode.OK, result);
}

Next steps

See other control flow activities supported by Data Factory: