question

MarvinFischenich-8510 avatar image
0 Votes"
MarvinFischenich-8510 asked MarvinFischenich-8510 answered

Store WebActivity output on ADLS

Hi guys i invoke an endpoint via a WebActivity.
The response looks like this:

 {
     "Links": {},
     "Items": [
         {
             "WebId": "123",
             "Id": "456",
             "Name": "ABC",
             "Path": "\\\\Server[DEF]",
             "IsConnected": false,
             "ServerVersion": "",
             "ServerTime": null,
             "Links": {
                 "Self": "https://DEF/api/data/789",
                 "Points": "https://DEF/api/data/890/points",
                 "EnumerationSets": "https://DEF/api/dataservers/901/enumerationsets"
             }
         }
     ],
     "ADFWebActivityResponseHeaders": {
         "Transfer-Encoding": "chunked",
         "X-Frame-Options": "SAMEORIGIN",
         "Date": "Wed, 25 Aug 2021 07:58:22 GMT",
         "Server": "Microsoft-HTTPAPI/2.0",
         "Content-Type": "application/json; charset=utf-8"
     },
     "effectiveIntegrationRuntime": "shir-adf-euw-p(shir-adf-euw-p)",
     "executionDuration": 0,
     "durationInQueue": {
         "integrationRuntimeQueue": 7
     },
     "billingReference": {
         "activityType": "ExternalActivity",
         "billableDuration": [
             {
                 "meterType": "SelfhostedIR",
                 "duration": 0.016666666666666666,
                 "unit": "Hours"
             }
         ]
     }
 }

How can i save that response on ADLS as json-File?
Preferable with ADF standard activites and not Azure Functions or Batch.

Thanks for your support!

azure-data-factoryazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarvinFischenich-8510 avatar image
0 Votes"
MarvinFischenich-8510 answered

solved it with REST dataset on source and JSON dataset on sink

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

NandanHegde-7720 avatar image
0 Votes"
NandanHegde-7720 answered
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarvinFischenich-8510 avatar image
0 Votes"
MarvinFischenich-8510 answered MarvinFischenich-8510 published

Thanks for your response!

From what i saw, the solution will only work if the WebActivity output contains only one value for one column, right?
However, i would need the JSON to hold all "Items":

  "Items": [
      {
          "WebId": "123",
          "Id": "456",
          "Name": "ABC",
          "Path": "\\\\Server[DEF]",
          "IsConnected": false,
          "ServerVersion": "",
          "ServerTime": null,
          "Links": {
              "Self": "https://DEF/api/data/789",
              "Points": "https://DEF/api/data/890/points",
              "EnumerationSets": "https://DEF/api/dataservers/901/enumerationsets"
          }
      }
  ],

I now hava a .json on my ADLS what is a step in the right direction but the result looks shitty:

 {"test":"[{\"WebId\":\"123\",\"Id\":\"456\",\"Name\":\"ABC\",\"Path\":\"\\\\\\\\Server[DEF]\",\"IsConnected\":true,\"ServerVersion\":\"3.4.425.1435\",\"ServerTime\":\"2021-08-25T12:58:24.8962426Z\",\"Links\":{\"Self\":\"https://DEF/api/data/789\",\"Points\":\"https://DEF/api/data/890/points\",\"EnumerationSets\":\"https://DEF/api/dataservers/901/enumerationsets"}}]"}

And preferable i would like to use the same pipeline for multiple endpoints which respond differnt columns/values.
So hardcoding a mapping wouldn't be my prefered way to go.

Do you also have a solution for that?

Thanks a lot!



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.