question

rgioiosa avatar image
0 Votes"
rgioiosa asked rgioiosa commented

How to use Azure Batch with Prefetched Container from Custom Batch Service in Azure Data Factory

Is it possible or is there documentation on how to run a batch service command from a pipeline in an Azure Data Factory against a batch pool that has a prefetched container image?

I created a batch pool that prefetches an image from our Azure container repository and it seems to start just fine. I can run simple a simple "hello world" command from the start task. However, every time I try to run a command from the Data Factory to this linked batch service, I get the following error:

Output
{
"exitcode": null,
"errorCategory": 0,
"code": "ContainerTaskSettingsNotFound",
"message": "Container-enabled compute node requires task container settings",
"details": null,

Is it possible to set TaskContainerSettings from the Data Factory? I see some doc here on doing this programmatically, but how can I get these JSON parameters to be created and sent from the Data Factory? (https://docs.microsoft.com/en-us/rest/api/batchservice/task/add#taskcontainersettings)




azure-data-factoryazure-batchazure-container-registry
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @rgioiosa and welcome to Microsoft Q&A.

This is not something I have done before, so I will need to bring in help from my colleagues working on Batch Service to understand this better.

0 Votes 0 ·
rgioiosa avatar image rgioiosa MartinJaffer-MSFT ·

Thanks! It would be great if this is possible.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered rgioiosa commented

So I notice there is a section under 'Settings' called Extended Properties. This would be the place to supply any additional parameters.

As of May 15 2020, Data Factory did not support Containers on Azure Batch. At that time there were no plans to go beyond cluster operations.

@rgioiosa

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for taking a look at this. I tried a few tests with adding the TaskContainerSetting as part of the extended properties, but that doesn't seem to work for this issue. I still get the same error.

It would be a nice feature for Azure Data Factory to implement; the ability to send commands to container batch nodes.

0 Votes 0 ·