Run job preparation and job release tasks on Batch compute nodes

An Azure Batch job often requires some form of setup before its tasks are executed, and post-job maintenance when its tasks are completed. You might need to download common task input data to your compute nodes, or upload task output data to Azure Storage after the job completes. You can use job preparation and job release tasks to perform these operations.

What are job preparation and release tasks?

Before a job's tasks run, the job preparation task runs on all compute nodes scheduled to run at least one task. Once the job is completed, the job release task runs on each node in the pool that executed at least one task. As with normal Batch tasks, you can specify a command line to be invoked when a job preparation or release task is run.

Job preparation and release tasks offer familiar Batch task features such as file download (resource files), elevated execution, custom environment variables, maximum execution duration, retry count, and file retention time.

In the following sections, you'll learn how to use the JobPreparationTask and JobReleaseTask classes found in the Batch .NET library.

Tip

Job preparation and release tasks are especially helpful in "shared pool" environments, in which a pool of compute nodes persists between job runs and is used by many jobs.

When to use job preparation and release tasks

Job preparation and job release tasks are a good fit for the following situations:

Download common task data

Batch jobs often require a common set of data as input for the job's tasks. For example, in daily risk analysis calculations, market data is job-specific, yet common to all tasks in the job. This market data, often several gigabytes in size, should be downloaded to each compute node only once so that any task that runs on the node can use it. Use a job preparation task to download this data to each node before the execution of the job's other tasks.

Delete job and task output

In a "shared pool" environment, where a pool's compute nodes are not decommissioned between jobs, you may need to delete job data between runs. You might need to conserve disk space on the nodes, or satisfy your organization's security policies. Use a job release task to delete data that was downloaded by a job preparation task, or generated during task execution.

Log retention

You might want to keep a copy of log files that your tasks generate, or perhaps crash dump files that can be generated by failed applications. Use a job release task in such cases to compress and upload this data to an Azure Storage account.

Tip

Another way to persist logs and other job and task output data is to use the Azure Batch File Conventions library.

Job preparation task

Before execution of a job's tasks, Batch executes the job preparation task on each compute node that is scheduled to run a task. By default, the Batch service waits for the job preparation task to be completed before running the tasks scheduled to execute on the node. However, you can configure the service not to wait. If the node restarts, the job preparation task runs again, but you can also disable this behavior.

The job preparation task is executed only on nodes that are scheduled to run a task. This prevents the unnecessary execution of a preparation task in case a node is not assigned a task. This can occur when the number of tasks for a job is less than the number of nodes in a pool. It also applies when concurrent task execution is enabled, which leaves some nodes idle if the task count is lower than the total possible concurrent tasks. By not running the job preparation task on idle nodes, you can spend less money on data transfer charges.

Note

JobPreparationTask differs from CloudPool.StartTask in that JobPreparationTask executes at the start of each job, whereas StartTask executes only when a compute node first joins a pool or restarts.

Job release task

Once a job is marked as completed, the job release task is executed on each node in the pool that executed at least one task. You mark a job as completed by issuing a terminate request. The Batch service then sets the job state to terminating, terminates any active or running tasks associated with the job, and runs the job release task. The job then moves to the completed state.

Note

Job deletion also executes the job release task. However, if a job has already been terminated, the release task is not run a second time if the job is later deleted.

Job prep and release tasks with Batch .NET

To use a job preparation task, assign a JobPreparationTask object to your job's CloudJob.JobPreparationTask property. Similarly, initialize a JobReleaseTask and assign it to your job's CloudJob.JobReleaseTask property to set the job's release task.

In this code snippet, myBatchClient is an instance of BatchClient, and myPool is an existing pool within the Batch account.

// Create the CloudJob for CloudPool "myPool"
CloudJob myJob =
    myBatchClient.JobOperations.CreateJob(
        "JobPrepReleaseSampleJob",
        new PoolInformation() { PoolId = "myPool" });

// Specify the command lines for the job preparation and release tasks
string jobPrepCmdLine =
    "cmd /c echo %AZ_BATCH_NODE_ID% > %AZ_BATCH_NODE_SHARED_DIR%\\shared_file.txt";
string jobReleaseCmdLine =
    "cmd /c del %AZ_BATCH_NODE_SHARED_DIR%\\shared_file.txt";

// Assign the job preparation task to the job
myJob.JobPreparationTask =
    new JobPreparationTask { CommandLine = jobPrepCmdLine };

// Assign the job release task to the job
myJob.JobReleaseTask =
    new JobReleaseTask { CommandLine = jobReleaseCmdLine };

await myJob.CommitAsync();

As mentioned earlier, the release task is executed when a job is terminated or deleted. Terminate a job with JobOperations.TerminateJobAsync. Delete a job with JobOperations.DeleteJobAsync. You typically terminate or delete a job when its tasks are completed, or when a timeout that you've defined has been reached.

// Terminate the job to mark it as Completed; this will initiate the
// Job Release Task on any node that executed job tasks. Note that the
// Job Release Task is also executed when a job is deleted, thus you
// need not call Terminate if you typically delete jobs after task completion.
await myBatchClient.JobOperations.TerminateJobAsync("JobPrepReleaseSampleJob");

Code sample on GitHub

To see job preparation and release tasks in action, check out the JobPrepRelease sample project on GitHub. This console application does the following:

  1. Creates a pool with two "small" nodes.
  2. Creates a job with job preparation, release, and standard tasks.
  3. Runs the job preparation task, which first writes the node ID to a text file in a node's "shared" directory.
  4. Runs a task on each node that writes its task ID to the same text file.
  5. Once all tasks are completed (or the timeout is reached), prints the contents of each node's text file to the console.
  6. When the job is completed, runs the job release task to delete the file from the node.
  7. Prints the exit codes of the job preparation and release tasks for each node on which they executed.
  8. Pauses execution to allow confirmation of job and/or pool deletion.

Output from the sample application is similar to the following:

Attempting to create pool: JobPrepReleaseSamplePool
Created pool JobPrepReleaseSamplePool with 2 small nodes
Checking for existing job JobPrepReleaseSampleJob...
Job JobPrepReleaseSampleJob not found, creating...
Submitting tasks and awaiting completion...
All tasks completed.

Contents of shared\job_prep_and_release.txt on tvm-2434664350_1-20160623t173951z:
-------------------------------------------
tvm-2434664350_1-20160623t173951z tasks:
  task001
  task004
  task005
  task006

Contents of shared\job_prep_and_release.txt on tvm-2434664350_2-20160623t173951z:
-------------------------------------------
tvm-2434664350_2-20160623t173951z tasks:
  task008
  task002
  task003
  task007

Waiting for job JobPrepReleaseSampleJob to reach state Completed
...

tvm-2434664350_1-20160623t173951z:
  Prep task exit code:    0
  Release task exit code: 0

tvm-2434664350_2-20160623t173951z:
  Prep task exit code:    0
  Release task exit code: 0

Delete job? [yes] no
yes
Delete pool? [yes] no
yes

Sample complete, hit ENTER to exit...

Note

Due to the variable creation and start time of nodes in a new pool (some nodes are ready for tasks before others), you may see different output. Specifically, because the tasks complete quickly, one of the pool's nodes may execute all of the job's tasks. If this occurs, you will notice that the job prep and release tasks do not exist for the node that executed no tasks.

Inspect job preparation and release tasks in the Azure portal

When you run the sample application, you can use the Azure portal to view the properties of the job and its tasks, or even download the shared text file that is modified by the job's tasks.

The screenshot below shows the Preparation tasks blade in the Azure portal after a run of the sample application. Navigate to the JobPrepReleaseSampleJob properties after your tasks have completed (but before deleting your job and pool) and click Preparation tasks or Release tasks to view their properties.

Job preparation properties in Azure portal

Next steps

Application packages

In addition to the job preparation task, you can also use the application packages feature of Batch to prepare compute nodes for task execution. This feature is especially useful for deploying applications that do not require running an installer, applications that contain many (100+) files, or applications that require strict version control.

Installing applications and staging data

This MSDN forum post provides an overview of several methods of preparing your nodes for running tasks:

Installing applications and staging data on Batch compute nodes

Written by one of the Azure Batch team members, it discusses several techniques that you can use to deploy applications and data to compute nodes.