MedianStoppingPolicy Class

  • Reference

Defines an early termination policy based on running averages of the primary metric of all runs.

Initialize a MedianStoppingPolicy.

Inheritance
azureml.train.hyperdrive.policy.EarlyTerminationPolicy
MedianStoppingPolicy

Constructor

MedianStoppingPolicy(evaluation_interval=1, delay_evaluation=0)

Parameters

evaluation_interval
int
default value: 1

The frequency for applying the policy.

delay_evaluation
int
default value: 0

The number of intervals for which to delay the first policy evaluation. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

evaluation_interval
int
Required

The frequency for applying the policy.

delay_evaluation
int
Required

The number of intervals for which to delay the first policy evaluation. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

Remarks

The Median Stopping policy computes running averages across all runs and cancels runs whose best performance is worse than the median of the running averages. Specifically, a run will be canceled at interval N if its best primary metric reported up to interval N is worse than the median of the running averages for intervals 1:N across all runs.

The Median Stopping policy takes the following optional configuration parameters:

  • evaluation_interval: The frequency for applying the policy. Each time the training script logs the primary metric counts as one interval.

  • delay_evaluation: The number of intervals to delay policy evaluation. Use this parameter to avoid premature termination of training runs. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

This policy is inspired from the research publication Google Vizier: A Service for Black-Box Optimization.

If you are looking for a conservative policy that provides savings without terminating promising jobs, you can use a Median Stopping Policy with evaluation_interval 1 and delay_evaluation 5. These are conservative settings, that can provide approximately 25%-35% savings with no loss on primary metric (based on our evaluation data).

Attributes

delay_evaluation

Return the value for the number of sequences the first evaluation is delayed.

Returns

The delay evaluation.

Return type

int

evaluation_interval

Return evaluation interval value.

Returns

The evaluation interval.

Return type

int

POLICY_NAME 

POLICY_NAME = 'MedianStopping'