Keyword: Curriculum

  • مقالة
  • قراءة خلال 6 دقائق

The curriculum statement is used within the concept statement to define how the training engine should train the AI. Define lessons to create a staged teaching plan and adjust optional training parameters to control how training episodes run.

Training for the curriculum stops when any of the following conditions are met:

  • Manually stopped by the user.
  • Training appears to have converged (the AI is no longer improving).
  • Limits on curriculum training parameters (for example, TotalIterationLimit) are reached.

During training, the platform periodically runs assessments consisting of groups of test episodes. Assessments produce the following information for all goals in the curriculum:

  1. Success rate: Success rate is a summary metric indicating the fraction of test episodes within an assessment where the AI achieves a given objective. Success is a binary measure (pass/fail) at the episode level while overall success rate is calculated across all episodes.
  2. Goal satisfaction rate: Goal satisfaction rate is a summary and episode-level percentage metric indicating how close the AI came to satisfying the associated objective during the episode, regardless of success. An AI may receive a high satisfaction rate for coming very close, despite ultimately failing in the objective. For example, an AI attempting to raise and hold temperature at 30°C may only reach 29°C in a given episode. Such an episode would have a high goal satisfaction rate, even if the AI ultimately failed. A 100% satisfaction rate is only possible if the AI successfully completes the objective.

See Keyword: Goal for objective-specific training results.

Usage

Important

There can be only one curriculum per concept, and every learned concept must have a curriculum.

Every curriculum must provide a source clause that specifies the data source for teaching the concept. Simulators are the only supported data source at this time. See Keyword: simulator for more information.

concept MyConcept(input): OutputType {
  curriculum {
    source MySimulator

    # Lessons specified here

  }
}

Curriculum training parameters

You can adjust some training parameters with the training clause:

Parameter Values Default Description
EpisodeIterationLimit Number.UInt32 1000 Total iterations allowed per training episode.
TotalIterationLimit Number.UInt32 50,000,000 Total iterations allowed for the concept.
NoProgressIterationLimit Number.UInt32 250,000 Number of iterations allowed with no improvement before training auto-terminates.
LessonRewardThreshold number None Minimum reward value that counts as success.
LessonSuccessThreshold number<0 .. 1> 0.90 (90%) Minimum success rate to complete the lesson.
LessonAssessmentWindow Number.UInt32 30 Number of episodes per assessment. Used to compute LessonRewardThreshold and LessonSuccessThreshold.

For example:

concept MyConcept(input: SimState): BrainAction {
  curriculum {
    training {
      EpisodeIterationLimit: 250,
      TotalIterationLimit: 100000
    }
  }
}

EpisodeIterationLimit

A new training episode begins after EpisodeIterationLimit iterations if the brain fails to reach a valid terminal condition.

TotalIterationLimit

The training engine ends training after TotalIterationLimit iterations, even if training performance is still improving. See also NoProgressIterationLimit.

Note

The actual number of training iterations may go slightly beyond the iteration limit to allow the last training batch to complete.

LessonRewardThreshold

Only supported for reward- and terminal-function based curriculum.

When a sufficient fraction of test episodes in an assessment (as indicated by LessonSuccessThreshold) have a cumulative reward that meets or exceeds the LessonRewardThreshold value, the training engine considers the lesson complete and moves to the next lesson in the curriculum.

If the lesson definition does not include a reward threshold, the training engine uses a general convergence test to determine when the lesson is complete.

LessonSuccessThreshold

When the episode success rate in an assessment exceeds the LessonSuccessThreshold value, the training engine considers the lesson complete and moves to the next lesson in the curriculum.

The value must be between 0 and 1 and represents a target fraction.

NoProgressIterationLimit

Training stops when brain performance has not improved in NoProgressIterationLimit iterations. The NoProgressIterationLimit iteration counter resets when training moves between lessons.

The NoProgressIterationLimit iteration counter does not reset when training is stopped or restarted. If training auto-terminates and you want to continue, you can increase the value of NoProgressIterationLimit and resume training.

Note

Progress checks happen after each assessment. The actual number of training iterations may go beyond the iteration limit to allow an assessment to finish and the last training batch to complete.

LessonAssessmentWindow

Sets the number of test episodes per assessment. Assessments are groups of test episodes periodically run to evaluate the AI during training. Lesson transitions based on the LessonSuccessThreshold and LessonRewardThreshold parameters are evaluated after each assessment. Auto-termination of training (see NoProgressIterationLimit) is also based on assessment performance.

Transform functions

In some cases, it may be necessary to translate communication between the simulator and the AI during training. For example, the simulator may produce more information than will be available to the AI when it is deployed. And action instructions coming from the AI may need to have a different format in the simulated and real environments.

In this case, you can use a transform function to ensure the incoming information is configured appropriately.

Inkling supports the following transform functions:

  • State transform: used to translate information about the environment for consumption by the AI. For example, applying scaling values, converting measurement units, or aggregating values.
  • Action transform: used to translate AI instructions for application within the simulated environment.

Tip

State and action transform functions that are only used once can be inlined instead of defining them as named functions within the global scope.

State transform functions

The state data provided by a simulator often represents the full, observable state of the simulated environment. In many cases, that results in the simulator producing more observable information than the AI can (or should) act on.

If the concept input type is a proper subset of the state type provided by the simulator, the Inkling compiler automatically transforms the input data into the appropriate format.

Tip

To be a proper subset of the observable state type, all fields of the input type must also exist in, and be compatible with, the set of fields in the observable state type.

To perform a more sophisticated transform on the state data, you must declare a state transform function and reference it with the state keyword in the curriculum statement.

# Observable state type
type SimState {
  AngleInRadians: number
}

# Desired input format for AI training
type SensorState {
  AngleInDegrees: number
}

# State transform function definition
function TranslateState(State: SimState): SensorState {
  # Convert from radians to degrees.
  return {
    AngleInDegrees: State.AngleInRadians * 180 / Math.Pi
  }
}

# Curriculum definition
concept MyConcept(input: SensorState): Action {
    curriculum {
        source simulator (action: Action): SimState {}
        state TranslateState
    }
}

Action transform functions

Action input provided by the AI is typically specified in a way that is compatible with the predicted action from the concept being trained. But it may be the case that the action space of the AI is different from the action space defined in the simulated environment. For example, a simulator may encode its accepted actions using metric units while the real-world system in which the brain will eventually be used is based on imperial units.

To translate between the two action spaces, you must declare an action transform function and reference it with the action keyword in the curriculum statement.

# Action space as understood by the simulator
type SimAction number<Left=-1, Right=1>

# Action space as understood by the AI.
# Note that the action space of the AI encodes "left" as 0
# while the action space of the simulator encodes "left" as -1
type BrainAction number<Left=0, Right=1>

# Action transform function definition
function TranslateAction(Command: BrainAction): SimAction {
  if Command == BrainAction.Left {
    return SimAction.Left
  } else {
    return SimAction.Right
  }
}

# Curriculum definition
concept MyConcept(input: SimState): BrainAction {
    curriculum {
        source simulator (action: SimAction): SimState {}
        action TranslateAction
    }
}

Action masking

See Keyword: Mask for details on using action masks, which restrict the set of available actions for particular states.