Tutorial: Create an assessment configuration file

هل هذه الصفحة مفيدة؟

هل لديك ملاحظات إضافية؟

سيتم إرسال الملاحظات إلى Microsoft: بالضغط على الزر «إرسال»، سيتم استخدام ملاحظاتك لتحسين منتجات Microsoft وخدماتها. نهج الخصوصية.

Follow this tutorial to learn:

• How to design custom assessments.
• How to write an assessment configuration file.
• How to translate SimConfig types to episode configurations.

To follow this tutorial, you must have a Microsoft or Azure account and a Bonsai workspace provisioned in Azure. If you need an account or Azure trial, follow the instructions in Microsoft account setup for Bonsai before continuing.

Screenshot of the Add Custom Assessment UI popup. The UI is prompting for a display name, description, and a prompt to upload a JSON file.

Design a custom assessment

In Machine Teaching, assessment means evaluating how well your AI enacts a policy across a set of scenarios (episode configurations).

Assessment lets you determine the quality and robustness of the learned behavior in simulation with data logging and, for simulations that support it, live visualizations.

When designing a custom assessment, keep the following in mind:

• Start by identifying the specific elements you want to evaluate about your policy performance and develop assessment plans for each of those elements.
• If you have a variety of scenarios you want to assess, consider breaking them into separate assessments for easier data analysis.
• Test a wide range of scenarios, including the common case, optimal cases and worst case.
• Consider scenarios that are extreme or happen infrequently (edge cases and corner cases).
• If your policy allows for unexpected or invalid starting states, include invalid scenarios in your assessment.
• If your simulation includes random probability distributions (a stochastic simulator) and you want to run a particular scenario multiple times, you need to have multiple entries of the same episode configuration.

Create an empty assessment configuration file

Bonsai uses assessment configuration files to initialize the simulation environment used to evaluate the targeted brain. Assessment configuration files are simple JSON files with three fields: version, context, and episodeConfigurations:

{
  "version": "ASSESSMENT_SCHEMA_VERSION",
  "context": {},
  "episodeConfigurations": []
}

episodeConfigurations

The core element of your assessment configuration file is the episodeConfigurations field. episodeConfiguration is an array of one or more entries that represent starting simulation configurations for your assessment based on the SimConfig type of you Inkling file. If you do not provide explicit starting values in your configuration file, Bonsai uses the default value as defined in the training simulation.

version

The version field is a string that indicates the assessment schema version used in the configuration file. Currently, the only supported version is "1.0.0". Bonsai uses the version field for version control and maintaining backward compatibility.

context

The context field is populated automatically by Bonsai after you upload the configuration file. Currently, this field cannot be used to customize your assessment and must be left empty.

To create a new assessment configuration:

1. Open your preferred plain text editor (Visual Studio Code, Atom, etc.)
2. Create a new .json file and name it with the date and the purpose of the assessment. For example, "BadTerrain-2021-01-21.json" or "HighTemps-2021-02-05.json".
3. Copy the configuration template below into your new file.

{
  "version": "1.0.0",
  "context": {},
  "episodeConfigurations": []
}

Tip

You can use the Bonsai UI to create a basic config file:

1. Open the Bonsai UI.
2. Select the brain version you want to assess.
3. Select on the Train tab.
4. Select the + New assessment button in the assessment panel.
5. Set the number of episodes for your assessment.
6. Specify a value, set of values, or value range for any configurable variables you want to constrain using Inkling syntax.
7. Click Generate.
8. Copy the generated JSON into a text editor and save the file with a .json extension.
9. Follow the rest of the tutorial to make additional customizations.

Translate your SimConfig type to episode configurations

The most important part of your assessment configuration file is the array of episode configurations. Episode configurations tell Bonsai how to configure your simulation when the assessment starts. Every assessment configuration file must have at least one episode configuration.

Episode configurations must have the same structure as the SimConfig type in the Inkling file for the brain version you want to assess.

The following Inkling type:

type SimConfig {
  ConfigField_1: Field_1_Type,
  ConfigField_2: {
    TypeSubField_1: SubField_1_Type,
    TypeSubField_2: SubField_2_Type,
  }
  ...
  ConfigField_N: Field_N_Type,
}

Translates to the following episode definition in the assessment configuration:

{
  "ConfigField_1": "Value_1",
  "ConfigField_2": {
    "TypeSubField_1": "SubValue_1",
    "TypeSubField_2": "SubValue_2",
  },
  ...
  "ConfigField_N": "Value_N",
}

Important

While most number fields are passed as numbers in the configuration file, enumerated number types must be passed as strings.

Translating single-field SimConfig types

Assume the SimConfig type in your Inkling file includes a single field called Drive of type Direction:

type Direction number<Left = 0, Right = 1, Straight = 2, Back = 3>
type SimConfig { Drive: Direction }

Every entry in the episode configuration array must provide a valid value for Drive as defined by the Direction type. For example:

[
  {"Drive": "Left"},
  {"Drive": "Right"},
  {"Drive": "Back"}
]

Since the simulation configuration information is a single value, Bonsai can map the values to the Drive field without an explicit field name. As a result, the entries in your episode configuration array can exclude the field name for simplicity, if you prefer:

[
  "Left",
  "Right",
  "Back"
]

Translating simple SimConfig types

Assume the SimConfig type in your Inkling file includes two values called Drive and RoadCondition of type Direction and Terrain:

type Direction number<Left = 0, Right = 1, Straight = 2, Back = 3>
type Terrain string<Bad = "B", Rough = "R", Clear = "C">

type SimConfig {
  Drive: Direction,
  RoadCondition: Terrain
}

Every entry in the episode configuration array must set Drive and RoadCondition fields with valid values as defined by the Direction and Terrain types.

For example, assume you want to assess the driving brain against the following two scenarios:

• Start by driving left on a rough road.
• Start by driving left on a clear road.

Your configuration file would have two episode configurations. One that sets the starting values to {Left, Rough} and one that sets the starting values to {Left, Clear}:

[
  {
    "Drive": "Left",
    "RoadCondition": "Rough"
  },
  {
    "Drive": "Left",
    "RoadCondition": "Clear"
  }

Translating SimConfig with nested dictionaries

Assume the SimConfig type in your Inkling file includes three fields:

Name	Type	Purpose
Drive	Movement	indicates the current orientation and road condition for an electric car.
BatteryLevel	number<0 .. 1>	indicates the current charge of the car battery as a percentage.
AmbientTemp	Temperature	indicates the temperature of the surrounding area in Celsius.

type Direction number<Left = 0, Right = 1, Straight = 2, Back = 3>
type Terrain string<Bad = "B", Rough = "R", Clear = "C">
type Movement {
  Orientation: Direction,
  RoadCondition: Terrain
}
type Temperature number<-90 .. 60>

type SimConfig {
  Drive: Movement,
  BatteryLevel: number<0 .. 1>,
  AmbientTemp: Temperature
}

Every entry in the episode configuration array must set valid starting values for Drive, BatteryLevel, and AmbientTemp as defined by the corresponding types.

For example, assume you want to assess the driving brain against the following scenarios:

• Start by driving left on a rough road with a full battery on a winter day in Berkeley, CA.
• Start by driving backward on a clear road with a nearly empty battery on a summer day in Berkeley, CA.
• Start by driving straight on a bad road with half a battery on a spring day in Berkeley, CA.

Your configuration file would have three episode configurations:

[
  {
    "Drive": {
      "Orientation": "Left",
      "RoadCondition": "Rough"
    },
    "BatteryLevel": 1,
    "AmbientTemp": 14
  },
  {
    "Drive": {
      "Orientation": "Back",
      "RoadCondition": "Clear"
    },
    "BatteryLevel": 0.1,
    "AmbientTemp": 22
  },
  {
    "Drive": {
      "Orientation": "Straight",
      "RoadCondition": "Bad"
    },
    "BatteryLevel": 0.5,
    "AmbientTemp": 18
  },

Next steps

Now that you know how to create assessment configuration files, try using one to assess a trained brain.