Jetpack

When a virtual machine is provisioned to become a node in a cluster, Jetpack is automatically installed when using Azure CycleCloud. Jetpack is required on every node of a cluster, as it provides three main functions:

  • Node Configuration

Azure CycleCloud uses Chef to automate the configuration of a provisioned VM into a working cluster node. A Chef client is embedded within Jetpack, and the Chef cookbooks needed for the configuration of the node are downloaded by Jetpack.

  • Distributed Synchronization

Jetpack manages communication between the node and the Azure CycleCloud application server, which enables CycleCloud to monitor the status of the provisioning VMs and synchronize the orchestration of multiple nodes in the cluster.

  • HeathCheck

Jetpack runs HealthCheck on VMs, and terminates them if they are unhealthy.

Jetpack Installation

When you first start a cluster using Azure CycleCloud, the software fetches the jetpack installer and caches it into a locker in your Azure Storage Account. When the VMs in the cluster are provisioned, a custom script extension is executed as part of the boot process. This will download the Jetpack installer from your locker cache and install it on the VM.

Jetpack Install Location

All Jetpack files are installed inside a singular directory tree:

  • Windows: C:\cycle\jetpack
  • Linux: /opt/cycle/Jetpack

In addition to creating this directory, the Jetpack installer:

  • Creates system init startup scripts which configure a VM as a cluster node
  • Creates udev rules on Linux
  • Installs the HealthCheck service
  • Sets the environment variable CYCLECLOUD_HOME

Jetpack Subdirectories

Directory Description
bin Useful binaries and scripts.
config User defined and cluster defined configuration files and scripts.
logs Logs generated by joining a cluster and converging the node, of particular interest is the chef-client.log which contains the results from converging Chef recipes.
system Internal files. We don't recommend directly using or accessing any files in this directory as they will change significantly from release to release.

HealthCheck

The HealthCheck service executes user-defined scripts to determine a VM's current viability as a cluster node. Scripts may be written in Python or be Bash (Linux) or batch (Windows) scripts. If the script exits with a code of 254, the VM will be terminated. Other non-zero return codes are treated as unknown and are logged to CycleCloud. Checks are stored in /opt/cycle/jetpack/config/healthcheck.d on Linux and in C:\cycle\jetpack\config\healthcheck.d on Windows.

To delay a HealthCheck related termination, use the jetpack keepalive command.

Jetpack Command Line Tool

The Jetpack command-line tool, located at /opt/cycle/jetpack/bin/jetpack or C:\cycle\jetpack\bin\jetpack, provides a useful set of subcommands for manipulating the current VM and interacting with Azure CycleCloud.

Command Description
jetpack config Retrieve a configuration value.
jetpack converge Execute a Chef converge.
jetpack keepalive Delay system termination by the HealthCheck Service.
jetpack log Log a message to CycleCloud cluster UI.
jetpack send Send an arbitrary AMQP message to the CycleCloud server.

Jetpack Subcommands

jetpack config

jetpack config is used to fetch information passed into a VM by Azure CycleCloud. It exposes all the system properties made available via Ohai, a subset of the VM's Azure metadata, and information about the parent CycleCloud cluster.

jetpack converge

jetpack converge downloads all CycleCloud Project (TODO: include link to CycleCloud project) Chef cookbooks and cluster-init scripts associated with the node, and starts a Chef converge process that runs all the Chef recipes and cluster-init scripts for the node.

jetpack keepalive

jetpack keepalive interacts with the HealthCheck service to delay the termination of the VM due to a failing HealthCheck. Termination can be delayed for a fixed period or indefinitely. By default, termination is delayed for one hour.

To delay system termination by one hour:

jetpack keepalive

To disable the HealthCheck service entirely, i.e. delay termination indefinitely:

jetpack keepalive forever

To delay system termination by six hours:

jetpack keepalive 6h

jetpack log

jetpack log sends a log message back to CycleCloud. The message will appear in application server log (typically /opt/cycle_server/cycle_server.log), the main event log, and the Cluster UI page.

Each message has two properties: level and priority.

The level property indicates the type of message. Valid levels are 'info', 'warn', and 'error'. The level does not indicate importance of a given message - for example, some errors are trivial and some informational messages critical.

Priority indicates the importance of the message. Valid priority values are 'low', 'medium', and 'high'. Only messages with a priority of medium or higher are displayed on the Cluster UI page to avoid flooding the page with low priority messages.

To send an informational log message that will appear on the Cluster UI page:

jetpack log 'system is now ready'

To send a low priority log message that you do not want to appear on the Cluster UI page:

jetpack log 'system is now ready' --priority low

By default, messages with a level of error have a high priority. To send an error message:

jetpack log 'the machine cannot process jobs' --level error

To send a trivial error message:

jetpack log 'the machine cannot process jobs' --level error --priority low

jetpack send

jetpack send is an advanced command that you can use to send an arbitrary AMQP message to CycleCloud. It is most useful if you have created a CycleCloud plugin that can process that information.

You can send arbitrary strings or files with specified AMQP routing keys.