Jetpack

  • Article

Jetpack is required on every node of a cluster. It is automatically installed by Azure CycleCloud on each virtual machine that is provisioned to become a node in a cluster. Jetpack provides three main functions:

  • Node Configuration -- CycleCloud uses scripts and Chef to automate the configuration of a provisioned VM into a working cluster node. A Chef client as well as the necessary resources for the configuration of the VM are embedded within Jetpack.
  • Distributed Synchronization -- Jetpack manages communication between the node and the CycleCloud application server. This enables CycleCloud to monitor the status of the provisioning VMs and synchronize the orchestration of multiple nodes in the cluster.
  • HealthCheck -- Jetpack uses HealthCheck to determine the health of VMs so that unhealthy VMs can be terminated.

Jetpack Installation

The Jetpack installer is cached in your Azure Storage Account when you first start a cluster using CycleCloud. As cluster VMs are provisioned, a custom script extension is executed as part of the boot process which downloads the Jetpack installer from your Azure Storage cache and then installs it on the VM.

The Jetpack installer:

  • Unpacks the Jetpack files to a single directory tree:
    • Windows: C:\cycle\jetpack
    • Linux: /opt/cycle/Jetpack
  • Creates system init startup scripts which configure a VM as a cluster node
  • Installs the HealthCheck service
  • Installs the Jetpack Command Line Tool to:
    • Windows: C:\cycle\jetpack\bin\jetpack
    • Linux: /opt/cycle/jetpack/bin/jetpack
  • Creates udev rules on Linux
  • Sets the environment variable CYCLECLOUD_HOME

Note

If Jetpack has been pre-installed on the image, the custom script extension will not re-install Jetpack. Instead, an initialization step will be run which will validate the node's connection to CycleCloud and start the healthcheck and jetpackd services before continuing to configure the node.

Jetpack Subdirectories

Directory Description
bin Useful binaries and scripts.
config User defined and cluster defined configuration files and scripts.
logs Logs generated by joining a cluster and converging the node, of particular interest is the chef-client.log which contains the results from converging Chef recipes.
run Runtime files generated by the system. We don't recommend directly accessing these files.
system Internal files. We don't recommend directly using any files in this directory as they can significantly change from release to release.

HealthCheck

The HealthCheck service executes user-defined scripts to determine a VM's current viability as a cluster node. Please see the HealthCheck documentation for more information.

Jetpack Command Line Tool

The Jetpack command-line tool provides a useful set of subcommands for manipulating the current VM and interacting with Azure CycleCloud.

Command Description
jetpack autoscale Autoscale the cluster this node belongs to.
jetpack config Retrieve a configuration value.
jetpack converge Execute a Chef converge.
jetpack download Download a blob resource from a project in Azure Storage.
jetpack keepalive Delay system termination by the HealthCheck Service.
jetpack log Log a message to CycleCloud cluster UI.
jetpack run_on_shutdown Add a script to be called prior to node termination.
jetpack send Send an arbitrary AMQP message to the CycleCloud server.
jetpack shutdown Request the shutdown of the VM by CycleCloud.
jetpack test Run tests associated with projects assigned to the VM.
jetpack users List users that CycleCloud will manage on this VM.
jetpack report_issue Archives log files from VM to Azure Storage

jetpack autoscale

jetpack autoscale sets the autoscaling targets for the cluster that the node belongs to. Clusters can be scaled by cores, instance count or custom definitions.

To scale to 100 cores:

jetpack autoscale --corecount=100

To scale the 'gpu' nodearray to 5 nodes:

jetpack autoscale --instancecount 5 --name=gpu

To customize autoscale, a json file must be written to disk containing the nodearray definition you would like to scale. To scale by 100 cores:

[
  {
      "Name": "execute",
      "TargetCoreCount": 100
  }
]

jetpack autoscale --file=custom-autoscale.json

jetpack config

jetpack config fetches information passed into a VM by CycleCloud. It exposes:

  • all the system properties made available via Ohai
  • a subset of the VM's Azure metadata
  • information about the parent CycleCloud cluster.

jetpack converge

jetpack converge downloads all CycleCloud projects associated with the node, and starts a Chef converge process that runs all the Chef recipes and cluster-init scripts for the node.

jetpack download

jetpack download downloads a blob that was uploaded with a project to the node. You must specify the project the blob belongs to.

To download the blob big-file.zip that was uploaded as part of the example-project project to the current directory:

jetpack download --project example-project big-file.zip .

jetpack keepalive

jetpack keepalive interacts with the HealthCheck service to delay the termination of the VM due to a failing HealthCheck. Termination can be delayed for a fixed period or indefinitely. By default, termination is delayed for one hour.

To delay system termination by one hour:

jetpack keepalive

To delay system termination by six hours:

jetpack keepalive 6h

To disable the HealthCheck service entirely, i.e. delay termination indefinitely:

jetpack keepalive forever

Note

Only the forever option is available for HealthCheck on Windows VMs

jetpack log

jetpack log sends a log message back to CycleCloud. The message will appear in the application server log (typically /opt/cycle_server/cycle_server.log), the main event log, and the Cluster UI page.

Each message has two properties: level and priority.

The level property indicates the type of message. Valid levels are 'info', 'warn', and 'error'. The level does not indicate importance of a given message - for example, some errors are trivial and some informational messages critical.

Priority indicates the importance of the message. Valid priority values are 'low', 'medium', and 'high'. Only messages with a priority of medium or higher are displayed on the Cluster UI page to avoid flooding the page with low priority messages.

To send an informational log message that will appear on the Cluster UI page:

jetpack log 'system is now ready'

To send a low priority log message that you do not want to appear on the Cluster UI page:

jetpack log 'system is now ready' --priority low

By default, messages with a level of error have a high priority. To send an error message:

jetpack log 'the machine cannot process jobs' --level error

To send a trivial error message:

jetpack log 'the machine cannot process jobs' --level error --priority low

jetpack run_on_shutdown

jetpack run_on_shutdown registers a bash script to be called prior to node termination.

The command takes the absolute path to the script as an argument.

When the node is terminated by Azure, if Termination Notifications are enabled, Jetpack will be notified of the termination and attempt to run the script prior to the node shutting down.

Nodes must enable Termination Notifications to enable run_on_shutdown.

jetpack run_on_shutdown /tmp/example.sh

This command is not supported for Windows nodes.

jetpack send

jetpack send sends an AMQP message to CycleCloud. It is an advanced command that is not recommended unless you are developing plugins for CycleCloud.

You can send arbitrary strings or files with specified AMQP routing keys.

jetpack shutdown

jetpack shutdown requests that CycleCloud terminate the node. Options can be passed to the command to specify the reason for the shutdown request (idle vs unhealthy) as well as how to terminate the node (terminate vs deallocate).

To shutdown an unhealthy node:

jetpack shutdown --unhealthy

To deallocate the node:

jetpack shutdown --deallocate

jetpack test

jetpack test runs any tests that are included with projects assigned to the node and prints the results to stdout.

jetpack users

jetpack users lists the users that CycleCloud will manage on the node. This list can change over time as users are assigned and removed to the cluster.

To get a user-friendly print out of the users assigned to the node:

$ jetpack users

Username: test-user
Full Name: Test User
UID: 10201
Is Admin: True
Is Owner: True

To get script-friendly JSON output:

$ jetpack users --json

[
    {
        "fullName": "Test User",
        "isAdmin": true,
        "isOwner": true,
        "name": "test-user",
        "publicKeys": [
            "ssh-rsa public-key-goes-here\n"
        ],
        "uid": 10201
    }
]

jetpack report_issue

jetpack report_issue archives log directories from the VM, optionally uploading them to Azure Storage and creating a signed URL for external access. Logs will be uploaded to the Azure Storage account referenced by the node's Locker. When signing an archive in Azure Storage the resulting SAS token will have read only access for 30 days.

Usage:

$ jetpack report_issue [LOG_PATH] [--upload/--no-upload] [--sign/--no-sign]

To archive, upload and sign the default Jetpack logs ($JETPACK_HOME/logs):

$ jetpack report_issue
Logs can be found at: https://testaccount.blob.core.windows.net/cyclecloud/issues/TestCluster-execute-1-77777964-8b74-420d-ad44-094edf7695f2.zip?sv=2017-11-09&rsct=binary&sig=jBJUlYo10lRq0eW94I%2B6syzYVmgo1qcTFUc35D/q0Tg%3D&se=2020-12-04T15%3A15%3A00Z&spr=https&rscd=disposition%3Dfile%3B%20attachment&sp=r&sr=b
Signed URL will expire on: 2020-12-04T15:15:00Z

To archive, upload but not sign a non-default log directory:

$ jetpack report_issue /var/log/azure --no-sign
Logs can be found at: https://testaccount.blob.core.windows.net/cyclecloud/issues/TestCluster-execute-1-d67fe991-1dac-4644-9af7-50c835726f5e.zip

To simply archive logs on the local VM:

$ jetpack report_issue --no-upload
Logs can be found at: /tmp/tmp4nscw705/TestCluster-execute-1-4249e973-3d87-4b14-94ed-6856a5267972.zip