HTCondor
HTCondor can easily be enabled on a CycleCloud cluster by modifying the "run_list" in the configuration section of your cluster definition. There are three basic components of an HTCondor cluster. The first is the "central manager" which provides the scheduling and management daemons. The second component of an HTCondor cluster is one or more schedulers from which jobs are submitted into the system. The final component is one or more execute nodes which are the hosts perform the computation. A simple HTCondor template may look like:
[cluster htcondor]
[[node manager]]
ImageName = cycle.image.centos7
MachineType = Standard_A4 # 8 cores
[[[configuration]]]
run_list = role[central_manager]
[[node scheduler]]
ImageName = cycle.image.centos7
MachineType = Standard_A4 # 8 cores
[[[configuration]]]
run_list = role[condor_scheduler_role],role[filer_role],role[scheduler]
[[nodearray execute]]
ImageName = cycle.image.centos7
MachineType = Standard_A1 # 1 core
Count = 1
[[[configuration]]]
run_list = role[usc_execute]
Importing and starting a cluster with definition in CycleCloud will yield a "manager" and a "scheduler" node, as well as one "execute" node. Execute nodes can be added to the cluster via the cyclecloud add_node command. To add 10 more execute nodes:
cyclecloud add_node htcondor -t execute -c 10
HTCondor Autoscaling
CycleCloud supports autoscaling for HTCondor, which means that the software will monitor the status of your queue and turn on and off nodes as needed to complete the work in an optimal amount of time/cost. You can enable autoscaling for HTCondor by adding Autoscale=true to your cluster definition:
[cluster htcondor]
Autoscale = True
HTCondor Advanced Usage
If you know the average runtime of jobs, you can define average_runtime (in minutes) in your job. CycleCloud will use that to start the minimum number of nodes (for example, five 10-minute jobs will only start a single node instead of five when average_runtime is set to 10).
Autoscale Nodearray
By default, HTCondor will request cores from the nodearray called 'execute'. If a job requires a different nodearray (for example if certain jobs within a workflow have a high memory requirement), you can specify a slot_type attribute for the job. For example, adding +slot_type = "highmemory" will cause HTCondor to request a node from the "highmemory" nodearray instead of "execute" (note that this currently requires htcondor.slot_type = "highmemory" to be set in the nodearray's [[[configuration]]] section). This will not affect how HTCondor schedules the jobs, so you may want to include the slot_type startd attribute in the job's requirements or rank expressions. For example: Requirements = target.slot_type = "highmemory".
Submitting Jobs to HTCondor
The most generic way to submit jobs to an HTCondor scheduler is the command (run from a scheduler node):
condor_submit my_job.submit
A sample submit file might look like this:
Universe = vanilla
Executable = do_science
Arguments = -v --win-prize=true
Output = log/$(Cluster).$(Process).out
Error = log/$(Cluster).$(Process).err
Should_transfer_files = if_needed
When_to_transfer_output = On_exit
+average_runtime = 10
+slot_type = "highmemory"
Queue
HTCondor Configuration Reference
The following are the HTCondor-specific configuration options you can set to customize functionality:
| HTCondor-Specific Configuration Options | Description |
|---|---|
| htcondor.agent_enabled | If true, use the condor_agent for job submission and polling. Default: false |
| htcondor.agent_version | The version of the condor_agent to use. Default: 1.27 |
| htcondor.classad_lifetime | The default lifetime of classads (in seconds). Default: 700 |
| htcondor.condor_owner | The Linux account that owns the HTCondor scaledown scripts. Default: root |
| htcondor.condor_group | The Linux group that owns the HTCondor scaledown scripts. Default: root |
| htcondor.data_dir | The directory for logs, spool directories, execute directories, and local config file. Default: /mnt/condor_data (Linux), C:\All Services\condor_local (Windows) |
| htcondor.ignore_hyperthreads | (Windows only) Set the number of CPUs to be half of the detected CPUs as a way to "disable" hyperthreading. If using autoscale, specify the non-hyperthread core count with the Cores configuration setting in the [[node]] or [[nodearray]] section. Default: false |
| htcondor.install_dir | The directory that HTCondor is installed to. Default: /opt/condor (Linux), C:\condor (Windows) |
| htcondor.job_start_count | The number of jobs a schedd will start per cycle. 0 is unlimited. Default: 20 |
| htcondor.job_start_delay | The number of seconds between each job start interval. 0 is immediate. Default: 1 |
| htcondor.max_history_log | The maximum size of the job history file in bytes. Default: 20971520 |
| htcondor.max_history_rotations | The maximum number of job history files to keep. Default: 20 |
| htcondor.negotiator_cycle_delay | The minimum number of seconds before a new negotiator cycle may start. Default: 20 |
| htcondor.negotiator_interval | How often (in seconds) the condor_negotiator starts a negotiation cycle. Default: 60 |
| htcondor.negotiator_inform_startd | If true, the negotiator informs the startd when it is matched to a job. Default: true |
| htcondor.remove_stopped_nodes | If true, stopped execute nodes are removed from the CycleServer view instead of being marked as "down". |
| htcondor.running | If true, HTCondor collector and negotiator daemons run on the central manager. Otherwise, only the condor_master runs. Default: true |
| htcondor.scheduler_dual | If true, schedulers run two schedds. Default: true |
| htcondor.single_slot | If true, treats the machine as a single slot (regardless of the number of cores the machine possesses). Default: false |
| htcondor.slot_type | Defines the slot_type of a node array for autoscaling. Default: execute |
| htcondor.update_interval | The interval (in seconds) for the startd to publish an update to the collector. Default: 240 |
| htcondor.use_cache_config | If true, use cache_config to have the instance poll CycleServer for configuration. Default: false |
| htcondor.version | The version of HTCondor to install. Default: 8.2.6 |
HTCondor Auto-Generated Configuration File
HTCondor has large number of configuration settings, including user-defined attributes. CycleCloud offers the ability to create a custom configuration file using attributes defined in the cluster:
| Attribute | Description |
|---|---|
| htcondor.custom_config.enabled | If true, a configuration file is generated using the specified attributes. Default: false |
| htcondor.custom_config.file_name | The name of the file (placed in htcondor.data_dir/config) to write. Default: ZZZ-custom_config.txt |
| htcondor.custom_config.settings | The attributes to write to the custom config file such as htcondor.custom_config.settings.max_jobs_running = 5000 |
Note
HTCondor configuration attributes containing a . cannot be specified using this method. If such attributes are needed, they should be specified in a cookbook or a file installed with cluster-init.
CycleCloud supports a standard set of autostop attributes across schedulers:
| Attribute | Description |
|---|---|
| cyclecloud.cluster.autoscale.stop_enabled | Is autostop enabled on this node? [true/false] |
| cyclecloud.cluster.autoscale.idle_time_after_jobs | The amount of time (in seconds) for a node to sit idle after completing jobs before it is scaled down. |
| cyclecloud.cluster.autoscale.idle_time_before_jobs | The amount of time (in seconds) for a node to sit idle before completing jobs before it is scaled down. |