Start/Stop VMs during off-hours solution (preview) in Azure Automation

The Start/Stop VMs during off-hours solution starts and stops your Azure virtual machines on user-defined schedules, provides insights through Azure Log Analytics, and sends optional emails by using SendGrid. It supports both Azure Resource Manager and classic VMs for most scenarios.

This solution provides a decentralized automation option for users who want to reduce their costs by using serverless, low-cost resources. With this solution, you can:

  • Schedule VMs to start and stop.
  • Schedule VMs to start and stop in ascending order by using Azure Tags (not supported for classic VMs).
  • Auto-stop VMs based on low CPU usage.

Prerequisites

  • The runbooks work with an Azure Run As account. The Run As account is the preferred authentication method, because it uses certificate authentication instead of a password that might expire or change frequently.

  • This solution manages only VMs that are in the same subscription as your Azure Automation account.

  • This solution is deployed only to the following Azure regions: Australia Southeast, Canada Central, Central India, East US, Japan East, Southeast Asia, UK South, and West Europe.

    Note

    The runbooks managing the VM schedule can target VMs in any region.

  • To send email notifications when the start and stop VM runbooks finish, while onboarding from Azure Marketplace, select Yes to deploy SendGrid.

    Important

    SendGrid is a third-party service. For support, contact SendGrid.

    Limitations with SendGrid are:

    • A maximum of one SendGrid account per user per subscription.
    • A maximum of two SendGrid accounts per subscription.

If you have deployed a previous version of this solution, you must first delete it from your account before deploying this release.

Solution components

This solution includes preconfigured runbooks, schedules, and integration with Log Analytics so you can tailor the startup and shutdown of your virtual machines to suit your business needs.

Runbooks

The following table lists the runbooks deployed to your Automation account. You should not make changes to the runbook code. Instead, write your own runbook for new functionality.

Important

Do not directly run any runbook with “child” appended to its name.

All parent runbooks include the WhatIf parameter. When set to True, WhatIf supports detailing the exact behavior the runbook takes when run without the WhatIf parameter and validates the correct VMs are being targeted. A runbook only performs its defined actions when the WhatIf parameter is set to False.

Runbook Parameters Description
AutoStop_CreateAlert_Child VMObject
AlertAction
WebHookURI
Called only from the parent runbook. Creates alerts on a per-resource basis for AutoStop scenario.
AutoStop_CreateAlert_Parent WhatIf: True or False
VMList
Creates or updates Azure alert rules on VMs in the targeted subscription or resource groups.
VMList: Comma-separated list of VMs. For example, vm1,vm2,vm3.
AutoStop_Disable none Disables AutoStop alerts and default schedule.
AutoStop_StopVM_Child WebHookData Called only from the parent runbook. Alert rules call this runbook to stop the VM.
Bootstrap_Main none Used one time to set up bootstrap configurations such as webhookURI, which are typically not accessible from Azure Resource Manager. This runbook is removed automatically upon successful deployment.
ScheduledStartStop_Child VMName
Action: Stop or Start
ResourceGroupName
Called only from the parent runbook. Executes stop or start for the scheduled stop.
ScheduledStartStop_Parent Action: Stop or Start
WhatIf: True or False
This affects all VMs in the subscription. Edit the External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames to only execute on these target resource groups. You can also exclude specific VMs by updating the External_ExcludeVMNames variable. WhatIf behaves the same as in other runbooks.
SequencedStartStop_Parent Action: Stop or Start
WhatIf: True or False
Create tags named SequenceStart and SequenceStop on each VM for which you want to sequence start/stop activity. The value of the tag should be a positive integer (1, 2, 3) that corresponds to the order in which you want to start or stop. WhatIf behaves the same as in other runbooks.
Note: VMs must be within resource groups defined as External_Start_ResourceGroupNames, External_Stop_ResourceGroupNames, and External_ExcludeVMNames in Azure Automation variables. They must have the appropriate tags for actions to take effect.

Variables

The following table lists the variables created in your Automation account. You should only modify variables prefixed with External. Modifying variables prefixed with Internal causes undesirable effects.

Variable Description
External_AutoStop_Condition The conditional operator required for configuring the condition before triggering an alert. Acceptable values are GreaterThan, GreaterThanOrEqual, LessThan, and LessThanOrEqual.
External_AutoStop_Description The alert to stop the VM if the CPU percentage exceeds the threshold.
External_AutoStop_MetricName The name of the performance metric for which the Azure Alert rule is to be configured.
External_AutoStop_Threshold The threshold for the Azure Alert rule specified in the variable External_AutoStop_MetricName. Percentage values can range from 1 to 100.
External_AutoStop_TimeAggregationOperator The time aggregation operator, which is applied to the selected window size to evaluate the condition. Acceptable values are Average, Minimum, Maximum, Total, and Last.
External_AutoStop_TimeWindow The window size during which Azure analyzes selected metrics for triggering an alert. This parameter accepts input in timespan format. Possible values are from 5 minutes to 6 hours.
External_EmailFromAddress Specifies the sender of the email.
External_EmailSubject Specifies the text for the subject line of the email.
External_EmailToAddress Specifies the recipients of the email. Separate names by using a comma.
External_ExcludeVMNames Enter VM names to be excluded, separating names by using a comma with no spaces.
External_IsSendEmail Specifies the option to send email notification upon completion. Specify Yes or No to not send email. This option should be No if you did not enable email notifications during the initial deployment.
External_Start_ResourceGroupNames Specifies one or more resource groups, separating values by using a comma, targeted for start actions.
External_Stop_ResourceGroupNames Specifies one or more resource groups, separating values by using a comma, targeted for stop actions.
Internal_AutomationAccountName Specifies the name of the Automation account.
Internal_AutoSnooze_WebhookUri Specifies Webhook URI called for the AutoStop scenario.
Internal_AzureSubscriptionId Specifies the Azure Subscription ID.
Internal_ResourceGroupName Specifies the Automation account resource group name.
Internal_SendGridAccountName Specifies the SendGrid account name.
Internal_SendGridPassword Specifies the SendGrid account password.


Across all scenarios, the External_Start_ResourceGroupNames, External_Stop_ResourceGroupNames, and External_ExcludeVMNames variables are necessary for targeting VMs, with the exception of providing a comma-separated list of VMs for the AutoStop_CreateAlert_Parent runbook. That is, your VMs must reside in target resource groups for start and stop actions to occur. The logic works a bit like Azure policy, in that you can target the subscription or resource group and have actions inherited by newly created VMs. This approach avoids having to maintain a separate schedule for every VM and manage starts and stops in scale.

Schedules

The following table lists each of the default schedules created in your Automation account. You can modify them or create your own custom schedules. By default, each of these are disabled except for Scheduled_StartVM and Scheduled_StopVM.

You should not enable all schedules, because this might create overlapping schedule actions. It's best to determine which optimizations you want to perform and modify accordingly. See the example scenarios in the overview section for further explanation.

Schedule name Frequency Description
Schedule_AutoStop_CreateAlert_Parent Every 8 hours Runs the AutoStop_CreateAlert_Parent runbook every 8 hours, which in turn stops the VM-based values in External_Start_ResourceGroupNames, External_Stop_ResourceGroupNames, and External_ExcludeVMNames in Azure Automation variables. Alternatively, you can specify a comma-separated list of VMs by using the VMList parameter.
Scheduled_StopVM User defined, daily Runs the Scheduled_Parent runbook with a parameter of Stop every day at the specified time. Automatically stops all VMs that meet the rules defined by asset variables. You should enable the related schedule, Scheduled-StartVM.
Scheduled_StartVM User defined, daily Runs the Scheduled_Parent runbook with a parameter of Start every day at the specified time. Automatically starts all VMs that meet the rules defined by the appropriate variables. You should enable the related schedule, Scheduled-StopVM.
Sequenced-StopVM 1:00 AM (UTC), every Friday Runs the Sequenced_Parent runbook with a parameter of Stop every Friday at the specified time. Sequentially (ascending) stops all VMs with a tag of SequenceStop defined by the appropriate variables. Refer to the Runbooks section for more details on tag values and asset variables. You should enable the related schedule, Sequenced-StartVM.
Sequenced-StartVM 1:00 PM (UTC), every Monday Runs the Sequenced_Parent runbook with a parameter of Start every Monday at the specified time. Sequentially (descending) starts all VMs with a tag of SequenceStart defined by the appropriate variables. Refer to the Runbooks section for more details on tag values and asset variables. You should enable the related schedule, Sequenced-StopVM.


Configuration

Perform the following steps to add the Start/Stop VMs during off-hours solution to your Automation account, and then configure the variables to customize the solution.

  1. In the Azure portal, click Create a resource.
    Azure portal
  2. In the Marketplace pane, type a keyword such as Start or Start/Stop. As you begin typing, the list filters based on your input. Alternatively, you can type in one or more keywords from the full name of the solution and then press Enter. Select Start/Stop VMs during off-hours [Preview] from the search results.
  3. In the Start/Stop VMs during off-hours [Preview] pane for the selected solution, review the summary information and then click Create.
  4. The Add Solution pane appears. You are prompted to configure the solution before you can import it into your Automation subscription.

    VM Management Add Solution blade

  5. On the Add Solution pane, select Workspace. Select an OMS workspace that's linked to the same Azure subscription that the Automation account is in. If you don't have a workspace, select Create New Workspace. On the OMS Workspace pane, perform the following:

    • Specify a name for the new OMS Workspace.
    • Select a Subscription to link to by selecting from the drop-down list, if the default selected is not appropriate.
    • For Resource Group, you can create a new resource group or select an existing one.
    • Select a Location. Currently, the only locations available are Australia Southeast, Canada Central, Central India, East US, Japan East, Southeast Asia, UK South, and West Europe.
    • Select a Pricing tier. The solution offers two tiers: free and OMS paid. The free tier has a limit on the amount of data collected daily, the retention period, and the runbook job runtime minutes. The OMS paid tier does not have a limit on the amount of data collected daily.

      Note

      Although the Per GB (Standalone) paid tier is displayed as an option, it is not applicable. If you select it and proceed with the creation of this solution in your subscription, it fails. This will be addressed when this solution is officially released.
      This solution only uses automation job minutes and log ingestion. It does not add additional OMS nodes to your environment.

  6. After providing the required information on the OMS workspace pane, click Create. You can track its progress under Notifications from the menu, which returns you to the Add Solution pane when done.

  7. On the Add Solution pane, select Automation account. If you're creating a new OMS workspace, you need to also create a new Automation account to be associated with it. Select Create an Automation account, and on the Add Automation account pane, provide the following:

    • In the Name field, enter the name of the Automation account.

      All other options are automatically populated based on the OMS workspace selected. These options cannot be modified. An Azure Run As account is the default authentication method for the runbooks included in this solution. After you click OK, the configuration options are validated and the Automation account is created. You can track its progress under Notifications from the menu.

      Otherwise, you can select an existing Automation Run As account. Note that the account you select cannot already be linked to another OMS workspace. If it is already linked, you receive a message and need to select a different Automation Run As account or create a new one.

      Automation Account Already Linked to OMS Workspace

  8. Finally, on the Add Solution pane, select Configuration. The Parameters pane appears.

    Parameters pane for solution

    Here, you're prompted to:

    • Specify the Target ResourceGroup Names. These are resource group names that contain VMs to be managed by this solution. You can enter more than one name and separate each by using a comma (values are not case sensitive). Using a wildcard is supported if you want to target VMs in all resource groups in the subscription.
    • Specify the VM Exclude List (string). This is the name of one or more virtual machines from the target resource group. You can enter more than one name and separate each by using a comma (values are not case sensitive). Using a wildcard is supported.
    • Select a Schedule. This is a recurring date and time for starting and stopping the VMs in the target resource groups. By default, the schedule is configured to the UTC time zone. Selecting a different region is not available. To configure the schedule to your specific time zone after configuring the solution, see Modifying the startup and shutdown schedule.
    • To receive Email notifications from SendGrid, accept the default value of Yes and provide a valid email address. If you select No but decide at a later date that you want to receive email notifications, you can update the External_EmailToAddress variable with valid email addresses separated by a comma, and then modify the variable External_IsSendEmail with the value Yes.
  9. After you have configured the initial settings required for the solution, select Create. After all settings are validated, the solution is deployed to your subscription. This process can take several seconds to finish, and you can track its progress under Notifications from the menu.

Collection frequency

Automation job log and job stream data is ingested into the Log Analytics repository every 5 minutes.

Using the solution

When you add the VM management solution, the StartStopVMView tile is added to your dashboard in your Log Analytics workspace from the Azure portal. This tile displays a count and a graphical representation of the runbook jobs for the solution that have started and finished successfully.

VM Management StartStopVM View Tile

In your Automation account, you can access and manage the solution by selecting the Workspace option. On the Log Analytics page, select Solutions from the left pane. On the Solutions page, select the solution Start-Stop-VM[workspace] from the list.

Solutions list in Log Analytics

Selecting the solution displays the Start-Stop-VM[workspace] solution pane. Here you can review important details such as the StartStopVM tile. As in your Log Analytics workspace, this tile displays a count and a graphical representation of the runbook jobs for the solution that have started and have finished successfully.

Automation Update Management solution page

From here, you can perform further analysis of the job records by clicking the donut tile. The solution dashboard shows job history and pre-defined log search queries. Switch to the Log Analytics Advanced portal to search based on your search queries.

All parent runbooks include the WhatIf parameter. When set to True, this supports detailing the exact behavior the runbook takes when run without the WhatIf parameter, and it validates that the correct VMs are being targeted. Runbooks only perform their defined actions when the WhatIf parameter is set to False.

Scenario 1: Daily start/stop VMs across a subscription or target resource groups

This is the default configuration when you first deploy the solution. For example, you can configure it to stop all VMs across a subscription when you leave work in the evening, and start them in the morning when you are back in the office. When you configure the schedules Scheduled-StartVM and Scheduled-StopVM during deployment, they start and stop targeted VMs.

Note

The time zone is your current time zone when you configure the schedule time parameter. However, it is stored in UTC format in Azure Automation. You do not have to do any time zone conversion as this is handled during the deployment.

You control which VMs are in scope by configuring the following variables: External_Start_ResourceGroupNames, External_Stop_ResourceGroupNames, and External_ExcludeVMNames.

Note

The value for Target ResourceGroup Names is stored as the value for both External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames. For further granularity, you can modify each of these variables to target different resource groups. For start action, use External_Start_ResourceGroupNames, and for stop action, use External_Stop_ResourceGroupNames. VMs are automatically added to the start and stop schedules.

To test and validate your configuration, manually start the ScheduledStartStop_Parent runbook and set the ACTION parameter to start or stop and the WHATIF parameter to True.

Configure parameters for runbook

Preview the scheduled action and make any necessary changes before implementing against production VMs. You can manually execute the runbook with the parameter set to False, or you can let the Automation schedule Schedule-StartVM and Schedule-StopVM run automatically following your prescribed schedule.

Scenario 2: Sequence the start/stop VMs across a subscription by using tags

In an environment that includes two or more components on multiple VMs supporting a distributed workload, supporting the sequence in which components are started and stopped in order is important. You can accomplish this by performing the following steps:

  1. Add a SequenceStart and a SequenceStop tag with a positive integer value to VMs that are targeted in External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames variables. The start and stop actions are performed in ascending order. To learn how to tag a VM, see Tag a Windows Virtual Machine in Azure and Tag a Linux Virtual Machine in Azure.
  2. Modify the schedules Sequenced-StartVM and Sequenced-StopVM to the date and time that meet your requirements and enable the schedule.

To test and validate your configuration, manually start the SequencedStartStop_Parent runbook. Set the ACTION parameter to start or stop and the WHATIF parameter to True.

Configure parameters for runbook
Preview the action and make any necessary changes before implementing against production VMs. When ready, manually execute the runbook with the parameter set to False, or let the Automation schedule Sequenced-StartVM and Sequenced-StopVM run automatically following your prescribed schedule.

Scenario 3: Automate start/stop VMs across a subscription based on CPU utilization

This solution can help manage the cost of running virtual machines in your subscription by evaluating Azure VMs that aren't used during non-peak periods, such as after hours, and automatically shutting them down if processor utilization is less than x%.

By default, the solution is pre-configured to evaluate the percentage CPU metric to see if average utilization is 5 percent or less. This is controlled by the following variables and can be modified if the default values do not meet your requirements:

  • External_AutoStop_MetricName
  • External_AutoStop_Threshold
  • External_AutoStop_TimeAggregationOperator
  • External_AutoStop_TimeWindow

You can enable either targeting the action against a subscription and resource group, or targeting a specific list of VMs, but not both.

Target the stop action against a subscription and resource group

  1. Configure the External_Stop_ResourceGroupNames and External_ExcludeVMNames variables to specify the target VMs.
  2. Enable and update the Schedule_AutoStop_CreateAlert_Parent schedule.
  3. Run the AutoStop_CreateAlert_Parent runbook with the ACTION parameter set to start and the WHATIF parameter set to True to preview your changes.

Target the stop action by VM list

  1. Run the AutoStop_CreateAlert_Parent runbook with the ACTION parameter set to start, add a comma-separated list of VMs in the VMList parameter, and then set the WHATIF parameter to True. Preview your changes.
  2. Configure the External_ExcludeVMNames parameter with a comma-separated list of VMs (VM1,VM2,VM3).
  3. This scenario does not honor the External_Start_ResourceGroupNames and External_Stop_ResourceGroupnames variables. For this scenario, you need to create your own Automation schedule. For details, see Scheduling a runbook in Azure Automation.

Now that you have a schedule for stopping VMs based on CPU utilization, you need to enable one of the following schedules to start them.

  • Target start action by subscription and resource group. See the steps in Scenario 1 for testing and enabling Scheduled-StartVM schedules.
  • Target start action by subscription, resource group, and tag. See the steps in Scenario 2 for testing and enabling Sequenced-StartVM schedules.

Configuring email notifications

To configure email notifications after the solution is deployed, modify the following three variables:

  • External_EmailFromAddress: Specify the sender's email address.
  • External_EmailToAddress: Specify a comma-separated list of email addresses (user@hotmail.com, user@outlook.com) to receive notification emails.
  • External_IsSendEmail: Set to Yes to receive emails. To disable email notifications, set value to No.

Modifying the startup and shutdown schedules

Managing the startup and shutdown schedules in this solution follows the same steps as outlined in Scheduling a runbook in Azure Automation.

Log Analytics records

Automation creates two types of records in the OMS repository: job logs and job streams.

Job logs

Property Description
Caller Who initiated the operation. Possible values are either an email address or system for scheduled jobs.
Category Classification of the type of data. For Automation, the value is JobLogs.
CorrelationId GUID that is the Correlation Id of the runbook job.
JobId GUID that is the Id of the runbook job.
operationName Specifies the type of operation performed in Azure. For Automation, the value is Job.
resourceId Specifies the resource type in Azure. For Automation, the value is the Automation account associated with the runbook.
ResourceGroup Specifies the resource group name of the runbook job.
ResourceProvider Specifies the Azure service that supplies the resources you can deploy and manage. For Automation, the value is Azure Automation.
ResourceType Specifies the resource type in Azure. For Automation, the value is the Automation account associated with the runbook.
resultType The status of the runbook job. Possible values are:
- Started
- Stopped
- Suspended
- Failed
- Succeeded
resultDescription Describes the runbook job result state. Possible values are:
- Job is started
- Job Failed
- Job Completed
RunbookName Specifies the name of the runbook.
SourceSystem Specifies the source system for the data submitted. For Automation, the value is OpsManager
StreamType Specifies the type of event. Possible values are:
- Verbose
- Output
- Error
- Warning
SubscriptionId Specifies the subscription ID of the job.
Time Date and time when the runbook job executed.

Job streams

Property Description
Caller Who initiated the operation. Possible values are either an email address or system for scheduled jobs.
Category Classification of the type of data. For Automation, the value is JobStreams.
JobId GUID that is the Id of the runbook job.
operationName Specifies the type of operation performed in Azure. For Automation, the value is Job.
ResourceGroup Specifies the resource group name of the runbook job.
resourceId Specifies the resource Id in Azure. For Automation, the value is the Automation account associated with the runbook.
ResourceProvider Specifies the Azure service that supplies the resources you can deploy and manage. For Automation, the value is Azure Automation.
ResourceType Specifies the resource type in Azure. For Automation, the value is the Automation account associated with the runbook.
resultType The result of the runbook job at the time the event was generated. A possible value is:
- InProgress
resultDescription Includes the output stream from the runbook.
RunbookName The name of the runbook.
SourceSystem Specifies the source system for the data submitted. For Automation, the value is OpsManager.
StreamType The type of job stream. Possible values are:
- Progress
- Output
- Warning
- Error
- Debug
- Verbose
Time Date and time when the runbook job executed.

When you perform any log search that returns category records of JobLogs or JobStreams, you can select the JobLogs or JobStreams view, which displays a set of tiles summarizing the updates returned by the search.

Sample log searches

The following table provides sample log searches for job records collected by this solution.

Query Description
Find jobs for runbook ScheduledStartStop_Parent that have finished successfully search Category == "JobLogs" | where ( RunbookName_s == "ScheduledStartStop_Parent" ) | where ( ResultType == "Completed" ) | summarize AggregatedValue = count() by ResultType, bin(TimeGenerated, 1h) | sort by TimeGenerated desc
Find jobs for runbook SequencedStartStop_Parent that have finished successfully search Category == "JobLogs" | where ( RunbookName_s == "SequencedStartStop_Parent" ) | where ( ResultType == "Completed" ) | summarize AggregatedValue = count() by ResultType, bin(TimeGenerated, 1h) | sort by TimeGenerated desc

Removing the solution

If you decide you no longer need to use the solution, you can delete it from the Automation account. Deleting the solution only removes the runbooks. It does not delete the schedules or variables that were created when the solution was added. Those assets you need to delete manually if you are not using them with other runbooks.

To delete the solution, perform the following steps:

  1. From your Automation account, select Workspace from the left pane.
  2. On the Solutions page, select the solution Start-Stop-VM[Workspace]. On the VMManagementSolution[Workspace] page, from the menu, select Delete.

    Delete VM Mgmt Solution
  3. In the Delete Solution window, confirm that you want to delete the solution.
  4. While the information is verified and the solution is deleted, you can track its progress under Notifications from the menu. You are returned to the Solutions page after the process to remove the solution starts.

The Automation account and Log Analytics workspace are not deleted as part of this process. If you do not want to retain the Log Analytics workspace, you need to manually delete it. This can be accomplished from the Azure portal:

  1. From the Azure portal home screen, select Log Analytics.
  2. On the Log Analytics pane, select the workspace.
  3. Select Delete from the menu on the workspace settings pane.

Next steps