question

EuanShiell-6018 avatar image
0 Votes"
EuanShiell-6018 asked SamaraSoucy-MSFT answered

Collecting metrics for deleted instances

Greetings all,

I am trying to collect metrics for vm instances that appear in the insights activity logs.

The instance ids for the vms are found by searching for Compute/virtualMachines write, deallocate/action delete and start/action events using
$"subscriptions/{subscriptionId}/providers/Microsoft.Insights/eventtypes/management/values?api-version=2015-04-01&$filter={filter}";


A list of metric definitions is then requested using
$"https://management.azure.com{vm.Value.ResourceId}/providers/microsoft.insights/metricDefinitions".

Finally, aggregated metric values are requested from the returned metric definitions using
$"https://management.azure.com{vm.Value.ResourceId}/providers/microsoft.insights/metrics?timespan={queryTimespan}&interval=PT5M&metricnames={metricDefinition.Name.Value}&aggregation=Average&api-version=2018-01-01&metricnamespace={metricDefinition.Namespace}"

Any vms that only appear in the insights activity logs return "NotFound".
Vms that are found by calling ComputeManagementClient.VirtualMachines.ListAllAsync() from the .net API return aggregated metrics when using this approach.

The Azure Monitor Metrics overview indicates that metrics are retained for 93 days, and I'm searching for virtual machines that are often created and deleted for an hour or so within the previous 24 hours.

Can anyone offer any suggestions?
Many thanks in advance.


azure-virtual-machinesazure-monitor
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SamaraSoucy-MSFT avatar image
0 Votes"
SamaraSoucy-MSFT answered

The error is due to the VM being deleted. In order for the metrics API link to work the subscription, resource group, and VM must all exist. The .NET SDK is using a different endpoint https://management.azure.com/subscriptions/{subscriptionID}/providers/Microsoft.Compute/virtualMachines?api-version=2020-12-01 which will only return current VMs, not deleted ones, whereas the activity logs will include deleted VMs. If you recreate the VM with the same name the metrics will still be there, as the resource ID is used as a key, but it isn't currently possible to access those metrics when the resource is deleted.

An alternative may be to use the Diagnostic Settings to stream the metrics into a Log Analytics workspace. There is a cost associated with this, but it would let you keep access to metrics on deleted resources. Since these VMs are being created and deleted often, then manually updating the settings each time may not be feasible. There are a couple ways to address this. The first is to add the setting to your deployment scripts or ARM templates. The second is to create an Azure Policy to require the setting with "applyIfNotExists" on the Resource Group or Subscription scope- whichever best fits your scenario.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.