Windows Service Monitoring (reduce false alerts with consecutive sample module)
Sometimes customers need to create their own Windows service monitors. This usually leads them to the Windows Service wizard in the Operations Console.
There is nothing wrong with using the wizard, but let me tell you about one of the drawbacks with using the wizard to create your service monitors. The resultant monitor will check if the service is running at an interval. If it is detected that the service is not running for a single sample, the monitor changes state and generates an alert.
This might be exactly what you want, but sometimes this type of “single pass” monitor can potentially generate a high volume of “false” alerts. The real problem lies in the fact that sometimes a service is stopped or paused for a very short period of time for some reason, and then the service might be running again at the next interval. If you’re using the monitor type created by the wizard, this will appear to create false alerts and “noise” in the environment.
The primary reason we will run into this condition is due to the fact that there is no consecutive sampling implemented here. For this, I created a monitor type that includes a consecutive sample module, which will help decrease the volume of state change events and “false” alerts, and add integrity to your service outage tests by requiring at least two or more intervals to pass while the service is not running.
In addition to including a consecutive sample module, there is another monitor type in this library you can leverage that includes a scheduler module. Reference this monitor type to schedule runtime of this monitor to only generate state changes during business hours. This is useful in cases where you do not want to be alerted about a particular service outage during non-peak times or overnight hours.
Monitor types in the attached sample library
· Check Service State Consecutive Samples Monitor Type
· Check Service State Consecutive Samples with Scheduler Monitor Type
Note: It is suggested to seal the sample Service.Monitoring.Library.xml file attached to this post before importing into your test environment. Sealing the library pack will enable you to reference the monitor types for use in your custom packs.
See part 2 of this article to learn more about how you can leverage this library to create your own Windows service monitors.