Using OpsMgr Notifications in the real world - Part 1

Using notifications such as email, has proven to be somewhat difficult in OpsMgr.  I get questions from each and every customer I work with on this topic.... as most people never get these to work as they want, or expect, without fully understanding the object oriented class model of OpsMgr and what that means to Notifications on alerts.  Even once this is understood, there are often challenges.


First.... lets make sure we use the correct terms.  Alerts are alerts.... a message that is sent to the OpsMgr database with details on something that happened.  A Notification is simply a message in email, IM, SMS, etc... that is triggered by an alert.


Ok - now... on to actually using these things.  :-) 

First off.... notifications are configured via subscriptions.  We can create a subscription and get a notification for every single alert that gets generated.... or we can filter them for specific alerts based on specific criteria. 

For instance... we can subscribe to "all critical alerts" by creating a subscription that includes all classes, all groups, and just looks at "Critical" alert severity on the criteria selection.

We can subscribe to "all critical alerts coming from the exchange management pack" by creating a subscription that includes ONLY Exchange classes, All groups, and just looks at "Critical" alert severity on the criteria selection.

Further filters can be created... using groups.... such as "all critical alerts from the exchange MP for servers in Texas".

Sound good?   At this point.... then - it is time to read my previous blog post on configuring notifications for specific groups and classes.  It will walk you through the steps to filter based on a group of servers, or filter based on specific alerts from MP's you are interested in... and some of the issues created when you do that.


Moving on.....


Now - you have set up your notification subscriptions that you like:

You have created groups that contain the Windows Computer objects and Health Service Watcher objects for each region or team.

You have created multiple subscriptions based on specific management packs... by filtering the subscription based on class, and group.

You have also specified specific criteria for these subscriptions... like only getting critical alerts in email.


Everything seems to be going well.... when someone throws a big wrench in your plan.  You get major complaints with your system because:


Users are complaining that they are getting WAY too many emails.... because they are constantly getting flooded with noise from critical alerts.   


Even with all the work you did.... filtering by specific class, and specific groups... you still get a LOT of email notifications.  The FIRST focus should be on tuning the management packs and fixing problems.   However, it is possible that some MP's are just noisy, or problematic, etc.  You can adjust some of the alert severity to Warning... for alerts you don't want notifications.... but ultimately... you are getting too much in email. 

Possibly - you want alerts in the console... but only want VERY SPECIFIC notifications on the "most critical" items.   Your users have decided to set up a rule in Outlook and send all these noisy critical notifications to a folder.... to keep them out of their Inbox.  This essentially makes sending these notifications useless.... because they will be ignored and annoy the end user.  If we were sending them to a phone/pager... the phone/pager is constantly going off.   What they REALLY want is a way to subscribe to only the alerts they deem REALLY critical.  This may be from 10 rules, or 300 rules.

Using the GUI.... there is no easy way to do this.  We cannot subscribe to a single rule, or monitor, that creates an alert.  We can only subscribe to a CLASS... and therefore we will get email notifications for any critical alert that uses that class as a TARGET.


So.... what we need to develop is a STRATEGY for notifications... so we only get emails/pages on alerts that WE DEEM to be actionable.  There are many ways to accomplish this... I will talk about a few that I have implemented with customers and seem to work pretty well.


1. Using Alert Priority as a notification filter strategy. (this blog post)

2. Using Powershell to create individual notification subscriptions for individual rules/monitors. This is blogged here:

3. Using the SDK and a product connector to modify alerts after they are generated, modifying some property on the alert such as Resolution State to a custom state, which we can then use as a criteria for a subscription.

4. Creating a custom class for a specific set of computers using the authoring console, and then subscribing only to that class.... therefore, we will only get alerts from rules/monitors that use that class as a target.

This is blogged here:


Ok....  let's get started.


#1. Using Alert Priority as a notification filter strategy.

This is my most preferred method right now for most scenarios.  In MOM 2005... we had 7 alert severities to choose from....


This was great.... because we could make our notification criteria something like "Critical Error or higher" and then tune the alerts down or up. 

Now, in OpsMgr - this won't work.... because with sealed MP's, and only three alert severities... it isn't as simple.  There are just too many alerts in our default MP's that are tagged as critical... and perhaps we really don't want to bump them down to warning... we just want to pick which critical alerts to notify on.  In comes "Priority".  

There is a new attribute of an alert in SCOM, called Priority.  High, Medium, or Low.  Essentially, when you think about this - we now have 9 possible states for an alert "severity":  Critical High, Critical Med, Critical Low, Warning High, Warning Med.... etc...etc...

In almost all of the default management packs you download... the alert priority is set to medium.  We have also updated most of the MP's so that alert severity and priority are exposed as adjustable via override.  This way - we can create a new notification subscription, and ONLY select "Critical" severity, and "High" priority:




What you will find.... is there you will get very few notifications by default with this criteria.  This is good!  The next step is to simply examine the rules and monitors that create alerts that you are interested in, and override those to change the alert priority to HIGH. 

Note - when creating this override - sometimes we show the word "High" and sometimes we show the numeric number which relates to "High".  In the database, this is translated numerically, just like Alert Severity is.  See the table below to understand these:

  Critical 2
  Warning 1
  Information 0
  High 2
  Medium 1
  Low 0


Here is an example - I am overriding my Exchange Services Monitor alert to High priority... so I will get email notifications for it based on my subscription above:





Here is an example, for an alert for very low disk space, from a rule, where we have to use the numeric:





From that point forward, you will continue to get all the alerts in the console, but only email notifications on the alerts you want, which you set to have a HIGH priority.


So now we have a strategy!   Now we can have alerts coming in the console as normal... and we will only get email notifications on specific alerts that we have deemed to be a high priority.

You could use this same design for a product connector as well... to limit what is forwarded up the chain to Tivoli, OpenView, Remedy, etc.  


For part two.... see the following blog post on using powershell to subscribe to individual notifications:

Creating granular alert notifications - rule by rule, monitor by monitor