Issue integrating Teams with SCOM 2022

Question

I am running into an issue integrating Teams in SCOM 2022.

I followed the Microsoft documentation. Created the app registration in AAD, Granted the delegated permissions ChannelMessage.Send and User.read, set allow public authentication to yes, and finally created a notification channel, subscriber and subscription using an account I assigned a Teams license to. The same account is also setup as an administrator for SCOM and the Management server. Everytime I test the notification I get the following error in the Event Viewer for Operation Manager. Any help would be greatly appreciated.

The callback method DeliverDataToModule failed with exception "Object reference not set to an instance of an object." in the module Microsoft.EnterpriseManagement.HealthService.Modules.Notification.Teams.TeamsNotificationTransportModule.

Workflow: Subscription9cc8cd53_198c_4204_a855_509d8bebe21d
Instance: Alert Notification Subscription Server
Instance ID: {E07E3FAB-53BC-BC14-1634-5A6E949F9230}
Management Group: XXXXXXX
Full Exception Text:
Microsoft.EnterpriseManagement.HealthService.ModuleException: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.EnterpriseManagement.HealthService.Modules.Notification.Teams.Content.TeamsNotificationTransport.Send(TeamsNotification notification)
at Microsoft.EnterpriseManagement.HealthService.Modules.Notification.NotificationTransportModuleBase3.OnDataItem(DataItemBase dataItem, DataItemAcknowledgementCallback acknowledgedCallback, Object acknowledgedState, DataItemProcessingCompleteCallback completionCallback, Object completionState) --- End of inner exception stack trace --- at Microsoft.EnterpriseManagement.HealthService.Modules.Instrumentation.InstrumentedModuleBase1.HandleFatalException(ITraceScope traceScope, Exception error)
at Microsoft.EnterpriseManagement.HealthService.Modules.Notification.NotificationTransportModuleBase3.OnDataItem(DataItemBase dataItem, DataItemAcknowledgementCallback acknowledgedCallback, Object acknowledgedState, DataItemProcessingCompleteCallback completionCallback, Object completionState) at Microsoft.EnterpriseManagement.HealthService.Internal.InputPortCallbacks.InputStreamAcknowledgedCallbackHandler1.InputPortCallbackImplementation(DataItemBase[] dataItems, Boolean isBatchLogicalSet, DataItemAcknowledgementCallback acknowledgementCallback, DataItemProcessingCompleteCallback completionCallback)
at Microsoft.EnterpriseManagement.HealthService.Internal.ModuleContainer`1.Microsoft.EnterpriseManagement.HealthService.Internal.IModuleContainer.DeliverDataToModule(UInt32 inputPortNumber, DataItemBase[] dataItems, Boolean isBatchLogicalSet, DataItemAcknowledgementCallback acknowledgementCallback, DataItemProcessingCompleteCallback completionCallback)
at Microsoft.EnterpriseManagement.HealthService.Internal.ManagedHostHelper.Microsoft.EnterpriseManagement.HealthService.Internal.IManagedHostHelper.DeliverBatchedDataToModule(UInt32 inputPortNumber, Object moduleInputReference, DataItemBase[] dataItems, UInt32 dataItemCount, Boolean isBatchLogicalSet, Object completionReference)

Answer 1

Hi @Christopher Jacobsen ,

Since the whole integration is pretty new, there is almost no helpful troubleshooting information.
Since the exception is generated by the "Alert Notification Subscription Server" Instance I would assume that there is something wrong with the subscrition ""

Could you please run this and check the output:

Get-SCOMNotificationSubscription -Name Subscription9cc8cd53_198c_4204_a855_509d8bebe21d  

Can you please check for relevaant events on your management servers, especially those running the alert notification subscription workflows (part of the respective RP).
Another option would be to open a case at MS...do you have that option?

----------

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)

Regards
Stoyan Chalakov

Answer 2

I don't see anything out of the ordinary with the subscription, I will try removing the the channel, notification and subscription and recreate them to see if that resolves the error. I am not finding any other relevant information in the event viewer. If recreating the subscription doesn't resolve the issue, I do have the option to open a case with Microsoft.

Configuration : Microsoft.EnterpriseManagement.Administration.AlertChangedSubscriptionConfiguration
ManagementGroup : XXXXXXXXXX
Name : Subscription9cc8cd53_198c_4204_a855_509d8bebe21d
DisplayName : Information - Test Teams - Alert on Test Event 100
Description : This is a test alert fired by event ID 100:
Event Description:
This is a Test event 100
Actions : {TeamsAction_16f316dc_ee83_4558_8605_3d9aea2aed29}
ToRecipients : {Testing Teams Integration}
CcRecipients : {}
BccRecipients : {}
Enabled : True
Id : a84e2114-1ede-7689-eda3-8403649a5ecd
ManagementGroupId : 7936cfcb-a52f-f1bd-c100-415c57a41838

Answer 3

Hi Christopher,

Have you ever solved your issue ? We are experiencing the exact same issue and you seem to be the only resource on Internet trying to deal with it.

Thanks.

François

Answer 4

It is an UI BUG where if additional Notification variables are added, the HTML formatting breaks. I am going to report this to the Product Group.

Workaround: Addition of Notification variables should be done manually, keeping the HTML formatting intact. Here is an example.

Added 'Severity' notification variable at the end. Keep a note of the formatting.

{\"text\" : \"Alert: $Data[Default='Not Present']/Context/DataItem/AlertName$ <br> Source: $Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ <br> Path: $Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$ <br> Last modified by: $Data[Default='Not Present']/Context/DataItem/LastModifiedBy$ <br> Last modified time: $Data[Default='Not Present']/Context/DataItem/LastModifiedLocal$ <br> Alert description: $Data[Default='Not Present']/Context/DataItem/AlertDescription$ <br> Alert view link: $Target/Property[Type="Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer"]/WebConsoleUrl$/#/monitoring/drilldown/alert/$UrlEncodeData/Context/DataItem/AlertId$ <br> Source view link: $Target/Property[Type="Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer"]/WebConsoleUrl$/#/monitoring/drilldown/object/$UrlEncodeData/Context/DataItem/ManagedEntity$ <br> Notification subscription ID generating this message: $MPElement$ <br> Severity: $Data[Default='Not Present']/Context/DataItem/Severity$\"}

Answer 5

For me the solution was found un MS Docos. (https://learn.microsoft.com/en-us/system-center/scom/manage-notifications-create-teams-channel?view=sc-om-2022&source=recommendations)

Run As account. Delegated API type is used in Teams integration and the Run As account credentials will be used to authenticate. Run As account should be

• A valid Azure Active Directory (Azure AD) account (sample: user@domain)
• Licensed to use Microsoft Teams
• A member of target Microsoft Teams channel
• Not enabled for multifactor authentication (MFA)

This RunAs account has to be Simple Authentication (select Simple Authentication for setting up authentication with Microsoft Teams/external mail server) from (https://learn.microsoft.com/en-us/system-center/scom/manage-notifications-create-configure?view=sc-om-2022)

Once you made above changes, either re-create Channel, Subscriptions and Subscribers again or just edit and re-add entries and save them.