question

ChristerAlm-3908 avatar image
0 Votes"
ChristerAlm-3908 asked ·

Azure Notification Hubs - difficult to decipher metrics

Hi,

My team develops and maintains a solution for a customer with technologies such as Azure Functions/Durable Functions, Cosmos DB, Azure Service Bus, .NET based APIs and Notification Hubs. We use templates for sending push notifications and there are several templates per device (iOS/Android). We use both APNS and GCM/FCM. Pricing tier is basic.

I am trying to understand what the different measurement types represent when I use Monitor | Metrics. I've looked at https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftnotificationhubsnamespacesnotificationhubs but not getting much wiser.

My basic task is to ensure that all messages sent to Notification Hubs also reach the devices. I do not see any errors appearing in the graph. It says 0 in, for example, "All incoming Failed Requests" and "APNS/GCM Errors". I also see that "All Outgoing Notifications" usually have the same value as "Successful notifications".

Questions:

  1. What does "All incoming Requests" mean?

  2. And how does it differ from "Incoming Messages"?

  3. How should I compare the above with "All Outgoing Notifications"?

  4. If you see a difference between "All Outgoing Notifications" and "Successful notifications" (in both directions), is there any way to find out the notifications that are failing or may be ongoing?

  5. If you get an error in, for example, "APNS Errors", is there any way to find out what went wrong?


Below is an example from our Azure environment that shows:

  • "All Incoming Requests" differ from "Incoming Messages".

  • "Incoming Messages" are different from "All Outgoing Notifications".

  • "Incoming Messages" + "Installations Management Operations" are not exactly the same as "All Incoming Requests".
    70625-notification-hubs-monitor-metrics-2.jpg




Best regards,

Crippe


azure-notification-hubs
· 3
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@ChristerAlm-3908, Thanks for a detailed description of the question.

Successful notifications mean simply that push notifications have been delivered to the external PNS (for example, APNs for iOS and macOS or FCM for Android devices). It is the responsibility of the PNS to deliver the notifications to target devices.

Push notifications depend on third-party Platform Notification Systems such as Apple's Push Notification Service (APNs) and Google's Firebase Cloud Messaging (FCM), there is no SLA guarantee for the delivery of these messages. After Notification Hubs sends the batches to Platform Notification Systems (SLA guaranteed), it is the responsibility of the Platform Notification Systems to deliver the pushes (no SLA guaranteed).

Notification hub is a single push resource holds cross-platforms (Android, iOS, and Windows versions) push information of one app. - 'All incoming Requests' – Is the total incoming requests for a notification hub (all platforms).

1 Vote 1 ·

@ChristerAlm-3908, Since you have already checked the doc for metric definition and the Portal tip. I'll follow-up with more information on the metrics details (to all your 5 questions ) shortly.

In regard to delivery of the message, Notification delivery can fail at any of the four stages in the push notification process (client, application back end, Notification Hubs, and the platform's push notification service).

Diagnose dropped notifications in Azure Notification Hubs -This article identifies why notifications might get dropped or not be received by devices. It also explains how to determine the root cause.


1 Vote 1 ·

@ChristerAlm-3908, I'll follow-up with more information on the metrics details (to all your 5 questions ) shortly. /I have also shared your feedback (metric definition improvements) internally with our product team.

71471-image.png

Kindly take a look at these docs for additional info:

Push notifications with Azure Notification Hubs: Frequently asked questions

This article provides an overview of all the operational and diagnostics logs that are available.
Enable diagnostics logs for Notification Hubs


1 Vote 1 ·
image.png (48.1 KiB)
ajkuma-MSFT avatar image
1 Vote"
ajkuma-MSFT answered ·

@ChristerAlm-3908 ,

Following-up to answer your 5 questions, after some background of the Notification Hub concepts (in the comments section):

As mentioned in the comments -Notification hub is a single push resource holds cross-platforms (Android, iOS, and Windows versions) push information of one app. - 'All incoming Requests' – Is the total incoming requests for a notification hub (all platforms).

1. What does "All incoming Requests" mean?
A. This is the number of API calls for the given hub (send, installation, etc).

2. And how does it differ from "Incoming Messages"?
A. This is the number of send API calls for the hub ( is all send API calls)

3. How should I compare the above with "All Outgoing Notifications"?
A. Outgoing notifications is the number of pushes we sent to the PNS.
This will not match “Incoming Messages” as this is the number of target devices, not the number of send API calls.

So, in general it will not match Incoming Messages unless each send request only targets a single device

4. If you see a difference between "All Outgoing Notifications" and "Successful notifications" (in both directions), is there any way to find out the notifications that are failing or may be ongoing?
A. For namespaces which are Standard SKU, we provide telemetry/feedback via the following APIs:
https://docs.microsoft.com/en-us/rest/api/notificationhubs/get-notification-message-telemetry
https://docs.microsoft.com/en-us/rest/api/notificationhubs/get-pns-feedback

5. If you get an error in, for example, "APNS Errors", is there any way to find out what went wrong?
A. See answer to #4 above


Summarizing my points from the comments section:

Successful notifications mean simply that push notifications have been delivered to the external PNS (for example, APNs for iOS and macOS or FCM for Android devices). It is the responsibility of the PNS to deliver the notifications to target devices.

Push notifications depend on third-party Platform Notification Systems such (APNs) and (FCM), there is no SLA guarantee for the delivery of these messages. After Notification Hubs sends the batches to Platform Notification Systems (SLA guaranteed), it is the responsibility of the Platform Notification Systems to deliver the pushes (no SLA guaranteed).

In regard to delivery of the message, Notification delivery can fail at any of the four stages in the push notification process (client, application back end, Notification Hubs, and the platform's push notification service).

Diagnose dropped notifications in Azure Notification Hubs -This article identifies why notifications might get dropped or not be received by devices. It also explains how to determine the root cause.

Hope this helps! Thanks for your feedback.

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChristerAlm-3908 avatar image
0 Votes"
ChristerAlm-3908 answered ·

Hi @ajkuma-MSFT

Thank you for very good comments/answers. It is clear that you want to help the community. The plan to divide comments based on how much you know at any given time is especially appreciated!

Now to some follow-up questions. I think it's the vocabulary that I have a hard time grasping.

1 and 2
Please see the new point 6 first.
a) What is the difference between "number of API calls" and "number of send API calls"?
b) Is the difference between point 1 and 2 that the installation management is included in 1?

2 and 3
If I understand correctly, the data in 2 and 3 is the same, ie the same notification text. I understand that a notification can be sent to several devices. But, shouldn't the vocabulary be the same? Shouldn't it be called "All Incoming Notifications" instead of "Incoming Messages"? (Then you can avoid the distortion between notification and message.)

6
The word send, what does it mean? Is that what I send to Notification Hubs? Ie the calls I make via SDKs, applications like PostMan etc? Or can you consider "send" both from outside and inside NH? Is it possible to explain a little more what send means, especially based on incoming and outgoing traffic.


Br,
Crippe

· 6 ·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@ChristerAlm-3908, This is our REST API documentation: - A call to any of these counts as an API call. A call to any of the send APIs (direct single/batch send, native/template send) is a send API call.

All APIs include everything - If you're using an application like PostMan to make raw REST calls to our service, those count as individual calls. Our SDKs wrap our REST APIs, but could technically call us more than once due to things like retries or de-bouncing logic for device registration/installation management.

As mentioned above, the data is not the same, so the vocabulary isn't the same. A single "send" API call to the Notification Hubs service is counted as an "Incoming Message".


1 Vote 1 ·

@ChristerAlm-3908 ,

"Incoming Message" really refers to an "Incoming Send Request" (which perhaps is the confusion). A send API request can send to any number of devices including none at all in the case there are no matching targets.
An "Outgoing Notification" is a push to a device (i.e. an outgoing API call to the third-party PNS). You're feedback has been shared internally with our product team.

Thanks again for your feedback and all the great questions.



To benefit the community find the right answers, please do mark the post which was helpful by clicking on ‘Accept Answer’ & ‘Up-Vote’.




1 Vote 1 ·

Many thanks for the reply again @ajkuma-MSFT

Right now I have no more questions on 1 to 3 and 5 or 6.

Follow-up question on 4.
If I want to check the individual status of all outgoing messages in the last 24 hours, can I do as below?

Start by calling [https://docs.microsoft.com/en-us/rest/api/notificationhubs/get-pns-feedback#response-body] from where you will then start using the "Azure Storage Services REST API". That API can help me get the messageId needed for the next question in [https://docs.microsoft.com/en-us/rest/api/notificationhubs/get-notification-message-telemetry#request]?

Or can I do it any other way?

Is there telemetry for Incoming Messages as well?

0 Votes 0 ·
ajkuma-MSFT avatar image ajkuma-MSFT ChristerAlm-3908 ·

@ChristerAlm-3908, Thanks for the follow-up.

The notification message ID required for the ‘get-notification-message-telemetry’ API is returned as part of the ‘`Location`’ response header as long as the namespace is Standard SKU.

The ‘`Location’` response header is actually the full URL needed to fetch message telemetry.
If our service returns a 200/201 success, we accepted the message and should process it. Then you should use the message telemetry to check actual PNS outcomes.

Hope this helps!
Much appreciate your great feedback.


To benefit the community find the right answers, please do mark the post which was helpful by clicking on ‘Accept Answer’ & ‘Up-Vote’.

1 Vote 1 ·
Show more comments