question

TSchneider-3686 avatar image
0 Votes"
TSchneider-3686 asked ·

SNMP v3 discovery not working

Hi all,

some information upfront:
- SNMP v1/v2 is working ok (APC UPS units)
- firewall is turned off on management servers
- SNMP v3 is properly configured on target system and can be queried ok using Paessler SNMP Tester

All of our APC UPS units are monitored via SCOM 2019 UR1 using SNMP v1/v2. Now the devices should be reconfigured for v3 to make them more secure. However, SCOM is not able to discover them. As mentioned, the credentials and setup have been confirmed using SNMP Tester from the very same machine as the discovery runs from.

I can see events 120xx in the OpsMgr event log stating that the discovery is running:

Event 12008

 Discovery Completed 
 Discovery type: FullDiscovery 
 Devices discovered: 0 
 Windows computers filtered: 0 
 Devices in pending list: 1 
 Devices excluded: 0 
 Duration Total (sec): 23 
 Duration of Probing (sec): 23 
 Duration of Processing (sec): 0 
    
 Workflow name: discovery09f29983.328e.48b1.9878.bec925cf1530 
 Instance name: APCv3 
 Instance ID: {823D56A6-EBA9-72AB-F1EA-1525D0AB636C} 

I've used the OpsMgr traces but couldn't see anything in there. Using Microsoft NetMonitor shows that data packets are sent to the APC device, but no response is received:
31967-grafik.png

Using SNMP Test I can see the following communication (taken from the same management server):
32013-grafik.png

I've tried a lot but are currently at a loss. Has anybody run into a similar issue ?

Thanks
Thorsten


msc-operations-manager-general
grafik.png (19.4 KiB)
grafik.png (22.7 KiB)
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

StoyanChalakov avatar image
0 Votes"
StoyanChalakov answered ·

Hi Thorsten,

any other related events on your management servers? Please check on all servers, which are part of the network device monitoring resource pool.
Firewall is always a topic, but surely not in your case, so nice try with disabling it :)
I would also check in the SCOM conolse, under "Pending management" you should also be able to see some kind of reason for this? Can you please check it out?
The other thing to doiuble check are your run as accounts and the requirements regarding those:

Run As accounts for network monitoring in Operations Manager

Last. but not least. According to:

Monitoring networks by using Operations Manager

after the ping and the initial contact over SNMP, SCOM should send a SNMP Get request. Do you see such request in the network traces?

I am just thinking out loud, hoping you might recall something that went forgotten.

Regards,


(If the reply was helpful please don't forget to upvote or accept as answer, thank you)
Stoyan


·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TSchneider-3686 avatar image
0 Votes"
TSchneider-3686 answered ·

Hi Stoyan,

thanks for your suggestions. I did read through them and can confirm that everything is configured as it should be. There are no additional messages in the event log. In Network Pending view I can see the to be monitored device as "No Response SNMP".

The network trace does not show any additional traffic apart from the four attempts as shown in the first screenshot I posted before. And this is the problem I'm having here as we do not see a response from the device. Using the SNMP Tester from the same management server we can see the communication going back and forth.

I have no idea why the APC unit would not send a response back to the discovery server. Even if some parameter is incorrect it should send some information back. Since the SNMP tester sits directly on the network port it would see all packets first before they hit SCOM.

Thanks
Thorsten

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

StoyanChalakov avatar image
0 Votes"
StoyanChalakov answered ·

Hi Thorsten,

very odd indeed. I am not quite sure how I would troubleshoot this. Usually such devices have a configuration option to which hosts they can send SNMP responses, but in your case this is also not a topic, because you mentioned that they have been monitored, using SNMPv1 or v2.
Is there an option to involve the vendor in a way? If they are not responding to SCOM, it can be some config related to SNMPv3 on the APC end.

Regards,
Stoyan

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TSchneider-3686 avatar image
0 Votes"
TSchneider-3686 answered ·

We've been going back and forth between Microsoft support and APC support. Lots of network traces were generated, we even built a temporary SCOM environment on the same subnet as the APC device to rule out any network related issues.

It looks like the issue is in SCOM not following the SNMP v3 implementation to 100% per RFC 3414. During the discovery phase traffic is unencrypted and unauthorized. The msgAuthenticationParameters should be set to a zero-length value. In the network trace it can be seen that SCOM fills that parameter with 12 octects of zeroes. Technically, that is not a zero-length value. The APC device in turn does not accept this and therefore refuses to respond.

And now we are in between not able to solve that.... Neither of them seems to be willing to adjust their code.

Let's see if we can get some traktion in this case

Thorsten

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.