SCOM 2019: Event ID 31555 every 10 minutes

Andrew Perry 1

Hi,

I am currently getting an error logged in the Operations Manager log on one of our Management Servers.

The event id is 31555 from Health Service Modules and the description is:

Failed to generate data to be forwarded to the Data Warehouse. The operation will be retried. Exception 'ArgumentOutOfRangeException': Specified argument was out of the range of valid values. Parameter name: dataItems One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode Instance name: Data Warehouse Synchronization Service Instance ID: {A55A5A06-86DF-47E5-DE03-F11CE415392D} Management group: XX

I have had a look around and not a lot out there on this particular event, but the ones I can find seem to be for a different issue

Taking a wild guess at the error, I am assuming that there is something wrong with Maintenance Mode entries, either scheduled or just in Maintenance.

I have gone through all schedules and there were some that when I tried to open, it gave an error and I couldn't open the schedule. Looking in the database using the query here: https://learn.microsoft.com/en-us/answers/questions/543376/scom-how-to-get-number-of-servers-currently-in-mai and this did show NULL object. I have removed all of those (which were old anyway). Not sure why SCOM breaks like this when I assume a server is later removed??

Anyway, I have gone through every schedule now and everything has objects, but I am still getting the event every 10 minutes.

Has anyone come across this? Can anyone help please?

Thanks

Andrew

1 answer

XinGuo-MSFT 14,386 Reputation points

2023-11-01T07:14:04.7766667+00:00

Hi,

The error you're encountering in System Center Operations Manager (SCOM) with Event ID 31555 typically indicates an issue related to data synchronization between the Management Server and the Data Warehouse. The error message suggests that there's a problem with the data being forwarded to the Data Warehouse, particularly related to Maintenance Mode workflows.

Your investigation into the Maintenance Mode schedules and removing any null objects is a step in the right direction. However, it seems there might still be some underlying issue causing this error to persist.

Here are a few additional steps you can consider to troubleshoot this problem further:

Review Workflow and Configuration: Check the configuration of the "Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode" workflow. Ensure that there are no misconfigurations or inconsistencies in the way it processes Maintenance Mode data.

Check Data Consistency: Validate the consistency of the data being sent to the Data Warehouse. Ensure that the data being generated is in the expected format and within the range of valid values. Data inconsistency might trigger the ArgumentOutOfRangeException you're encountering.

SCOM Database Health: Check the health of the SCOM database. Any database corruption or inconsistency can lead to such issues. Running database maintenance or integrity checks might be helpful.
Please sign in to rate this answer.
Andrew Perry 1 Reputation point

2023-11-01T17:54:50.1233333+00:00

Thanks Xin-Guo,

With regards to the Database Health, everything appears normal. All jobs running and last ran successfully. I have however run the Integrity tasks manually (scheduled weekly and last ran on 28th) and all ran successfully. No other maintenance jobs are failing.

Still getting the events. I have literally removed every schedule now that is either not currently running or doesn't have a future run date but hasn't stopped the events.

Regarding the workflow, I must admit this goes a bit beyond my understanding and I am not sure who to identify if this Microsoft internal workflow is configured how they intend it to be but following through from the rule is as follows:

<Rule ID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode" Target="Microsoft.SystemCenter.DataWarehouseSynchronizationService" Enabled="true" ConfirmDelivery="true">

This then links to:

<WriteActionModuleType ID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceModeWriter" Accessibility="Internal" RunAs="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.ActionAccount">

...which then links to:

<DataType ID="Microsoft.SystemCenter.MaintenanceModeData" Base="System!System.BaseData" Accessibility="Internal">

So, not quite sure what to make of that and whether it is configured correctly.

Finally, with the Data Consistency, is there a tool that I can use to inspect what actual data is being passed at the time? As the event log doesn't really help much on that score. Or is this going to have to be something like WireShark? Any help on this would be appreciated as to how to get at this to check it.

Thanks again for your reply.

Andrew

Andrew Perry 1 Reputation point

2023-11-01T18:59:38.5566667+00:00

Not sure why, but the code I am pasting is being chopped down and only showing the first bit. Here they are in plain text:

Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode (Rule)

<Rule ID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceMode" Target="Microsoft.SystemCenter.DataWarehouseSynchronizationService" Enabled="true" ConfirmDelivery="true">

<Category>System</Category>

<DataSources>

<DataSource ID="MaintenanceModeSource" TypeID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceModeDataSource"> <BatchSize>100</BatchSize> <BatchGenerationFrequencySeconds>30</BatchGenerationFrequencySeconds> <MaintenanceModeEndMaxAgeMinutes>15</MaintenanceModeEndMaxAgeMinutes> <ManagementGroupId>$Target/ManagementGroup/Id$</ManagementGroupId> <SynchronizationProcessId>$MPElement$</SynchronizationProcessId> <SqlAuthentication> <Login>$RunAs[Name="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.SqlAuthenticationActionAccount"]/UserName$</Login> <Password>$RunAs[Name="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.SqlAuthenticationActionAccount"]/Password$</Password> </SqlAuthentication> </DataSource>

</DataSources>

<WriteActions>

<WriteAction ID="WA" TypeID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceModeWriter"> <ManagementGroupId>$Target/ManagementGroup/Id$</ManagementGroupId> <SynchronizationProcessId>$MPElement$</SynchronizationProcessId> <SqlAuthentication> <Login>$RunAs[Name="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.SqlAuthenticationActionAccount"]/UserName$</Login> <Password>$RunAs[Name="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.SqlAuthenticationActionAccount"]/Password$</Password> </SqlAuthentication> </WriteAction>

</WriteActions>

</Rule>

Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceModeWriter (WriteActionModuleType)

<WriteActionModuleType ID="Microsoft.SystemCenter.DataWarehouse.Synchronization.MaintenanceModeWriter" Accessibility="Internal" RunAs="DataWarehouse!Microsoft.SystemCenter.DataWarehouse.ActionAccount">

<Configuration>

<IncludeSchemaTypes> <SchemaType>DataWarehouse!Microsoft.SystemCenter.DataWarehouse.SqlAuthenticationSchema</SchemaType> </IncludeSchemaTypes> <xsd:element name="ManagementGroupId" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="SynchronizationProcessId" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="SqlAuthentication" type="SqlAuthenticationType" minOccurs="0" maxOccurs="1"/>

</Configuration>

<ModuleImplementation>

<Managed> <Assembly>Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse, Culture="", PublicKeyToken="31bf3856ad364e35", Version="7.0.5000.0"</Assembly> <Type>Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.MaintenanceModeWriterModule</Type> </Managed>

</ModuleImplementation>

<InputType>Microsoft.SystemCenter.MaintenanceModeData</InputType>

</WriteActionModuleType>

Microsoft.SystemCenter.MaintenanceModeData (DataType)

<DataType ID="Microsoft.SystemCenter.MaintenanceModeData" Base="System!System.BaseData" Accessibility="Internal">

<Implementation>

<Assembly>Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse, Culture="", PublicKeyToken="31bf3856ad364e35", Version="7.0.5000.0"</Assembly> <Type>Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.MaintenanceModeDataItem</Type>

</Implementation>

</DataType>

Brian W 121 Reputation points

2024-03-28T10:46:32.7966667+00:00

Helpful comments but how would one "Review Workflow and Configuration" and "Data consistency". My SQL servers and databases seem healthy, no bad errors in the weekly DBCCs or other maintenance tasks.
Sign in to comment