INFO: FIM Service stops intermittently - FIM Service fails to start

Overview

Recently, I worked on an issue where we saw the FIM Service (Forefront Identity Manager Service) intermittently stopping as well as failing to start.

In issue such as this one, the first place that we need to start is reviewing the Event Logs.  Specifically the System Event Log which captures the starting and stopping of services.  We also need to review the Forefront Identity Manager (FIMService) Event Log.

In the System Event Log I found three messages pertaining to the Forefront Identity Manager Service.

System Event Log: Source = Service Control Manager
7031 The Forefront Identity Manager Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 6000 milliseconds: Restart the service.
7009 A timeout was reached (30000 milliseconds) while waiting for the Forefront Identity Manager Service service to connect.
7000 The Forefront Identity Manager Service service failed to start due to the following error:  The service did not respond to the start or control request in a timely fashion.

 

The Forefront Identity Manager Event Log contained one error at the same time that the Forefront Identity Manager Service (FIMService) was attempting to start.

Forefront Identity Manager Event Log: Source = Microsoft.ResourceManagement
3 mscorlib: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.IdentityManagement.ExternalSettingsManager, Version=4.1.3114.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) File name: 'Microsoft.IdentityManagement.ExternalSettingsManager, Version=4.1.3114.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'    at Microsoft.ResourceManagement.Utilities.ExceptionManager.IsFatal(Exception exception)    at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.ProcessException(Guid workflowInstanceIdentifier, Exception exception)    at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.ProcessWorkflowInstanceOnFinalSignaledEvents(Guid workflowInstanceIdentifier, WorkflowInstanceResourceStatus workflowStatus, Guid cause, Exception exception, Guid& requestIdentifier, String& workflowInstanceType, String& requestStatusMessage)    at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.WorkflowInstanceEnded(Guid workflowInstanceIdentifier, WorkflowInstanceResourceStatus status, Guid cause, Exception exception)    at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.WorkflowInstanceTerminated(Object workflowEventWrapperObj) WRN: Assembly binding logging is turned OFF. To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1. Note: There is some performance penalty associated with assembly bind failure logging. To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].    --- End of inner exception stack trace ---    at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)    at System.Delegate.DynamicInvokeImpl(Object[] args)    at Microsoft.ResourceManagement.Utilities.ThreadManager.ProcessCallBack(Object stateObj)

 

Here are a few key items that stick out in the FIM Service Event Log:

  • Could not load file or assembly 'Microsoft.IdentityManagement.ExternalSettingsManager
    • This line indicates that the FIM Service is attempting to load a Microsoft .NET Assembly and is not able to do so.
  • File Version of the .NET Assembly 4.1.3114.0
    • This particular version is an old Forefront Identity Manager 2010 R2 SP1 build.
    • Noticing this build version, we checked the build version of the Microsoft.ResourceManagement.Service.Exe.  In this particular issue the version of the Microsoft.ResourceManagement.Service.Exe was 4.1.3479.0.
    • We have a mismatch between the Microsoft.ResourceManagement.Service.Exe and the Microsoft .NET assembly that the FIM Service is attempting to load.

Identifying the mismatch of the file versions the next place to look is the FIM Service Configuration File (Microsoft.ResourceManagement.Service.Exe.Config).

  1. Navigate to where you have installed Microsoft Forefront Identity Manager Service
    (%programfiles%\Microsoft Forefront Identity Manager\2010\Service)
  2. Make a backup copy of the Microsoft.ResourceManagement.Service.Exe.Config file.
  3. Edit the Microsoft.ResourceManagement.Service.Exe.Config file
  4. Locate the <runtime> section
  5. In the <runtime> section, you will find multiple items called <bindingRedirect with a property called newVersion.  It is here that you will see the version of the .NET assembly that the FIM Service is attempting to load.
    1. In this particular issue the FIM Service was attempting to load the .NET Assembly with a version of 4.1.3114.0
      <bindingRedirect oldVersion="4.0.0.0-4.65535.65535.65535" newVersion="4.1.3114.0" />

Now we have proven that the error message in the FIM Service Event Log correlate to the FIM Service Configuration File and the System Event Log.  Our next step is to understand how to resolve the issue.

Resolution

In order to resolve the issue, we need to be able to load the correct .NET Assembly.  We can accomplish this task by modifying the bindingRedirect newVersion to the correct build version number.

*NOTE: The correct build version number is the build that you currently have installed.

  1. Follow the steps 1 through 5 documented above
  2. Replace the 4.1.3114.0 with the build of FIM that you are currently running.
  3. Example:
  4. Bad Config:
    1. <bindingRedirect oldVersion="4.0.0.0-4.65535.65535.65535" newVersion="4.1.3114.0" />
  5. Good Config: ( *NOTE: In this particular issue, the current FIM build was 4.1.3479.0.  Remember, you want to replace the newVersion property with the build of FIM that you are currently running. )
    1. <bindingRedirect oldVersion="4.0.0.0-4.65535.65535.65535" newVersion="4.1.3479.0" />  

 

Additional Information

Here is some other links that speak on the FIM Service not starting.