Case in the corner series: The crashing Add User Wizard
[Today’s post comes to us courtesy of Damian Leibaschoff from Commercial Technical Support ]
The symptoms for this case were that after a few seconds of launching the Add User Wizard, the wizard would crash, without any user input.
When we face a user mode crash like this we usually try to get our hands on the dump that is generated so we can review what triggered the crash. Depending on the crash, the associated wizard log might also contain valuable information (this was not the case here). Furthermore, we might also rely on tools such as Sysinternal’s Process Monitor.
For this case we launched the application within our Windows Debugger to see what was taking place, here is a summary of what we found and how we found it:
Launched Windbg (elevated rights)
Verified that the public symbols were configured properly.
Selected File, Open Executable, and picked C:\Program Files\Windows Small Business Server\Bin\adduser.exe
Once it loaded, proceeded to let the program run while connected to the debugger by issuing the GO ‘g’ command.
The Debugger will break on the second chance exception, on this case a CLR exception, pointing us to an issue while running .Net Framework code.
At this point, we need to load the proper extensions to handle .Net Framework debugging so we can take a peek into what is going on. You do that by loading the SOS extension:
.loadby sos mscorwks
To view the exception, you can now use:
The output for this (key lines highlighted):
0:006> .loadby sos mscorwks
Exception object: 00000000027f30a0
Exception type: System.ArgumentException
Message: An entry with the same key already exists.
SP IP Function
000000001B2AEFC0 000007FEF62AC16DSystem_ni!System.Collections.Generic.SortedList`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Add(Int32, System.__Canon)+0x5d
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x9c
We can easily tell by looking at the exception that we are getting a duplicate entry in a list of what should be unique items. By looking at the CLR call stack that led to the exception we can also see our code trying to find the Primary SMTP address from Exchange. From this, we rapidly found the lines of code that we were using when we hit the exception, confirmed what they were doing and quickly found the cause of the problem. Before we go into the details of what was wrong, let’s continue this scenario as if we did not have access to the source code, how would we have found what was wrong going the long way.
When we look at a working Add User Wizard, we can see where we are using the SMTP address in plain sight:
Once we see that, it is just a matter of understanding how Exchange 2007 manages SMTP addresses to find the place where the problem lies, in our case, the E-mail Address Policies <More information> . On normal Exchange server in SBS, they would look like this:
Yet on the system where we had the crash, they looked like this:
You can also get the output from Exchange Management Shell
Name Priority RecipientFilter
---- -------- ---------------
Default Policy Lowest Alias -ne $null
Windows SBS Email Address Lowest Alias -ne $null
Can you spot what is wrong?
Both policies have the SAME priority, not only that, they have the default LOWEST priority. This priority is a special case, where its actual value in active directory is 2147483647, this is the highest this value can be, thus the lowest priority. Our code queries for all AD objects that are of objectClass=msExchRecipientPolicy and the one with the lowest msExchPolicyOrder is the primary, we should not have duplicates.
To fix this we need to correct the priority of the “Windows SBS Email Address Policy” as that is clearly incorrect. Unfortunately we cannot change it using the Exchange native tools as we get an error:
[PS] C:\>Set-EmailAddressPolicy "Windows SBS Email Address Policy" -Priority 1
Set-EmailAddressPolicy : The operation cannot be completed on the default e-mail address policy.
At line:1 char:23
+ Set-EmailAddressPolicy <<<< "Windows SBS Email Address Policy" -Priority 1
To get this fixed we had to use ADSIEDIT.MSC and manually edit the AD object:
Changing the attribute to the correct value was all that was needed to regain functionality.
This was a very strange situation, the most likely cause was that the original Default Policy had been manually renamed to the Windows SBS Email Address Policy and a subsequent service pack for Exchange tried to recreate the missing Default Policy or it could have been as simple as some incorrect manipulation of AD objects..