How to recover from iisadmin startup "Error 0x80090016 - NTE_BAD_KEYSET"

Article
04/15/2008

Here is the short version:
You need at least the 3.x package of JailBreak and a working IIS Server. The .Net Framework 2.0 (or later) must also be installed on both to give you the aspnet_regiis.exe tool.

On the "broken" server backup the data to enable rollback:
From %windir%\system32\}inetsrv save the metabase.xml and the history and metaback folders
execute the following to save the keyset that the IISAdmin service has created last using the Jailbreak command line tool for this:
jbcsp.exe "Microsoft Internet Information Server" IISKey-backup.xml

Update:
It has been found that aspnet_regiis doesn't export the signature keyset from the "Microsoft Internet Information Server" key container. So the steps need to be ammended by using a separate tool to get that keyset exported. Here is the code you will need to get the Jailbreak toolset to run on the "Microsoft Internet Information Server" key container.

<code name="ImportExportRSAKeys.cs">
using System;
using System.IO;
using System.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.Xml;

public class Sample {
    private static void PrintUsage() {
        Console.WriteLine("Usage: ImportExportRSAKeys.exe [-exp|-imp] fileName RSA-KeyContainer-Name");
    }
    static void Main(string [] args) {
        try {
            if (args.Length < 3) {
                PrintUsage();
            } else if (args[0] == "-exp")
                DoExport(args[1], args[2]);
            else if (args[0] == "-imp")
                DoImport(args[1], args[2]);
            else {
                PrintUsage();
            }
        } catch(Exception e) {
            Console.WriteLine(e.ToString());
        }
    }

    public static void DoExport(string fileName, string containerName) {
        RSACryptoServiceProvider rsa = GetCryptoServiceProvider(containerName, true);
        string xmlString = rsa.ToXmlString(true);
        File.WriteAllText(fileName, xmlString);
        rsa.Clear();
    }

    public static void DoImport(string fileName, string containerName) {
        RSACryptoServiceProvider rsa = GetCryptoServiceProvider(containerName, false);
        rsa.FromXmlString(File.ReadAllText(fileName));
        rsa.PersistKeyInCsp = true;
        rsa.Clear();
    }

    private static RSACryptoServiceProvider GetCryptoServiceProvider(string containerName, bool keyMustExist) {
        CspParameters csp = new CspParameters();
        csp.KeyContainerName = containerName;
        csp.KeyNumber = 2; // AT_SIGNATURE

        csp.Flags |= CspProviderFlags.UseMachineKeyStore;
        if (keyMustExist)
            csp.Flags |= CspProviderFlags.UseExistingKey;

return new RSACryptoServiceProvider(csp);
}

}
</code>

Now get the "good" configuration from the working server:

Start with saving the IIS keyset used on this machine so we can later use that on the "broken" box:
execute the following to save the keyset that the IISAdmin service has created last:
jbcsp.exe "Microsoft Internet Information Server" IISKey-Save.xml
Ensure that the IISAdmin service is stopped (net stop iisadmin /y) and then copy the IISKey-Save.xml and %windir%\system32\inetsrv\metabase.xml file from the working server to the broken machine into the %windir%\system32\inetsrv on that server.

Back on the "broken" machine:

Finalize the repair by importing the keyset that does work with the metabase.xml you just copied:
%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -pi "Microsoft Internet Information Server" %windir%\system32\inetsrv\IISKey-Save.xml –exp

Result:

net start iisadmin will now work. You will of course have to add the unique settings for the machine into the metabase.xml using the backup data but the above steps will save you from having to reinstall IIS (and the Applications that depend on it, like Exchange).

If you are interested in the details of the scenario, read on.

Please keep in mind that the XML/BIN file is only a persistence container while the real metabase you work on using the ABO/ADSI API is always acting on the in-memory representation! When moving the IIS configuration store (i.e. metabase) away from the binary format (metabase.bin in IIS5) to XML (metabase.xml in IIS6) the need to save "confidential" data appropriately didn't vanish and so the IIS team chose to use a bin2hex serialization for those entries/attributes that need to be protected while on-disk.

Obviously that itself does not address the security and so, of course, the blob persisted is encrypted. To help make this encryption fast, the IIS Setup generates a SessionKey that is used to encrypt the protected data before persisting it into the metabase file.

To protect the SessionKey itself a CAPI key pair is used that we create/use using its container name "Microsoft Internet Information Server". This key set is also created by the IISAdmin service on startup if opening the key container fails for some reason (it tries to always create if and only uses it if the error says it is already there). Using this key pair the SessionKey is encrypted and prepended by a header. Then the complete blob is signed using the same key set and the complete data construct is then serialized into the SessionKey attribute in the metabase.

As the "Microsoft Internet Information Server" key set isn't marked to allow the exporting of the private key, the resulting metabase is now bound to this machine as long as you are "playing nice". Jailbreak allows you as an administrator to overcome/override the decision and you can now export the key set.

Relevant Tools/Commands:
List all machine key sets: certutil -key
Show the details of the IIS6 key container: certutil -v -key "Microsoft Internet Information Server"
Import/export/generate CAPI key sets: aspnet_regiis.exe

How to recover from iisadmin startup "Error 0x80090016 - NTE_BAD_KEYSET"

Additional resources