Authentication and MSMQ4
As part of a general security sweep of Vista and Windows Server 2008, there was a review of the cryptographic algorithms used anywhere within the products. Some of the older ones were deprecated as being too weak to provide meaningful security anymore, and certain stronger algorithms were recommended. This affected the authentication and encryption features of MSMQ and in this post I'm going to cover how MSMQ responded to these new requirements for the authentication feature.
MSMQ has historically offered four hashing algorithms with which to sign a message: MD2, MD4, MD5, and SHA1. In previous versions, MD5 was the default for most messages, and SHA1 was used for HTTP and multicast messaging, which were introduced in MSMQ3. The first change is that SHA1 is now the default for all types of messages, since MD2, MD4, and MD5 have been deprecated as weak. The second change is that by default, MSMQ4 will neither accept messages signed with the weak algorithms nor generate them.
What are the implications for your existing applications?
For messages being sent from MSMQ4 to an older version, it's pretty smooth sailing. MSMQ3 accepts SHA1-signed messages and so does MSMQ2, at least for later service packs of Windows 2000, with no configuration required. Furthermore, your application gets the benefits of SHA1 without any code changes: even if your code explicitly asks for one of the weak algorithms, the message will actually be sent with SHA1. If, for whatever reason, you really need to generate messages with one of the weak algorithms, there is a registry key which will turn off the upgrading and make MSMQ4 honor the algorithm requested by your code. Create a DWORD registry value named WeakHashAlgUpgrade under the key HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\security and set it to 0, then restart the MSMQ service.
For messages being sent to MSMQ4 from an older version, things are more complicated. MSMQ2 and MSMQ3 can generate SHA1-signed messages, but only if the application code explicitly requests SHA1 (or HTTP or multicast messaging is being used, for MSMQ3), so one alternative is to upgrade the application code, but that's easier said than done. The other alternative is to set the MSMQ4 server to accept messages signed with weak algorithms. The first step to doing that is to create a new registry key (not value!) named HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\Security\WeakHashAlgorithms. If that key is present when the MSMQ service is restarted, then MSMQ4 will accept messages signed with any of the weak algorithms, which is probably more wide open than is really necessary — you probably only need to accept MD5. You can selectively re-disable algorithms by creating DWORD values under that new key. It doesn't matter what the values are named, but something like "TurnOffMD4" will make it obvious later on what was intended. The hexadecimal values are 8001 to turn off MD2, 8002 to turn off MD4, and 8003 to turn off MD5. Create as many values are you need to turn off unnecessary weak algorithms, then restart the MSMQ service, and MSMQ4 will accept messages signed with whichever weak algorithms you left enabled while rejecting ones signed with algorithms not used by your application.
What happens to your incoming messages if you don't do that? MSMQ4 rejects them as having bad signatures. If your application requests deadlettering, then they will get deadlettered. If your application requests NACKs, you will get "bad signature" NACKs (message class 0x8006).