New information has been added to this article since publication.
Refer to the Editor's Update below.

Secure By Design

Your Field Guide To Designing Security Into Networking Protocols

Mark Novak and Andrew Roths

Contents

Design Before You Build
Establish Security Guarantees
Always Question Your Assumptions
Man-in-the-Middle Attacks
Undue Trust Relationships
Prepare for Versioning
Conclusion

If you are faced with creating a new communications protocol, what are you going to do to ensure that it is safe and secure? While a complete answer might take an entire volume, here we will highlight some of the most common scenarios and concerns.

Many of the topics considered here are not unique to networking. They apply to any software that has security features, be it encrypting files or making access control decisions. Software security principles transcend technologies and tend to be pretty universal.

What exactly is a protocol? Wikipedia defines it as "a convention or standard that controls or enables the connection, communication, and data transfer between two computing endpoints." A protocol definition will include descriptions of state machines at either end of the communication, detailed specification of message formats, cryptographic algorithms, endianness, port numbers and a myriad other details concerning syntax, semantics, and synchronization. And every aspect of a protocol definition is subject to an attack by a malicious party. We'll go over some examples of attacks against protocols and rules following, which will help you when designing and implementing protocols of your own.

Design Before You Build

This should be obvious, but it is violated constantly in practice. Even the best implementation will not hide design flaws. And while most implementation issues can be fixed quickly, design flaws may require a near total rework of the implementation. The costs of redesign can be considerable even for standalone applications. They can be downright daunting if an application is distributed and has client and server components that must adhere to the same design. [ Editor's Update - 8/14/2006: In Windows Vista, NTLM will still be enabled by default.] Later in this article we discuss steps you should take to prepare your protocols for versioning.

Establish Security Guarantees

Secure software is all about security guarantees. What security protections does your end product promise to deliver? The question may seem trivial or too general, but answering it completely and correctly leads directly to more solid designs, as well as more targeted testing of the finished product. Equally important, if your company is developing a commercial product, the surest way to earn a black eye in security is to oversell its capabilities. Understanding your security guarantees will allow you to avoid making promises that you can not possibly deliver on.

Software vendors today are notoriously bad at explaining their security guarantees. Suppose you are faced with evaluating the security of data backup software. One of your requirements could be to keep your backups secure against a thief. Is an application that promises to "password-protect your data" doing a good job? How is it protecting your password? Does it even encrypt the data? If so, how strong are its cryptographic algorithms? Before you can explain your security guarantees to your customer, you have to be crystal clear on what you're delivering. In the case just described, a good security guarantee would be something like "password-protecting your data will ensure that it will not be recoverable in any reasonable length of time without the knowledge of your password, provided you choose a strong password." Unfortunately, you will rarely see statements like this today.

Even the security guarantees of very well-engineered protocols can and should be questioned. Take Kerberos, for example. Security of Kerberos is based, in part, on the strength of a cipher suite negotiated by the client with the Kerberos Key Distribution Center (KDC). If you are not an expert on Kerberos, you probably do not know how this is done. You may guess (correctly) that the client and the KDC are likely to support different, overlapping suites of cryptographic protocols. Given this, the security guarantee you might expect from such a negotiation could be stated as follows: "the two parties will negotiate the strongest possible cipher suite." So if both client and KDC support 40-bit RC4 and 168-bit 3DES, the stronger 3DES encryption will win.

Let's take a look what happens in practice. First, the client sends the list of cipher suites it understands to the KDC. The KDC goes through this list and selects the strongest cipher suite it understands, and uses that.

On the surface, the goal is achieved. Upon a closer examination, you see the mistake: the designers forgot to consider the possibility that an active man-in-the-middle attacker may alter the data sent from the client to the KDC. As a result, the list of cipher suites is sent to the KDC without any provision for tamper resistance. This enables an attacker to strip all but the weakest cipher suite from the list. The weakest algorithm could be 40-bit RC4—a cipher that can be cracked in 24-48 hours on today's desktop hardware via brute force!

The correct security guarantee of cipher suite negotiation in Kerberos is thus different from the one guessed earlier. It is this: "the two parties will negotiate a cipher suite that is no weaker than the weakest option presented by the client to the KDC." Stating the security guarantee this way enables us to take precautions on the client. Without changing the protocol, we can, as a matter of policy, make it request only cipher suites that are strong enough.

Here are some security guarantees that should be characteristic of most well-engineered protocols:

• A participant in a communication should not be able to adversely affect (or attack) other participants—servers, clients, intermediate nodes, and so on.
• A man-in-the-middle attacker should not be able to eavesdrop on the conversation or alter the contents of the conversation in a way that has security implications.
• An authenticated participant's access must be limited to only what is authorized for that participant.

These are very high-level guarantees that in practice would have to be dissected further, often covering every minute aspect of communication.

As you can see, establishing security guarantees can be a complex and lengthy process. However, some simplifications can be made. For instance, for purposes of security analysis, you should not differentiate between actively malicious, compromised, misconfigured, or malfunctioning nodes. Indeed, all types of anomalous behavior can be caused by an attacker and should be treated as such. Too often we see multiple security guarantees articulated that can be reduced greatly.

Always Question Your Assumptions

Design-level security analysis of any piece of software starts with questioning the assumptions made by its designers. Hackers are especially good at this. If you want your design to stand up to scrutiny, you have to do this analysis yourself. Doing this during the design phase affords you the freedom to uncover and redesign insecure aspects without having to worry about backwards compatibility or making a bad name for yourself. As implementation begins, you begin to lose this advantage. Once your product ships, it is effectively lost.

Here are some assumptions that software designers often make, at their peril. First, a new security technology is built on top of other technologies or protocols without understanding the security guarantees they make. You are probably not going to be designing your protocol from scratch. Rather, you are likely to put together a solution from various building blocks: communication transports, cryptographic primitives, authentication algorithms, and so on. A prerequisite to defining your own security guarantees is understanding the security guarantees of the technologies your design will depend on. An incomplete understanding in this area can easily introduce cracks and limitations into your own guarantees. You cannot build much on a shaky foundation.

Every source of information your application uses to make trust decisions needs to be carefully examined and understood. One example of this is the DNS protocol used in network name resolution. DNS has no security features and makes no security claims about the IP addresses returned during name resolution. An IP address may be that of the intended host or possibly the address of an attacker waiting to strike. A design utilizing DNS must take this lack of guarantee into account. One way to do so is to establish a TLS/SSL session with the resolved address. This way, the machine at the resolved address is forced to provide a certificate signed by someone in the sender's trusted root store. This typically ensures that the identity of the server has been verified by a trusted certificate authority.

As a more advanced example, consider security guarantees you rely upon when using an authentication protocol. Each authentication protocol involves an exchange of messages between the client and server. In addition to the obvious property of proving the client's identity to the server, a good authentication protocol will have at least three properties. First, the client's credentials will not be disclosed to either the server against which the client is authenticating or an eavesdropper listening in on the conversation. Second, the message exchange will be safe against replay (so a recording of the exchange cannot be used by an attacker to spoof the client in a subsequent exchange). And third, the protocol will allow the client and server to establish a secret session key that can be used to sign and encrypt ensuing communications (more on that later). This session key must not be something an attacker with full access to every message sent between client and server can derive.

Commonly used Windows® authentication protocols—Digest, NTLM, and Kerberos—satisfy all three of these properties (see Figure 1). Is this sufficient? The answer is not so simple.

Figure 1 Capabilities of Different Authentication Protocols

For a slightly more complicated example, consider the NTLM authentication protocol. NTLM was designed to verify the authenticity of a client to a server. It provides no verification of a server's authenticity back to the client. In security speak, NTLM suffers from lack of mutual authentication—that fact is shown in the last row of Figure 1. This missing guarantee can cause disastrous results if the client is not designed in a way that accounts for the lack of this security guarantee.

Weaknesses associated with lack of mutual authentication are numerous. What if the client were to run executables sent by the server? What if, upon successfully authenticating to the server, it were to send to the server confidential information such as credit card numbers or various forms of personally identifiable information (PII)? Generally, you solve this problem by utilizing secure transport such as SSL—a transport mechanism that does guarantee the server's authenticity to the client and protects all traffic against eavesdropping and tampering.

It's important to realize you cannot produce the set of security guarantees of your design until you take into account the security guarantees of every protocol, technology, and feature it uses.

Man-in-the-Middle Attacks

It's rarely safe to assume that communication flow is safe from man-in-the-middle attacks, as there exist multiple ways for an attacker to insert himself or herself into your communications channel. The consequences can be grave. You already saw how a man-in-the-middle attack can cause Kerberos to negotiate a dangerously weak cipher suite. Due to the insecure design of early Internet Engineering Task Force (IETF) protocols, attacks such as Address Resolution Protocol (ARP) poisoning or DNS spoofing are well documented, easily executed and effective in many environments. Alternately, a network may contain a rogue DHCP server capable of setting the default gateway of all available clients to a malicious router. Even without an attacker on the local subnet, it is very difficult to be certain that all routers handling traffic passing between two machines are invulnerable to attack and operating in the user's best interest. This is especially true when some of these routers are owned or operated by different corporations—or governments! Finally, most networks are not immune to simple physical alteration—that Ethernet cable in the back of your box may not lead where you think it does. This list is far from exhaustive.

This assumption is often evident in protocols that do not sign or encrypt traffic following authentication. Under these conditions, an attacker can wait for an authentication to complete and then observe or even alter the data that follows. In many cases, this will allow the attacker to masquerade as the authenticated user.

Many protocols will send all their traffic unsecured following authentication: e-mail protocols such as IMAP and POP3 are good examples. A common solution is to send all traffic over a secure transport (IPsec or SSL/TLS). This way, the man-in-the-middle cannot observe the traffic, and any attempt to alter it will be detected by the recipient.

What else can be done to thwart such attacks? A strong authentication protocol (such as Kerberos and, to a lesser extent, NTLM), will enable parties to negotiate a shared session key as part of authentication. This shared key can then be used for two purposes: integrity—signing the messages, thus preventing tampering, and privacy—encrypting traffic, preventing eavesdropping. As mentioned already, a properly engineered authentication protocol provides you with a security guarantee of its own: that a man-in-the-middle will not be able to obtain the shared session key. The secrecy of this key is what protects the entire communication—use it!

Implementing integrity and privacy does not have to be difficult. If you use Security Service Provider Interface (SSPI) for authentication, integrity and privacy can be introduced with just a couple of steps.

Step 1 is to configure the client and server to request integrity and privacy. On the client side, this is done by adding two flags when calling InitializeSecurityContext. ISC_REQ_INTEGRITY and ISC_REQ_CONFIDENTIALITY should be added to the fContextReq parameter. Two flags need to be added on the server side. The parameter is still fContextReq, but in this case the function is AcceptSecurityContext. The two flags here are ASC_REQ_INTEGRITY and ASC_REQ_CONFIDENTIALITY. It is important that both the client and the server use these flags. SSPI is a transport-independent authentication mechanism, so it makes no association between an authentication and the protocol performing that authentication.

Now consider two unique protocols that both use SSPI and negotiate privacy. For Protocol A the client requests privacy, but the server does not. For Protocol B the opposite is true; the server requests privacy, but the client does not. On the surface, everything seems fine; all connections should have at least one side requesting privacy. Unfortunately, an attacker can actually exploit the situation through a cross-protocol attack.

The attacker would establish two connections: one with a Protocol B client and one with a Protocol A server. Once established, each connection would attempt an SSPI authentication with the man-in-the-middle. The attacker doesn't have to worry about either authentication because SSPI is not tied to a specific protocol. The attacker could extract SSPI data from one protocol and insert it into the other. By doing this several times, the attacker would have the authentications for both connections succeed. This attack is illustrated in Figure 2.

Figure 2** Cross-Protocol Attack **

Sadly, neither the client nor the server in this authentication requested privacy. This means the attacker is now free to attack two unencrypted connections. By having both the client and the server request privacy, either protocol could have prevented the situation. And even though two protocols had to make the mistake, it is very difficult to guarantee the security of all other protocols. It is much better to ensure your own security.

Step 2 in implementing integrity and privacy is to verify that integrity/privacy were successfully negotiated. There is no guarantee that the attributes you requested in the last step were successfully negotiated during authentication. To see if the negotiation succeeded, you must check the pfContextAttr parameter returned by InitializeSecurityContext and AcceptSecurityContext. The check consists of ensuring that the flags set in the fContextReq parameter are still set in the pfContextAttr parameter. If something is missing, then the corresponding flag was not successfully negotiated. Communication must not proceed unless each party receives the "quality of protection" it has requested.

Step 3 is to sign and encrypt your data. Even though you successfully negotiated integrity and privacy, data will not be automatically protected. To do so, you must explicitly pass the data requiring protection through one of two functions. For signing, the function is MakeSignature. For encryption, it is EncryptMessage. Whichever function you use, pay particular attention to the fQOP parameter, which specifies the desired quality of protection. Acceptable values vary based on the underlying authentication protocol. In some cases, it is possible to select a value that offers no protection at all. An example of this is the SECQOP_WRAP_NO_ENCRYPT flag available through the EncryptMessage function during Kerberos authentications. If used, this flag will not encrypt the data being passed through EncryptMessage.

Using the negotiated session key to secure your messages mitigates an interesting attack against the NTLM authentication protocol. Partly because of lack of mutual authentication, authenticating with NTLM is unsafe in that when attempted against a malicious server, that server can act as a proxy, forwarding authentication messages and responses to another server—a phenomenon known as a forwarding attack. If a session key is not used to protect messages following authentication, the malicious server can then act as the client against the target server.

The attack (illustrated in Figure 3) is most likely to be pulled off in a corporate environment where people readily follow links to intranet files sent to them by fellow employees. The attack starts with an attacker (the evil server) sending a luring message to the client—for instance, inviting the client to download a file of interest from the evil server's share. In order to obtain the file, the client must authenticate to the evil server, which claims to only support NTLM. The evil server then proxies all messages sent by the client to a target server, against which the evil server wants to act as the client. All authentication messages by the target server are sent back to the client. After authentication completes, the target server believes it has a legitimate session with the client, but is in reality talking with the evil server. Meanwhile, the client is completely unaware that it has authenticated itself to the target server!

Figure 3** NTLM Authentication Forwarding **

The forwarding attack will not work against Kerberos, because Kerberos authentication messages can only be understood by the server they're intended for. Even when using NTLM, successful authentication means that the client and the target server are in possession of a secret session key that the evil server can not obtain. Thus, if the target server insists that messages sent to it are signed and encrypted using this shared session key, the evil server will not be able to spoof a valid message to the target server, and will not be able to understand or alter any message the client sends to the target server, or any of the target server's responses. Do not forget, however, that even when using Kerberos, traffic must still be signed and encrypted, unless you are using a secure transport (such as SSL), since doing so protects against any man-in-the-middle between you and the server.

Undue Trust Relationships

It is easy to assume that utilizing a secure transport will immediately nullify man-in-the-middle attacks. Not necessarily. Let's look at the security guarantees provided by secure transports.

For a common problem, consider SSL and the security guarantees it offers. As often deployed, it will ensure the connection is encrypted and the client it talking with the same server it requested. What ensures that the client requested the appropriate server? You've seen several situations where the request is based on an earlier cleartext communication. In such a situation, a man-in-the-middle attacker can obtain a legitimate SSL certificate and then simply tamper with the plaintext protocol to redirect the SSL connection. With a legitimate SSL connection to the client, the attacker is free to exploit any trust the client puts on having an SSL connection.

Another common problem arises when SSL connections protect insecure authentication protocols. All too often, the insecure authentication protocol is not restricted to the same machine as the SSL connection. Suppose the server at the end of a legitimate SSL connection is actually malicious. In such a position, the malicious server might choose to redirect the insecure authentication protocol to a different server over a different SSL connection (illustrated in Figure 4). If the client doesn't have a way of restricting authentication to the server it has an SSL connection with, it might actually be authenticating to an arbitrary server. Depending on the insecure authentication protocol, this could allow the malicious server to masquerade as the client.

Figure 4** Authentication Redirect **

IPsec is not immune to this type of problem either. Take a security policy requiring an IPsec tunnel every time a certain protocol is used. If done correctly, this will tend to ensure that all participants have valid IPsec credentials. However, there generally will not be any link between the IPsec credentials of the participants and the protocol being used over the tunnel. Thus anyone with valid credentials is free to attack the protocol. While many would-be attackers will not have valid credentials, is the policy really providing the intended level of protection? Suppliers, consultants, and disgruntled employees might all have valid credentials. Are they to be trusted? How about the security of their machines?

Regardless of the secure transport, it is very important to understand the weakness of the underlying protocol and how the transport will protect against an attacker exploiting those weaknesses. Take an extremely poor plaintext authentication protocol. Using a secure transport will definitely help protect against password disclosure, but it may not always provide enough protection. Suppose a client's credentials are valid on multiple servers. If the client logs into one server with the plaintext protocol, that server will be able to reuse the client's credentials on any of the other servers. This could be very bad if one of the servers were malicious or became compromised. In situations like this one, no secure transport will provide complete protection. When that happens, obtaining sufficient protection may necessitate either creating a new version of the weak protocol or replacing it with a different protocol. Here the plaintext authentication could be replaced with SSPI authentication.

Most economic losses from security breaches are due not to Internet-wide disasters such as the Blaster worm, but to insider attacks. Placing undue trust in authenticated parties is what allows a lion's share of these attacks to be perpetrated.

An authenticated party to a conversation is generally more trusted than an anonymous one. This additional trust, however, must be limited strictly to those actions and data allowed by the system's access control policy. What we often see is that an authenticated party in a conversation is considered more trusted to be a good citizen than an anonymous one: input checks are relaxed, access control is loosened, and so on.

By "authenticated parties" we mean not just clients, but also servers. A properly engineered system will not only protect a server against a malicious client, but also protect a client against a malicious server. This is often overlooked. Consider the exchange between client and server shown in Figure 5. The client requests some data and allocates the amount of memory specified by the server. The server sends an extra kilobyte of data. The client copies all 101KB without checking the actual size of the data sent, overflowing the buffer and allowing the server to run arbitrary code on the client.

Figure 5** Malicious Client-Server Exchange **

It may seem odd that a server would ever try to attack a client, but it happens. The machine hosting the server may have become compromised by an attacker. In other cases, the administrator of the server may have become malicious. There is also a possibility that the attack is staged by a man-in-the-middle rather than the server. Whatever the cause, the client is responsible for looking out for its own security. It must verify the input provided by the server and also not assume consistency between successive responses.

Security considerations of this sort extend far beyond just worrying about implementation flaws. Imagine that a legitimate server gets hacked or that the administrator becomes disgruntled. What risk should this pose to clients of the server? While the ideal answer is none, this isn't always practical. Suppose, for instance, that the protocol conveys PII. If the server's role is to process this PII, then a compromised or malicious server will almost always present some risk to a client. In practice, a client's exposure should be restricted to the minimum functionality required by the server. This minimum isn't always obvious.

Consider a protocol that has a self-update feature. As part of the update process, the server may need to send an update executable to a client. At first glance, it appears that the minimum functionality of this feature will allow a compromised server to compromise all clients of that server simply by sending them malicious update executables. This does not have to be the case. The minimum functionality required for this feature actually only involves the distribution of trusted executables, not arbitrary executables. Suppose that the clients are designed such that they will only run executables signed by a particular private key and that the private key is never stored on the server. Suddenly a compromised server doesn't immediately put clients at risk. Now, the private key used for signing executables must also be compromised by an attacker or malicious admin.

All too often we find that clients are exposed to much more than the minimum server functionality. As one last note regarding servers, keep in mind that a server to which many clients connect is a great target for propagating worms.

Prepare for Versioning

Security science does not stand still. A strong encryption or authentication protocol of today may fall to hacker onslaught tomorrow. Your own creation may be discovered to contain a design-level flaw or an implementation bug that fixing would break compatibility between old and new versions. For all of these reasons, you must be prepared to version your protocol. Design your protocol in such a way that the version of the connecting party can be determined early on in the message exchange. Prepare to support multiple authentication schemes, additional ciphers and older clients or servers in newer designs. Here's advice on each of these forms of versioning.

Adding Authentication Schemes SSPI alleviates the need for end protocols to handle the details of authentication mechanisms. It also creates a central location for authentication protocols. In the event of a design flaw in one of these protocols, this central location makes it much easier to update all impacted protocols. Generally, Microsoft recommends that you use the Negotiate package for authentication. This package will do its best to negotiate the use of the stronger Kerberos protocol, and will fall back to NTLM if it is unable to do so. Generally, you should not attempt to accomplish this task better than the Negotiate package. Additionally, the Negotiate package will be updated in the future to use a stronger authentication protocol, if one appears.

Planning for Cryptographic Agility If your protocol depends on the strength of a particular cipher, say 3DES, you might want to prepare yourself in case this cipher will be proven weak in the future. You saw earlier an example of a protocol that negotiates a cipher suite. If you include similar provisions in your design, you stand a chance of upgrading your installed base to stronger cryptography without breaking compatibility. The Cryptography Next Generation (CNG) API planned for Windows Vista will allow you to plug your own cryptographic protocols seamlessly into Windows.

Phasing Out Obsolete Protocols Protocol versioning tends to follow a stepping stone process. When a new version of a protocol is introduced, both versions tend to be supported for some time. This is when the participants are most prone to downgrade attacks—any client or server can claim to be at a lower version in hopes of exploiting some security hole in the design or implementation of the other side or the protocol itself. A versioning strategy must contain a provision of turning off support for downlevel clients or servers once the deployment has reached some critical mass.

Conclusion

Designing a secure protocol is a treacherous task. We have outlined some of the most common pitfalls that await you, but there are many others. Perhaps the best insurance against making mistakes is having your design reviewed by knowledgeable peers, documenting your security guarantees as well as your thought process, and making sure that no holes are introduced during the inevitably iterative software development process.

You cannot succeed at this task without understanding in great detail the security characteristics of your building blocks, including the fact that they, in turn, are not infallible. You can safely assume that security holes will be found eventually, and be prepared to version your design, including the "pluggable" aspects such as authentication and cryptographic primitives.

Mark Novak and Andrew Roths are members of a team at Microsoft whose charter is to analyze upcoming Microsoft products for security vulnerabilities. This article is based on real-world experiences analyzing network protocols.