Authentication and Encryption

Archived content. No warranty is made as to technical accuracy. Content may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Chris Brenton

Chapter 9 from Mastering Network Security, published by Sybex, Inc.

Authentication and encryption are two intertwined technologies that help to insure that your data remains secure. Authentication is the process of insuring that both ends of the connection are in fact who they say they are. This applies not only to the entity trying to access a service (such as an end user) but to the entity providing the service, as well (such as a file server or Web site). Encryption helps to insure that the information within a session is not compromised. This includes not only reading the information within a data stream, but altering it, as well.

While authentication and encryption each has its own responsibilities in securing a communication session, maximum protection can only be achieved when the two are combined. For this reason, many security protocols contain both authentication and encryption specifications.

On This Page

The Need for Improved Security
Good Authentication Required
Encryption 101
Good Encryption Required

The Need for Improved Security

When IP version 4, the version currently in use on the Internet, was created back in the '70s, network security was not a major concern. While system security was important, little attention was paid to the transport used when exchanging information. When IP was first introduced, it contained no inherent security standards. The specifications for IP do not take into account that you may wish to protect the data that IP is transporting. This will change with IP version 6, but it appears that wide acceptance of this new specification is still many years away.

Clear Text Transmissions

IP currently transmits all data as clear text, which is commonly referred to as transmitting in the clear. This means that the data is not scrambled or rearranged; it is simply transmitted in its raw form. This includes data and authentication information. To see how this appears, let's start by looking at Figure 9.1.

Figure 9.1: A packet decode of an authentication session initializing

Figure 9.1: A packet decode of an authentication session initializing

Figure 9.1 shows a network analyzer's view of a communication session. We have a user who is in the process of retrieving mail with a POP3 mail client. Packets 3–5 are the TCP three-packet handshake used to initialize the connection. Packets 6 and 7 are the POP3 mail server informing the client that it is online and ready. In packet 8 we start finding some very interesting information. If you look towards the bottom of Figure 9.1, you will see the decoded contents of the data field within packet 8. The command USER is used by a POP3 client to pass the logon name to a POP3 server. Any text following the USER command is the name of the person who is attempting to authenticate with the system.

Figure 9.2 shows the POP3 server's response to this logon name. If you look at the decode for packet 9, you can see that the logon name was accepted. This tells us that the logon name you captured in Figure 9.1 is in fact legitimate. If you can discover this user's password, you will have enough information to gain access to the system.

Figure 9.2: The POP3 server accepting the logon name

Figure 9.2: The POP3 server accepting the logon name

In Figure 9.3 you can see a decode of packet 11. This is the next set of commands sent by the POP3 mail client to the server. The command PASS is used by the client to send the password string. Any text that follows this command is the password for the user attempting to authenticate with the system. As you can see, the password is plainly visible.

Figure 9.3: The POP3 client sending the user's password

Figure 9.3: The POP3 client sending the user's password

In Figure 9.4 we see a decode of packet 12. This is the server's response to the authentication attempt. Notice that the server has accepted the logon name and password combination. We now know that this was a valid authentication session and that we have a legitimate logon name and password combination in order to gain access to the system. In fact, if we decoded further packets, we would be able to view every e-mail message downloaded by this user.

Figure 9.4: The POP3 server accepting the authentication attempt

Figure 9.4: The POP3 server accepting the authentication attempt

Passively Monitoring Clear Text

This POP3 authentication session was captured using a network analyzer. A network analyzer can be either a dedicated hardware tool or a software program which runs on an existing system. Network analyzer software can be purchased for less than $1,000 for Windows or Mac platforms and is freely available for UNIX.

Network analyzers operate as truly passive devices, meaning that they do not need to transmit any data on to the network in order to monitor traffic. While some analyzers do transmit traffic (usually in an effort to locate a management station), it is not a requirement. In fact, an analyzer does not even need a valid network address. This means that a network analyzer can be monitoring your network and you would have no means of detecting its presence without tracing cables and counting hub and switch ports.

It is also possible for an attacker to load network analyzer software onto a compromised system. This means that an attacker does not need physical access to your facility in order to monitor traffic. She can simply use one of your existing systems to capture the traffic for her. This is why it is so important to perform regular audits on your systems. You clearly do not want a passively monitoring attack to go unnoticed.

In order for a network analyzer to capture a communication session, it must be connected somewhere along the session's path. This could be on the network at some point between the system initializing the session and the destination system. This could also be accomplished by compromising one of the systems at either end of the session. This means that an attacker cannot capture your network traffic over the Internet from a remote location. She must place some form of probe or analyzer within your network.

Note: As you saw in Chapter 4, you can reduce the amount of traffic that an analyzer can capture using bridges, switches, and routers.

Clear Text Protocols

POP3 is not the only IP service that communicates via clear text. Nearly every nonproprietary IP service which is not specifically designed to provide authentication and encryption services transmits data as clear text. Here is a partial list of clear text services:

  • FTP Authentication is clear text.

  • Telnet Authentication is clear text.

  • SMTP Contents of mail messages are delivered as clear text.

  • HTTP Page content and the contents of fields within forms are sent clear text.

  • IMAP Authentication is clear text.

  • SNMPv1 Authentication is clear text.

Warning: The fact that SNMPv1 uses clear text is particularly nasty. SNMP is used to manage and query network devices. This includes switches and routers, as well as servers and even firewalls. If the SMTP password is compromised, an attacker can wreak havoc on your network. SNMPv2 and SNMPv3 include a message algorithm similar to the one used with Open Shortest Path First (OSPF). This provides a much higher level of security and data integrity than the original SNMP specification. Unfortunately, not every networking device supports SNMPv2, let alone SNMPv3. This means that SNMPv1 is still widely used today.

Good Authentication Required

The need for good authentication should by now be obvious. A service that passes logon information as clear text is far too easy to monitor. Easily snooped logons can be an even larger problem in environments that do not require frequent password changes. This gives our attacker plenty of time to launch an attack using the compromised account. Also of concern is that most users try to maintain the same logon name and password for all accounts. This means that if I can capture the authentication credentials from an insecure service (such as POP3), I may now have a valid logon name and passwords to other systems on the network, such as NT and NetWare servers.

Good authentication goes beyond validating the source attempting to access a service during initial logon. You should also validate that the source has not been replaced by an attacking host in the course of the communication session. This type of attack is commonly called session hijacking.

Session Hijacking

Consider the simple network drawing in Figure 9.5. A client is communicating with a server over an insecure network connection. The client has already authenticated with the server and has been granted access. Let's make this a fun example and assume that the client has administrator-level privileges. Woolly Attacker is sitting on a network segment between the client and the server and has been quietly monitoring the session. This has given the attacker time to learn what port and sequence numbers are being used to carry on the conversation.

Figure 9.5: An example of a man-in-the-middle attack

Figure 9.5: An example of a man-in-the-middle attack

Now let's assume that Woolly Attacker wishes to hijack the administrator's session in order to create a new account with administrator-level privileges. The first thing he does is force the client into a state where it can no longer communicate with the server. This can be done by crashing the client by sending it a Ping of death or through a utility such as WinNuke. This can also be done by launching an attack such as an ICMP flood. No matter what type of attack Woolly launches, his goal is to insure that the client cannot respond to traffic sent by the server.

Note: When an ICMP flood is launched against a target, the target spends so much time processing ICMP requests that it does not have enough time to respond to any other communications.

Now that the client is out of the way, Woolly Attacker is free to communicate with the server as if he were the client. He can do this by capturing the server's replies as they head back to the client in order to formulate a proper response. If Woolly has an intimate knowledge of IP, he may even be able to completely ignore the server's replies and transmit port and sequence numbers based on what the expected responses from the server will be. In either case, Woolly Attacker is now communicating with the server—except that the server thinks it is still communicating with the original client.

So good authentication should also verify that the source remains constant and has not been replaced by another system. This can be done by having the two systems exchange a secret during the course of the communication session. A secret can be exchanged with each packet transmitted or at random intervals during the course of the session. Obviously, verifying the source of every packet is far more secure than verifying the source at random intervals. The communication session would be even more secure if you could vary the secret with each packet exchange. This would help to insure that your session would not be vulnerable to session hijacking.

Verifying the Destination

The need to authenticate the source both before and during a communication session is apparent. What may not be apparent is the need to verify the server. Many people take for granted that they will either connect to the intended server or that they will receive some form of host unreachable message. It may not dawn on them that what they assume is the server may actually be an attacker attempting to compromise the network.


The C2MYAZZ utility is an excellent example of a server spoofing attack. When Windows 95 was originally introduced, it included two methods of authenticating with a session message block (SMB) system. The default was to authenticate using an encrypted password. This was the preferred method for authenticating with a Windows NT domain. LANMAN authentication was also included, however, for backwards compatibility with SMB LANMAN server. LANMAN authentication requires that the logon name and password be sent in the clear.

When C2MYAZZ is run, it passively waits for a client to authenticate to the NT server. When a logon is detected, C2MYAZZ transmits a single packet back to the client requesting that LANMAN authentication be used instead. The client, trusting that this is the server sending the request, happily obliges and retransmits the credentials in the clear. The C2MYAZZ utility would then capture and display the logon name and password combination. C2MYAZZ causes no disruption in the client's session, as the user will still be able to logon and gain system access.

What makes this utility even more frightening is that it can be run from a single bootable floppy disk. An attacker only needs to place this disk into the floppy drive of a system, power the system on, and come back later to collect the captured credentials.

Note: Microsoft did release a patch for this vulnerability, but you need to install it on every Windows 95 workstation.

DNS Poisoning

Another exploit which displays the need for authentication is DNS poisoning. DNS poisoning, also known as cache poisoning, is the process of handing out incorrect IP address information for a specific host with the intent to divert traffic from its true destination. Eugene Kashpureff proved this was possible in the summer of 1997 when he diverted requests for InterNIC hosts to his alternate domain name registry site called AlterNIC. He diverted these requests by exploiting a known vulnerability in DNS services.

When a name server receives a reply to a DNS query, it does not validate the source of the reply or ignore information not specifically requested. Kashpureff capitalized on these vulnerabilities by hiding bogus DNS information inside valid replies. The name server receiving the reply would cache the valid information, as well as the bogus information. The result was that if a user tried to resolve a host within the InterNIC's domain (for example which is used for whois queries), she would receive an IP address within AlterNIC's domain and be diverted to a system on the AlterNIC network.

While Kashpureff's attack can be considered to be little more than a prank, it does open the door to some far nastier possibilities. In an age when online banking is the norm, consider the ramifications if someone diverted traffic from a bank's Web site. An attacker, using cache poisoning to divert bank traffic to an alternate server, could configure the phony server to appear identical to the bank's legitimate server.

When a bank client attempts to authenticate to the bank's Web server in order to manage his bank account, an attacker could capture the authentication information and simply present the user with a banner screen stating that the system is currently offline. Unless digital certificates are being used, the client would have no way of knowing he'd been diverted to another site unless he happened to notice the discrepancy in IP addresses.

Note: Digital certificates are described in the "Digital Certificate Servers" section later in this chapter.

It is just as important that you verify the server you are attempting to authenticate with as it is to verify the client's credentials or the integrity of the session. All three points in the communication process are vulnerable to attack.

Encryption 101

Cryptography is a set of techniques used to transform information into an alternate format which can later be reversed. This alternate format is referred to as the ciphertext and is typically created using a crypto algorithm and a crypto key. The crypto algorithm is simply a mathematical formula which is applied to the information you wish to encrypt. The crypto key is an additional variable injected into the algorithm to insure that the ciphertext is not derived using the same computational operation every time the algorithm processes information.

Let's say the number 42 is extremely important to you and you wish to guard this value from peering eyes. You could create the following crypto algorithm in order to encrypt this data:

data / crypto key + (2 x crypto key)

This process relies on two important pieces: the crypto algorithm itself and the crypto key. Both are used to create the ciphertext, which would be a new numeric value. In order to reverse the ciphertext and produce an answer of 42, you need to know both the algorithm and the key. There are less secure crypto algorithms known as Caesar ciphers which do not use keys, but these are typically not used because they do not have the additional security of a crypto key. You only need to know the algorithm for a Caesar cipher in order to decrypt the ciphertext.

Note: Julius Caesar is credited as being one of the first people to use encryption. It is believed that he used a simple form of encryption in order to send messages to his troops.

Since encryption uses mathematical formulas, there is a symbiotic relationship between

  • The algorithm

  • The key

  • The original data

  • The ciphertext

This means that knowing any three of these pieces will allow you to derive the fourth. The exception is knowing the combination of the original data and the ciphertext. If you have multiple examples of both, you may be able to discover the algorithm and the key.

Methods of Encryption

The two methods of producing ciphertext are

  • The stream cipher

  • The block cipher

The two methods are similar except for the amount of data each encrypts on each pass. Most modern encryption schemes use some form of a block cipher.

Stream Cipher

The stream cipher is one of the simplest methods of encrypting data. When a stream cipher is employed, each bit of the data is sequentially encrypted using one bit of the key. A classic example of a stream cipher was the Vernam cipher used to encrypt teletype traffic. The crypto key for the Vernam cipher was stored on a loop of paper. As the teletype message was fed through the machine, one bit of the data would be combined with one bit of the key in order to produce the ciphertext. The recipient of the ciphertext would then reverse the process, using an identical loop of paper to decode the original message.

The Vernam cipher used a fixed-length key, which can actually be pretty easy to deduce if you compare the ciphertext from multiple messages. In order to make a stream cipher more difficult to crack, you could use a crypto key which varies in length. This would help to mask any discernible patterns in the resulting ciphertext. In fact, by randomly changing the crypto key used on each bit of data, you can produce ciphertext that is mathematically impossible to crack. This is because using different random keys would not generate any repeating patterns which can give a cracker the clues required to break the crypto key. The process of continually varying the encryption key is known as a one-time pad.

Block Cipher

Unlike stream ciphers, which encrypt every single bit, block ciphers are designed to encrypt data in chunks of a specific size. A block cipher specification will identify how much data should be encrypted on each pass (called a block) as well as what size key should be applied to each block. For example, the Data Encryption Standard (DES) specifies that DES encrypted data should be processed in 64-bit blocks using a 56-bit key.

There are a number of different algorithms that can be used when processing block cipher encryption. The most basic is to simply take the data and break it up into blocks while applying the key to each. While this method is efficient, it can produce repetitive ciphertext. If two blocks of data contain exactly the same information, the two resulting blocks of ciphertext will be identical, as well. As mentioned earlier, a cracker can use ciphertext which repeats in a nonrandom fashion to break the crypto key.

A better solution is to use earlier resultants from the algorithm and combine them with later keys. Figure 9.6 shows one possible variation. The data you wish to encrypt is broken up into blocks labeled DB1–DB4. An initialization vector (IV) is added to the beginning of the data to insure that all blocks can be properly ciphered. The IV is simply a random character string to insure that two identical messages will not create the same ciphertext. To create your first block of ciphertext (CT1), you mathematically combine the crypto key, the first block of data (DB1), and the initialization vector (IV).

Figure 9.6: Block cipher encryption

Figure 9.6: Block cipher encryption

When you create the second block of ciphertext (CT2), you mathematically combine the crypto key, the first block of ciphertext (CT1), and the second block of data (DB2). Because the variables in your algorithm have changed, DB1 and DB2 could be identical, but the resulting ciphertext (CT1 and CT2) will contain different values. This helps to insure that the resulting ciphertext is sufficiently scrambled so that it appears completely random. This process of using resulting ciphertext in order to encrypt additional blocks of data will continue until all the data blocks have been processed.

There are a number of different variations on how to mathematically combine the crypto key, the initialization vector, and previously created ciphertext. All these methods share the same goal, which is to create a seemingly random character string of ciphertext.

Public/Private Crypto Keys

So far, all the encryption techniques we have discussed use secret key algorithms. A secret key algorithm relies on the same key to encrypt and to decrypt the ciphertext. This means that the crypto key must remain secret in order to insure the confidentiality of the ciphertext. If an attacker learns your secret key, she would be able to unlock all encrypted messages. This creates an interesting Catch-22, because you now need a secure method of exchanging the secret key in order to use the secret key to create a secure method of exchanging information!

In 1976, Whitfield Diffie and Martin Hellman introduce the concept of public cipher keys in their paper "New Directions in Cryptography." Not only did this paper revolutionize the cryptography industry; the process of generating public keys is now known as Diffie-Hellman.

In layman's terms, a public key is a crypto key that has been mathematically derived from a private or secret crypto key. Information encrypted with the public key can only be decrypted with the private key; however, information encrypted with the private key cannot be decrypted with the public key. In other words, the keys are not symmetrical. They are specifically designed so that the public key is used to encrypt data, while the private key is used to decrypt ciphertext.

This eliminates the Catch-22 of the symmetrical secret key, because a secure channel is not required in order to exchange key information. Public keys can be exchanged over insecure channels while still maintaining the secrecy of the messages they encrypted. If your friend Fred Tuttle wants to send you a private message, all Fred has to do is encrypt it using your public key. The resulting ciphertext can then only be decrypted using your private key.

Diffie-Hellman can even be used to provide authentication. This is performed by signing a message with your private key before encrypting it with the recipient's public key. Signing is simply a mathematical algorithm which processes your private key and the contents of the message. This creates a unique digital signature which is appended to the end of the message. Since the contents of the message are used to create the signature, your digital signature will be different on every message you send.

For example, let's say you want to send Fred a private message. First you create a digital signature using your private key, then you encrypt the message using Fred's public key. When Fred receives the message, he first decrypts the ciphertext using his private key and then checks the digital signature using your public key. If the signature matches, Fred knows that the message is authentic and that it has not been altered in transit. If the signature does not match, Fred knows that either the message was not signed by your private key or that the ciphertext was altered in transit. In either event, the recipient knows that he should be suspicious of the contents of the message.

Encryption Weaknesses

Encryption weaknesses fall into one of three categories:

  • Mishandling or human error

  • Deficiencies in the cipher itself

  • Brute force attacks

When deciding which encryption method best suits your needs, make sure you are aware of the weaknesses of your choice.

Mishandling or Human Error

While the stupid user syndrome may be an odd topic to bring up when discussing encryption methods, it does play a critical role in insuring that your data remains secure. Some methods of encryption lend themselves better to poor key management practices than others. When selecting a method of encryption, make sure you have the correct infrastructure required to administer the cipher keys in an appropriate manner.

Proper Key Management Is Key

Back in the 1940s, the Soviet Union was using a one-time pad in order to encrypt its most sensitive data. As you saw in the section on stream ciphers, it is mathematically impossible to break encryption using a one-time pad. This, of course, assumes that the user understands the definition of "one-time." Apparently, the Soviet Union did not.

Since cipher keys were in short supply, the Soviet Union began reusing some of its existing one-time pad keys by rotating them through different field offices. The assumption was that as long as the same office did not use the same key more than once, the resulting ciphertext would be sufficiently secure (how many of you can see your pointy-haired boss making a similar management decision?).

Apparently, this assumption was off base: the United States was able to identify the duplicate key patterns and decrypt the actual messages within the ciphertext. For more than five years, the United States was able to track Soviet spying activity within the United States. This continued until information regarding the cracking activity was relayed to a double agent.

While a one-time pad may be the most secure cipher to use, you must be able to generate enough unique keys to keep up with your data encryption needs. Even if you will use a regular secret key cipher, you must make sure that you have a secure method of exchanging key information between hosts. It does little good to encrypt your data if you are simply going to transmit your secret key over the same insecure channel.

Simple key management is one of the reasons that public/private cipher keys have become so popular. The ability to exchange key information over the same insecure channel that you wish to use for your data has great appeal. This greatly simplifies management: you can keep your private key locked up and secure while transmitting your public key using any method you choose.

Warning: You must make sure that the public keys you use to encrypt data have been received from the legitimate source and not from an attacker who has swapped in a private key of his own. The validity of a public key can easily be authenticated through a phone call or some other means.

Cipher Deficiencies

Determining whether there are any deficiencies in the cipher algorithm of a specific type of encryption is probably the hardest task a non-cryptographer can attempt to perform. There are, however, a few things you can look for to insure that the encryption is secure:

  • The mathematical formula which makes up the encryption algorithm should be public knowledge. Algorithms which rely on secrecy may very well have flaws that can be extorted in order to expedite cracking.

  • The encryption algorithm should have undergone open public scrutiny. Anyone should be able to evaluate the algorithm and be free to discuss his or her findings. This means that analysis of the algorithm cannot be restricted by confidentiality agreements or contingent on the cryptographer's signing a nondisclosure agreement.

  • The encryption algorithm should have been publicly available for a reasonable amount of time in order to insure that a proper analysis has been performed. An encryption algorithm with no known flaws that has only been publicly available for a few months has not stood the test of time. One of the reasons that many people trust DES encryption is that it has been around for nearly 15 years.

  • Public analysis should have produced no useful weaknesses in the algorithm. This can be a gray area because nearly all encryption algorithms have some form of minor flaw. As a rule of thumb, the flaws found within an algorithm should not dramatically reduce the amount of time needed to crack a key beyond what could be achieved by trying all possible key combinations.

By following these simple guidelines, you should be able to make an educated estimation about the relative security of an encryption algorithm.

Brute Force Attacks

A brute force attack is simply an attempt to try all possible cipher key combinations in order to find the one that unlocks the ciphertext. This is why this attack is also known as an exhaustive key search. The cracker makes no attempt to actually crack the key, but relies on the ability to try all possible key combinations in a reasonable amount of time. All encryption algorithms are vulnerable to brute force attacks.

There are a couple of key terms in the preceding paragraph. The first is "reasonable." An attacker must feel that launching a brute force attack is worth the time. If an exhaustive key search will produce your VISA platinum card number in a few hours, the attack may be worth the effort. If, however, four weeks of work are required in order to decrypt your father-in-law's chili recipe, a brute force attack may not be worth the attacker's effort.

The other operative word is "vulnerable." While all encryption algorithms are susceptible to a brute force attack, some may take so long to try all possible key combinations that the amount of time spent cannot be considered reasonable. For example, encryption using a one-time pad can be broken using a brute force attack, but the attacker had better plan on having many of his descendants carry on his work long after he is gone. To date, the earth has not existed long enough for an attacker to be able to break a proper one-time pad encryption scheme using existing computing power.

So the amount of time required to perform a brute force attack is contingent on two factors: how long it takes to try a specific key and how many possible key combinations there are. The amount of time required to try each key is dependent on the device providing the processing power. A typical desktop computer is capable of testing approximately five keys per second. A device specifically designed to break encryption keys may be able to process 200 keys or more per second. Of course, greater results can be achieved by combining multiple systems.

As for the number of possible key combinations, this is directly proportional to the size of the cipher key. Size does matter in cryptography: the larger the cipher key the more possible key combinations exist. Table 9.1 shows some common methods of encryption, along with their associated key size. Notice that as the size of the key increases, the number of possible key combinations increases exponentially.

Table 9.1 Methods of Encryption and Their Associated Keys


Bits in Key

Number of Possible Keys







Triple DES (2 keys)






Triple DES (3 keys)



Future standard?



Of course, all this leads to the question: how long does it take to perform an exhaustive key search on a particular encryption algorithm? The answer should scare you. DES encryption (discussed in the DES section of this chapter) has become somewhat of an industry standard. Over the past few years, RAS Laboratories has staged a DES challenge in order to see how long it would take for a person or persons to crack a string of ciphertext and discover the message hidden inside.

In 1997, the challenge was completed in approximately five months. In January 1998, the challenge was completed in 39 days. During the last challenge, in July 1998, the Electronic Frontier Foundation (EFF) was able to complete the challenge in just under three days.

The EFF accomplished this task through a device designed specifically for brute forcing DES encryption. The cost of the device was approximately $250,000—well within the price range of organized crime and big business. Just after the challenge, the EFF published a book entitled Cracking DES (O'Reilly and Associates), which completely documents the design of the device they used. Obviously, this has put a whole new spin on what key lengths are considered secure.

Government Intervention

As you may know, the federal government regulates the export or use of encryption across U.S. borders. These regulations originated back in World War II, when the use of encryption was thought to be limited to spies and terrorists. These regulations still exist today due in part to the efforts of the National Security Agency (NSA). The NSA is responsible for monitoring and decrypting all communication that can be considered of interest to the security of the United States government.

Specifically, the regulations control the cipher key size which may be exported or used across U.S. borders. The current limitation is a maximum key size of 40 bits, but there are exceptions to this rule. Organizations that wish to use a larger key size must apply to the Department of Commerce and obtain a license to do so under the International Traffic in Arms Regulations (ITAR). In order to obtain a usage license for keys larger than 40 bits, you typically must be a financial institution or a U.S.-based company with foreign subsidiaries.

In order to sell encryption software larger than 40 bits, the Department of Commerce will typically force you to escrow the portion of the key in excess of 40 bits. For example, Lotus Notes uses a 56-bit key in the international version of its product. This is because Lotus has escrowed 16 bits of the key. If the NSA ever has a need to decrypt one of these Lotus Notes sessions, it already has 16 of the required 56 bits. This means that the NSA would only need to brute force 40 bits, the same as any other exportable encryption.

Given the decreasing times required to perform brute force attacks, a change in the government regulations regarding encryption is obviously overdue. It will not be long before the amount of time needed to perform exhaustive key searches is measured in hours instead of days or weeks. Once that point is reached, the small key sizes currently in use will have outlived their usefulness.

Good Encryption Required

If you are properly verifying your authentication session, why do you even need encryption? Encryption serves two purposes:

  • To protect the data from snooping

  • To protect the data from being altered

In the section on clear text transmissions earlier in this chapter, you saw how most IP services transmit all information in the clear. This should be sufficient justification for why you need encryption to shield your data from peering eyes.

Encryption can also help to insure that your data is not altered during transmission. This is commonly referred to as a man-in-the-middle attack, because it relies on the attacker's ability to disrupt the data transfer. Let's assume you have a Web server configured to accept online catalog orders. Your customer fills out an online form, which is then saved on the Web server in a plain text format. At regular intervals, these files are transferred to another system via FTP or SMTP.

If an attacker can gain access to the Web server's file system, she would be able to modify these text files prior to processing. A malicious attacker could then change quantities or product numbers in order to introduce inaccuracies. The result is a very unhappy client when the wrong order is received. While this example assumes that the attacker has gained access to a file system, it is possible to launch a man-in-the-middle attack while information is in transit on the network, as well.

So while your attacker has not stolen anything, she has altered the data—and disrupted your business. Had this information been saved using a good encryption algorithm, this attack would have been far more difficult to stage because the attacker would not know which values within the encrypted file to change. Even if she were a good guesser, the algorithm decrypting the cipher would detect the change in data.


There are a number of solutions available for providing authentication and encryption services. Some are products produced by a specific vendor, while others are open standards. Which option is the right one for you depends on your specific requirements. The options listed below are the most popular for providing authentication, encryption, or a combination of the two. Most likely, one of these solutions can fill your needs.

Data Encryption Standard (DES)

DES is the encryption standard used by the United States government for protecting sensitive, but not classified, data. The American National Standards Institute (ANSI) and the Internet Engineering Task Force (IETF) have also incorporated DES into security standards. DES is by far the most popular secret key algorithm in use today.

The original standard of DES uses a 40-bit (for export) or 56-bit key for encrypting data. The latest standard, referred to as Triple DES, encrypts the plain text three times using two or three different 56-bit keys. This produces ciphertext which is scrambled to the equivalent of a 112-bit or 168-bit key, while still maintaining backwards compatibility.

DES is designed so that even if someone knows some of the plain text data and the corresponding ciphertext, there is no way to determine the key without trying all possible keys. The strength of DES encryption–based security rests on the size of the key and on properly protecting the key. While the original DES standard has been broken in brute force attacks of only three days, the new Triple DES standard should remain secure for many years to come.

Digital Certificate Servers

As you saw in the section on public and private cipher keys, a private key can be used to create a unique digital signature. This signature can then be verified later with the public key in order to insure that the signature is authentic. This process provides a very strong method of authenticating a user's identity. A digital certificate server provides a central point of management for multiple public keys. This prevents every user from having to maintain and manage copies of every other user's public cipher key. A Lotus Notes server will act as a digital certificate server, allowing users to sign messages using their private keys. The Notes server will then inform the recipient on delivery whether the Notes server could verify the digital signature.

Digital certificate servers, also known as certificate authorities (CA), provide verification of digital signatures. For example, if Toby receives a digitally signed message from Lynn but does not have a copy of Lynn's public cipher key, Toby can obtain a copy of Lynn's public key from the CA in order to verify that the message is authentic. Also, let's assume that Toby wishes to respond to Lynn's e-mail but wants to encrypt the message in order to protect it from prying eyes. Toby can again obtain a copy of Lynn's public key from the CA so that the message can be encrypted using Lynn's public key.

Certificate servers can even be used to provide single sign-on and access control. Certificates can be mapped to access control lists for files stored on a server in order to restrict access. When a user attempts to access a file, the server verifies that the user's certificate has been granted access. This allows a CA to manage nearly all document security for an organization.

Note: Netscape Certificate Server is a good example of a CA that supports file-level access control.

The largest benefit comes from using a CA which supports X.509, an industry standard format for digital certificates. This allows certificates to be verified and information to be encrypted between organizations. If the predominant method of exchanging information between two domains is e-mail, a CA may be far more cost effective than investing in virtual private networking.

IP Security (IPSEC)

IPSEC is public/private key encryption algorithm which is being spearheaded by Cisco Systems. It is not so much a new specification as a collection of open standards. IPSEC uses a Diffie-Hellman exchange in order to perform authentication and establish session keys. IPSEC also uses a 40-bit DES algorithm in order to encrypt the data stream. IPSEC has been implemented at the session layer, so it does not require direct application support. Use of IPSEC is transparent to the end user.

One of the benefits of IPSEC is that it is very convenient to use. Since Cisco has integrated IPSEC into its router line of products, IPSEC becomes an obvious virtual private network (VPN) solution. While IPSEC is becoming quite popular for remote network access from the Internet, the use of a 40-bit DES algorithm makes it most suited for general business use. Organizations that need to transmit sensitive or financial data over insecure channels may be prudent to look for a different encryption technology.


Kerberos is another authentication solution which is designed to provide a single sign-on to a heterogeneous environment. Kerberos allows mutual authentication and encrypted communication between users and services. Unlike security tokens, however, Kerberos relies on each user to remember and maintain a unique password.

When a user authenticates to the local operating system, a local agent sends an authentication request to the Kerberos server. The server responds by sending the encrypted credentials for the user attempting to authenticate to the system. The local agent then tries to decrypt the credentials using the user-supplied password. If the correct password has been supplied, the user is validated and given authentication tickets, which allow the user to access other Kerberos-authenticated services. The user is also given a set of cipher keys which can be used to encrypt all data sessions.

Once the user is validated, she is not required to authenticate with any Kerberos-aware servers or applications. The tickets issued by the Kerberos server provide the credentials required to access additional network resources. This means that while the user is still required to remember her password, she only needs one password to access all systems on the network to which she has been granted access.

One of the biggest benefits of Kerberos is that it is freely available. The source code can be downloaded and used without cost. There are also many commercial applications, such as IBM's Global Sign-On (GSO) product, which are Kerberos-compatible but sport additional features and improved management. A number of security flaws have been discovered in Kerberos over the years, but most, if not all, have been fixed as of Kerberos V.

Point-to-Point Tunneling Protocol

A discussion on encryption techniques would not be complete without at least mentioning PPTP. Developed by Microsoft, PPTP uses authentication based on the Point-to-Point Protocol (PPP) and encryption based on a Microsoft algorithm. Microsoft has integrated support for PPTP into both NT Server and Windows 95/98.

Many within the cryptography field refer to PPTP as kindergarten crypto. This is due to the relative ease in which people have broken both the authentication mechanism and the encryption algorithm. There are a number of tools available on the Internet which will capture password information within a PPTP session. This is a bit disappointing, considering that PPTP is less than two years old. With so many tools available which will break PPTP, it is of little use as a protocol for protecting your data.

For additional information on the insecurities of PPTP, check out

Remote Access Dial-In User Service (RADIUS)

RADIUS allows multiple remote access devices to share the same authentication database. This provides a central point of management for all remote network access. When a user attempts to connect to a RADIUS client (such as a terminal access server), he is challenged for a logon name and password. The RADIUS client then forwards these credentials to the RADIUS server. If the credentials are valid, the server returns an affirmative reply and the user is granted access to the network. If the credentials do not match, the RADIUS server will reply with a rejection, causing the RADIUS client to drop the user's connection.

RADIUS has been used predominantly for remote modem access to a network. Over the years, it has enjoyed widespread support from such vendors as 3COM, Cisco, and Ascend. RADIUS is also starting to become accepted as a method for authenticating remote users who are attempting to access the local network through a firewall. Support for RADIUS has been added to Check Point's FireWall-1 and Cisco's PIX firewall.

The biggest drawback to using RADIUS for firewall connectivity is that the specification does not include encryption. This means that while RADIUS can perform strong authentication, it has no process for insuring the integrity of your data once the session is established. If you do use RADIUS authentication on the firewall, you will need an additional solution in order to provide encryption.

RSA Encryption

The RSA encryption algorithm was created by Ron Rivest, Adi Shamir, and Leonard Adleman in 1977. RSA is considered the de facto standard in public/private key encryption: it has found its way into products from Microsoft, Apple, Novell, Sun, and even Lotus. As a public/private key scheme, it is also capable of performing authentication.

The fact that RSA is widely used is important when considering interoperability. You cannot authenticate or decrypt a message if you are using a different algorithm from the algorithm used to create it. Sticking with a product which supports RSA helps to insure that you are capable of exchanging information with a large base of users. The large installation base also means that RSA has received its share of scrutiny over the years. This is also an important consideration when you are selecting an algorithm to protect your data.

RSA encryption is owned by RSA Laboratories, which in turn is owned by Security Dynamics. The patent for the RSA algorithm was issued in 1983 and will expire in the year 2000. While RSA Labs still holds control of the patent, the company has been quite generous in the number of institutions to which it has made the technology freely available. RSA Labs has even published source code, which is freely available for noncommercial use.

Secure Shell (SSH)

SSH is a powerful method of performing client authentication and safeguarding multiple service sessions between two systems. Written by a Finnish student named Tatu Ylšnen, SSH has received widespread acceptance within the UNIX world. The protocol has even been ported to Windows and OS/2.

Systems running SSH listen on port 22 for incoming connection requests. When two systems running SSH establish a connection, they validate each other's credentials by performing a digital certificate exchange using RSA. Once the credentials for each system have been validated, triple DES is used to encrypt all information that is exchanged between the two systems. The two hosts will authenticate each other in the course of the communication session and periodically change encryption keys. This helps to insure that brute force or playback attacks are not effective.

SSH is an excellent method of securing protocols which are known to be insecure. For example, Telnet and FTP sessions exchange all authentication information in the clear. SSH can encapsulate these sessions to insure that no clear text information is visible.

Secure Sockets Layer (SSL)

Created by Netscape, SSL provides RSA encryption at the session layer of the OSI model. By encrypting at the session layer, SSL has the ability to be service independent. Although SSL works equally well with FTP, HTTP, and even Telnet, the main use of SSL is in secure Web commerce. Since the RSA encryption is a public/private key encryption, digital certificates are also supported. This allows SSL to authenticate the server and optionally authenticate the client.

Netscape includes SSL in its Web browser and Web server products. Netscape has even provided source code so that SSL can be adapted to other Web server platforms. A Webmaster developing a Web page can flag the page as requiring an SSL connection from all Web browsers. This allows online commerce to be conducted in a relatively secure manner.

Netscape has submitted the SSL protocol to the Internet Engineering Task Force (IETF) so that it may be considered for approval as an Internet standard. The Secure Hypertext Transfer Protocol (HTTPS) is based on SSL.

Security Tokens

Security tokens, also called token cards, are password-generating devices that can be used to access local clients or network services. Physically, a token is a small device with an LCD display that shows the current password and the amount of time left before the password expires. Once the current password expires, a new one is generated. This provides a high level of authentication security, as a compromised password has a very limited life span. Figure 9.7 shows a number of security tokens produced by Security Dynamics Technologies. These tokens are referred to as SecurID cards.

Figure 9.7: SecurID cards from Security Dynamics Technologies

Figure 9.7: SecurID cards from Security Dynamics Technologies

Security tokens do not directly authenticate with an existing operating system or application. An agent is required in order to redirect the logon request to an authentication server. For example, FireWall-1 supports inbound client authentication via SecurID. When a user out on the Internet wishes to access internal services protected by FireWall-1, she uses her SecurID token to authenticate at the firewall. FireWall-1 does not handle this authentication directly; rather, an agent on the firewall forwards the logon request to a SecurID authentication server, known as an ACE/Server. If the credentials are legitimate, validation is returned to the agent via an encrypted session and the user is allowed access to the internal network.

Each security token is identified by a unit ID number. The unit ID number uniquely identifies each security token to the server. The unit ID is also used to modify the algorithm used to generate each password so multiple tokens will not produce the same sequence of passwords. Since passwords expire at regular intervals (usually 60 seconds), the security token needs to be initially synchronized with the authentication server.

There are a number of benefits to this type of authentication. First, users are no longer required to remember their passwords. They simply read the current password from the security token and use this value for authentication. This obviously removes the need to have users change their passwords at regular intervals, because this is done automatically by the security token. Also, it is far less likely that a user will give out his password to another individual because the token is a physical device which needs to be referenced during each authentication attempt. Even if a user does read off his password to another user, the consequences are minimized because the password is only valid for a very short period of time.

Security tokens are an excellent means of providing authentication. Their only drawback is that they do not provide any type of session encryption. They rely on the underlying operating system or application to provide this functionality. For example, this means that authentication information could still be read as clear text if an attacker snoops on a Telnet session. Still, the limited life span of any given password makes this information difficult to capitalize on.

Simple Key Management for Internet Protocols (SKIP)

SKIP is similar to SSL in that it operates at the session level. As with SSL, this gives SKIP the ability to support IP services regardless of whether the services specifically support encryption. This is extremely useful when you have multiple IP services running between two hosts.

What makes SKIP unique is that it requires no prior communication in order to establish or exchange keys on a session-by-session basis. The Diffie-Hellman public/private algorithm is used to generate a shared secret. This shared secret is used to provide IP packet–based encryption and authentication.

While SKIP is extremely efficient at encrypting data, which improves VPN performance, it relies on the long-term protection of this shared secret in order to maintain the integrity of each session. SKIP does not continually generate new key values, as SSH does. This makes SKIP encryption vulnerable if the keys are not protected properly.


In this chapter, you saw why good authentication is important and what kinds of attacks can be launched if you do not use it. You also learned about encryption and the differences between secret and public/private algorithms. Finally, we looked at a number of authentication and encryption options that are currently available.

Now that you understand encryption, it is time to put it to use by creating a virtual private network (VPN). Extranets have become quite popular, and the ability to create a secure VPN has become a strong business need.

About the Author

Chris Brenton is a Certified Novell Engineer (CNE), Microsoft Certified Systems Engineer (MCSE), and Cisco Design Specialist (CDS). As a Technology Specialist for Alpine Computers, he serves as a security consultant to clients and a mentor to engineering staff.

Copyright © 1999 Sybex, Inc.

We at Microsoft Corporation hope that the information in this work is valuable to you. Your use of the information contained in this work, however, is at your sole risk. All information in this work is provided "as -is", without any warranty, whether express or implied, of its accuracy, completeness, fitness for a particular purpose, title or non-infringement, and none of the third-party products or information mentioned in the work are authored, recommended, supported or guaranteed by Microsoft Corporation. Microsoft Corporation shall not be liable for any damages you may sustain by using this information, whether direct, indirect, special, incidental or consequential, even if it has been advised of the possibility of such damages. All prices for products mentioned in this document are subject to change without notice. International rights = English only.

International rights = English only.

Click to order