Identity & Access Management

Extreme VPN Makeover

David Conrad


At a Glance:

  • VPN vulnerabilities
  • Deploying a PKI
  • Authentication options
  • Quarantined connections

Who decided that Virtual Private Networks, or VPNs, were a good idea? With VPNs, we could let armies of mobile workers with "legitimate" business needs connect to our nice corporate

networks—using the same home PCs their teenagers used for gaming and who knows what else. Quick to see the value of VPNs, IT managers immediately proceeded to fund projects, ensuring that legitimate productivity would forever be slowed as workers learned they could check their e-mail and play online poker at the same time.

Fast forward to today and not much has changed. Many organizations still use the basic VPN scenario they deployed years ago, relying on the Point-To-Point Tunneling Protocol (PPTP) and letting all kinds of infected, unpatched, and misconfigured machines connect to their network. In today's environment of ever-more-sophisticated Internet threats, this kind of setup just doesn't cut it. If your VPN solution hasn't evolved to stay ahead of these types of threats, do yourself a favor and give your organization an extreme VPN makeover.

An extreme VPN makeover based on Microsoft technologies will improve your VPN security in two main areas. First, it ensures that your data protection is based on strong IP Security (IPsec), and second, it uses the Microsoft® VPN quarantine service to inspect client machines as they connect, ensuring that clients have the required security patches and antivirus definitions installed before they can access the rest of the corporate network.

In this article, I'll discuss a secure VPN solution built on two key products, Windows Server™ 2003 and Internet Security and Acceleration (ISA) Server 2004. I'll describe how to provision X.509 digital certificates to your VPN users and look at the VPN quarantine service. If you need a step-by-step guide for installing and configuring a secure VPN solution, see the ISA Web site.

Setting Our Goals

Let's say we need to support a mobile community and our users need access to our network not only from their managed PCs (members of the corporate domain), but also from their home PCs and other networks. We also need to make sure no one introduces viruses, malware, or other threats into our corporate network by connecting with unpatched, unprotected machines. How can we accomplish these goals?

First, we want every VPN client coming into our network to use strong IPsec encryption for data protection, requiring an X.509 digital certificate on each client. It's easy to issue certificates to managed workstations, but what about home machines and contractor PCs? We can provision certificates to these machines using the cmgetcer.dll tool from the Windows Server 2003 Resource Kit. The strategy is to let clients connect using the weaker PPTP protocol just long enough to get a digital certificate and then forcibly disconnect them and have them reconnect using the certificate-based IPsec encryption. We'll use a couple of remote access policies defined on a domain controller to manage this behavior.

Next, we need to make sure no one brings the latest virus, worm, or other exploit into the network. For this we'll use the VPN quarantine service, which allows VPN clients to be held in quarantine until they've passed a health check. Administrators can script additional checks, but at a minimum each client should be checked for current antivirus definitions, the latest security patches, and a few other odds and ends, such as ensuring that the Windows® Firewall is enabled. Once a client is deemed healthy, the VPN quarantine service releases the client from quarantine, allowing it to access network resources. Clients that fail the check can be held in quarantine and patched by a server on the quarantine network, then reevaluated. For a general description of the quarantine procedure, see the "The Quarantine Process" sidebar.

Finally, we'll assume that our users have Active Directory accounts and that we'll use usernames and passwords (MSCHAPv2) for our authentication scheme. As we walk through the solution scenario, you should assume that users enter valid Active Directory user name and password credentials for authentication.

To understand the mechanics of this solution, let's dig into its components.

The Quarantine Process by Steve Riley

1 Pre-Connection A pre-tunnel script runs before the remote client initiates the VPN connection. This script checks most of the policy elements and stores the results. If the script fails, it stops the connection attempt and informs the user.

2 Connection If the script succeeds, the remote client establishes a connection to the RAS server.


3 User Authentication The RAS server contacts a RADIUS server to authenticate the user. If authentication fails, the script aborts the connection and informs the user.

4 In Quarantine If authentication succeeds, the RAS server places the remote client in quarantine. Here a post-tunnel script performs additional non-mandatory checks and limits the remote client in two ways: IP filters restrict the computer's communications capabilities to the quarantine subnet only, and a timeout ensures the computer doesn't remain in quarantine forever.

5 Notification The post-tunnel script returns the results of the pre-tunnel script to the RAS server. If the remote client is out of policy compliance or if the timeout expires before the remote client replies with acceptable results, the script drops the connection.

6 Access Granted If the post-tunnel script succeeds, the RAS server removes the IP filter restrictions and allows the remote client full regular access to the network (as specified in the remote access policy).

VPN Protocols

For years network administrators deployed VPN solutions based on PPTP because it was simple. PPTP clients were built into Windows, available for many other platforms, and didn't require any additional infrastructure services. While PPTP provides good security, it lacks advanced data protection features such as establishing a secure connection before authentication takes place, providing data authentication, and protecting against replay attacks. Chances are good that your current VPN solution uses PPTP, and a big part of this extreme VPN makeover is to upgrade to the stronger Layer Two Tunneling Protocol (L2TP).

The Microsoft implementation of L2TP addresses the security shortcomings of PPTP by using IPsec to provide enhanced data protection features, so it might more correctly be called L2TP over IPsec, or L2TP/IPsec. One of IPsec's most significant advantages over PPTP is that the entire user authentication process takes place after an encrypted connection is established between the client and the VPN server. IPsec requires X.509 digital certificates on each client machine, which in turn requires a public key infrastructure (PKI) to deploy the certificates.

PKI Certificate Autoprovisioning

The first step in our VPN makeover is to move the Windows clients to certificate-based L2TP/IPsec to improve data protection. If you have an existing PKI, your clients may already have machine certificates that will work (they must contain the Enhanced Key Usage OID "" which is used for client authentication.) If you don't have a PKI, or you need to extend your solution to home machines or contractors, you can take advantage of the Certificate Services built into Windows Server 2003.

Be aware that deploying any kind of PKI is a serious matter that requires planning. Every implementation should include two legal documents that describe the overall security policies. A Certificate Policy (CP) must describe how users are authenticated before they receive certificates, the certificates' intended uses, and other such information. A Certificate Practice Statement (CPS) should describe how the Certificate Authorities (CAs) in your PKI are maintained and the environmental security precautions you've put in place to protect it. If you're setting up a new PKI just to enable secure VPN, take some time to look at the example CP and CPS documents available on the Microsoft Web site and perhaps adjust them to reflect your organization's policies. A corporate security officer often takes the lead with these documents.

Now back to the technology. Let's assume we don't have an existing PKI and that our only requirements are to deploy certificates to VPN clients. We'll need a Windows Server 2003 Enterprise Edition server for our Certificate Authority (CA) because it's the only edition of Windows that supports Version 2 Certificate Templates. This machine should be a member of our domain because we want to take advantage of the certificate auto-enrollment features available on domain members. We should include IIS and the certificate enrollment Web pages. When we enable the Certificate Services on our server, we need to configure it as an Enterprise Root CA. This publishes the CA information into Active Directory, and it allows us to take advantage of the auto-enrollment features supported by domain member CAs. Once the CA is enabled, we configure it to allow client machine certificates to be automatically issued to authenticated users, thus enabling our provisioning scenario. Finally, we'll deploy a machine certificate to the VPN server, which is going to run ISA Server 2004.

With the CA configured to issue client certificates, we're ready to let our VPN clients request certificates using the cmgetcer.dll tool. The client connects to the VPN server using PPTP, then requests the certificate by running a script. The script and its parameters are specified in the Connection Manager Client VPN software, which we'll get to in a bit. The cmgetcer.dll tool solves two of the toughest problems in PKI deployments. When an authenticated client requests a certificate, not only does cmgetcer.dll take care of enrolling for a certificate, but it also registers the issuing CA and any associated certificate chain into the client's trusted CA store.

The VPN Endpoint

ISA Server 2004 is the VPN endpoint for our solution. Though Windows Server natively includes a VPN server as part of the Routing and Remote Access Services, enterprise customers need the additional security, manageability, and rules provided by the ISA Server 2004 VPN service. ISA Server 2004 is an application-layer firewall (ALF) that provides stateful packet inspection at Layer 7, as well as secure VPN services based on PPTP or L2TP. The ISA Server VPN services may be load balanced across multiple machines and, importantly, may run on servers that are not domain members. Deploying ISA Server on workgroup member servers in the DMZ is an important part of a defense-in-depth strategy, as no services run with domain credentials on the VPN servers.

Configuring ISA as a workgroup member in the DMZ is straightforward. You need to specify which VPN protocols are allowed, in our case both PPTP and L2TP. You'll recall that L2TP is really L2TP/IPsec, so we'll manually request a machine certificate from our CA using the built-in certificate Web pages. With the certificate properly configured on our ISA server, we can select how we want to handle authentication.

RADIUS Authentication

Since the ISA VPN server is not a member of our domain, it can't directly authenti-cate our domain users. As a result, we'll need to configure the ISA VPN server to use Remote Access Dial-In User Service (RADIUS) for authentication.

The RADIUS server we'll use is the Internet Authentication Service (IAS), another service that is built into Windows Server (see Figure 1). By enabling this service on one or more domain controllers on the internal network, our domain controllers can serve as RADIUS servers for RADIUS clients, which in our case is the ISA VPN Server.

Figure 1 Configuring IAS

Figure 1** Configuring IAS **

The traffic traveling between the RADIUS client and server is encrypted based on a shared secret password, so use the same long, complex password on each machine. One of the nice things about this configuration is that your ISA VPN server in the DMZ uses a single port (usually UDP 1812) to send authentication requests into the internal network. There is no need to open numerous ports to allow typical Kerberos-style authentication through the DMZ's internal firewall.

Now, with the IAS service enabled on one or more domain controllers, we need to build a couple of remote access policies. IAS uses these policies to determine whether a user should be granted VPN access. One common strategy is to create a Windows security group called VPN Users and to place any appropriate Active Directory users in that group. Then you can create a remote access policy that allows members of that group to VPN into the network.

Organizations can have many remote access policies defined for many different scenarios. As clients attempt to authenticate via RADIUS, policies are evaluated in the order they are listed on the IAS server. The first matching policy is the policy that's used for that client, and if no policies match, the client is not permitted to connect.

Now let's define our remote access policies. The first policy will indicate that the user should be using the L2TP tunnel type, and it will specify some parameters specific to the VPN quarantine service. The second policy will allow PPTP tunnel types, but will only allow authenticated users to stay connected for one minute, in order to allow the certificate autoprovisioning mechanism we've set up to work. After one minute, IAS will cause the ISA VPN server to disconnect the client, enforcing the policy.

VPN Connections

With these two policies in place, we've set the stage for the behavior we want. Shortly, we'll look at how to configure the client VPN software, but essentially here's what happens. First, clients attempting to VPN into our network will have to use the Connection Manager dialer package we create later. This custom dialer runs the VPN quarantine scripts locally on the client and subsequently reports its health to the VPN quarantine service. Without the custom dialer, clients would be stuck in quarantine indefinitely.

Clients attempting to connect to our VPN server will try to establish an L2TP connection first and then fall back to PPTP if necessary (see Figure 2). This is another configuration choice made in the Connection Manager dialer. The first time a new client tries to connect to the VPN server, it will not have a certificate so L2TP will fail. The client will then transparently attempt a PPTP connection. At the VPN server, we've chosen to allow PPTP traffic, so the ISA server accepts the PPTP connection attempt and uses RADIUS to request authentication for the user. The IAS RADIUS server begins checking its remote access policies. Since the first policy allows connections only with L2TP, the connection type doesn't match and the next policy is evaluated. Since the next policy allows PPTP, and since the user has the proper Active Directory credentials, the connection is permitted, but only for one minute, thanks to our PPTP policy.

Figure 2 The Connection Process

Figure 2** The Connection Process **

It is during this critical one minute of connection that we need our certificate autoprovisioning to work. Connection Manager provides the capability to execute scripts or other commands on the client after a successful connection is made. We'll use this feature to run a batch file that requests an X.509 certificate from our certificate server, which is all we need to do over our PPTP connection. Remember, the certificate server has to be accessible from the VPN quarantine network that ISA defines.

After the PPTP connection is forcibly disconnected, Connection Manager attempts to reconnect. The Connection Manager software is set to try L2TP first, and since we now have a machine certificate, the client will try this connection type. Since our ISA VPN server was configured to allow L2TP connections (and it also has a machine certificate), it will accept this connection attempt and use RADIUS to authenticate the user. The IAS server will check its list of remote access policies, starting at the top. Since the first policy matches our criteria, with a tunnel type of L2TP, and since we've used valid Active Directory credentials, we're allowed to connect.

Now our client is placed on the VPN quarantine network, a special virtual network ISA creates when you configure it to use the VPN quarantine service. A client that successfully authenticates via the remote access policies is automatically placed on this VPN quarantine network until ISA gets the signal that it's safe to move the client out of quarantine and over to the VPN client network, which permits access to some or all of the corporate network. Let's take a closer look at the VPN quarantine service.

VPN Quarantine Service

The VPN quarantine service has a simple mission: place each VPN client on a restricted virtual network, check it for required security patches, antivirus definitions, and any other prerequisites set forth by your security policy, and then allow the client access to the network if it passes the administrator's checks. Administrators may choose to put one or more software update servers on the quarantine network (as defined by ISA) to allow clients that don't pass the checks to download the necessary patches and re-attempt to clear quarantine.

The VPN quarantine service consists of two components. The first component, RQS.exe, is a listener that runs on the ISA VPN server. It's the service that allows ISA to transparently move healthy clients out of quarantine and onto the network. During the installation of RQS.exe, an administrator specifies a text string that represents a healthy status. The second component, RQC.exe, is used by clients to alert the quarantine service that the health checks have passed. Clients automatically run RQC.exe after the client-side scans have completed successfully, passing the required text string to the RQS.exe listener on the server. When the RQS.exe service receives the appropriate text string, the client is released from quarantine.

Many network administrators are surprised to learn that none of the actual scanning for security patches and antivirus definitions is performed by the Microsoft quarantine client, RQC.exe. Instead, you choose how you'd like to scan your clients—batch files, Windows Scripting Host, .NET Framework-based applications, or whatever you like. This flexibility allows customers to design their health check based on their specific requirements and lets them leverage the development skill they're most comfortable with. There's only one requirement: your client-scanning routine must call the RQC.exe client-side executable at the completion of a successful scan, passing the appropriate text string. That's how RQS.exe knows it's allowed to remove the client from quarantine and allow it access to the network.

The last obstacle we need to tackle is how to get our scanning scripts and the RQC.exe client-side component of the quarantine service delivered to our VPN clients. For that, we'll use the Connection Manager Administrator Kit (CMAK), yet another feature built into Windows Server 2003.

Online Resources

The TechNet Online Virtual Private Network Web site offers a host of information and related resources. To learn more about the solution covered in this article, follow the Step-by-Step Guides listed on the site:

Also see the following:

Connection Manager Administrator Kit

The CMAK has been included with Windows Server for years and is used to build customized VPN client connectivity packages. In our scenario, a custom-built Connection Manager dialer is essential because it specifies the appropriate VPN settings for our environment and delivers the key client-side files we use: the cmgetcer.dll file from the resource kit, the batch file to request a machine certificate, the client-side scanning scripts, and last but not least, the RQC.exe file.

Once a Connection Manager package is built using the CMAK, it can be made available to clients in many ways. For managed machines, it can be delivered through various software delivery mechanisms, such as Systems Management Server (SMS). For unmanaged machines, authorized users could retrieve the package from an extranet Web site, or a secure FTP site. Connection Manager packages are Microsoft Installer (MSI) files that use the Windows Installer Service to install.

Wrap Up

With Internet threats growing ever more sophisticated, VPN entry points can pose a significant risk to a corporate network. The steps covered in this extreme VPN makeover scenario help to reduce those risks by establishing strong data encryption and ensuring that vulnerable machines are not permitted to connect to the network.

David Conrad has worked at Microsoft for over nine years in a variety of technical roles. He is currently focused on technology solutions for the federal government in Washington, D.C. David can be reached at

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.