Deep dive into UAG DirectAccess (Certificate Enrollment)
Today I want to talk about how to configure the UAG DirectAccess server security policy to enable certificate enrollment from the Certificates MMC console.
By default, when you try to use the Certificates MMC console for certificate enrollment from the UAG DirectAccess server you will see the RPC server is unavailable message, as seen in figure 1.
This is due to the default security policy on the UAG machine.
To help you solve this problem, I’d like to shed some light about the networking protocols that are initiated behind the scenes when you try to request a certificate using the Certificates MMC console and how you can enable these protocols in the Forefront UAG server security policy so that you can successfully request certificates using the Certificates MMC.
The Networking Protocols used by the Certificates MMC for Certificate Enrollment
When you run the Certificate Enrollment wizard, in most cases you will try to connect to the Active Directory Enrollment Policy. That policy exists in Active Directory, and requires LDAP connectivity from the UAG DirectAccess server to your domain controllers.
After the Wizard connects to the Active Directory, it retrieves the different elements associated with Certificate Enrollment process, including a list of Certificate Authorities, and the templates list used by the Certificate Authorities. Then, after the wizard has this information it will ask you to answer a few questions about the characteristics of the certificate you want to enroll. At the end of the wizard the certificate request is created on the client and sent to the CA. The important thing to note here is that communication between the client and the CA is done using DCOM, and requires DCOM connectivity between the client and the CA.
Networking Policy Changes Required on the Forefront UAG Server to Make the Certificate Request
To enable DCOM and LDAP to the CA and domain controllers, you need to configure Forefront UAG security policy to enable LDAP traffic from the Forefront UAG server to the domain controllers, and DCOM traffic from the Forefront UAG server to the CAs.
Fortunately, the default security policy already allows LDAP connectivity from the Forefront UAG server to the domain controllers. This leaves us with the task of enabling DCOM traffic between the Forefront UAG server and the CA.
DCOM traffic starts by using TCP port 135, which is the RPC endpoint mapper’s port. Later it switches to a different TCP port based on negotiations that took place after connecting to the endpoint mapper. The port number it switches to is given to the DCOM client (the Certificates MMC in this example) during the initial conversation on TCP port 135. This conversation is encrypted by default. Since the traffic is encrypted, we can’t tell in advance what port will be used to enable traffic from the Forefront UAG server to the CA.
Another important point is that by default the Forefront UAG server looks into all conversations on TCP port 135 and when it sees encrypted traffic it blocks the communication altogether (before the two sides have a chance to negotiate the port that would be subsequently used).
Therefore, what we need to do is:
- Enable all traffic from the Forefront UAG Server to the CA
- Tell the Forefront UAG server not to terminate encrypted communications between the Forefront UAG server and the CA on TCP port 135
Enabling Certificate Enrollment from the TMG console on the Forefront UAG Server – Step by Step
The Forefront UAG server includes Forefront TMG server technology. TMG acts as a firewall for the UAG machine. What we want to do is open the TMG management console and enable DCOM traffic and encrypted conversations on TCP port 135.
Important note In general, we recommend that you do not change the TMG configuration (customize TMG firewall rules, create new firewall rules) from within the TMG management console. The exceptions are when you use documented procedures such as this one. The reason for this is that UAG behavior relies on certain elements configured in TMG, and changing them might make Forefront UAG server act in an unpredictable way.
The first thing you need to do is open the TMG management console by starting the Forefront TMG Management application.
Next, left click on the Firewall Policy node and view the list of existing Firewall rules. Then, left click on the first rule named Publishing Rule::Anchor::Begin
The next step is to create a new Access Rule. To do this, right click on the Firewall Policy node in the left pane of the console, point to New and click on Access Rule.
a. On the Welcome to the New Access Rule Wizard page, name the rule Allow all protocols from localhost to CA.
b. On the Rule Action page, configure the rule as an Allow rule.
c. On the Protocols page, configure the rule to allow All outbound traffic.
d. On the Access Rule Sources page, set the source Network as the Local Host Network.
e. On the Access Rule Destination page, create a new Computer Object for your CA and set that CA as the destination for the Access Rule. The Computer Object should contain the IP address of the CA.If there is more than one CA you should create multiple Computer Objects with different IP address and add all of them.
f. On the User Sets page, accept the default, which is All Users.
Next thing you need to do is enable encrypted traffic on TCP port 135. To do that you need to open the System Policy Rules. Right click on Firewall Policy in the left pane of the console, point to All Tasks and click Edit System Policy.
a. On the left side of the System Policy Editor, navigate to the Authentication Services Configuration Group and click on Active Directory.
b. On the General tab, remove the checkmark from the Enforce strict RPC compliance checkbox.
Click the Apply button to save the changes to the TMG configuration. We recommend that you export the configuration before applying the changes
After the changes are applied, it should take only a few moments for the Forefront UAG server to apply the new configuration. Once the new settings are applied you should have no problems requesting or renewing certificates from the Forefront UAG server.
That’s about it for today!
Ben Bernstein, Senior Program Manager, Forefront Edge
Tom Shinder, Technical Writer, Forefront Edge