ADC Design for the Federation

Sometimes, the rules are complicated enough that it's easier just to break them; just ask anyone who's applied for a remodel permit. The same can be said about ADC Deployment in a federated environment. The ADC was designed, for an organization of 77,000 objects to be deployed on 1-2 servers to service the entire organization. However, the ADC requires access to domain controllers and Exchange 5.5 servers in every domain and site in order to carry out its work. When those servers are "protected" by insane firewall rules and challenging bureaucracies, it can actually be easier to deploy 30 instances of the ADC to do the work of 2. 

That was the case with our model. Every possible combination of NT 4.0, 2000, 2003 and Exchange 5.5 deployment had been committed and firewalls allowed only minimal communication between between the agencies' email servers and domain controllers with the outside world (including other agencies in the same forest). Our design, therefore, called for the installation of one ADC Server in every agency's domain, creating a third level of replication/synchronization, as both 5.5 and Active Directory were already replicating in hub and spoke models. The ADC was the axle that held together these two wheels.

The design was simple enough in its complexity. Each ADC would host at least two connection agreements (one for recipients and one for public folders) that would establish synchronization between the local Exchange site and the local child domain in Active Directory. The 5.5 and AD replication models would be responsible for carrying out the changes to the other agencies' 5.5 GAL and global catalog. 

Initially, the federation wanted to deploy the ADC in a very conservative fashion, rolling out a few agencies at a time, as the need dictated. Our testing, however, provided us with enough information to conclude that we would be better off to configure everything in advance and launch the Connection Agreements simultaneously. Because of the varying degrees of configuration and skill-sets, our deployment of the ADCs took roughly 3 weeks to touch every agency. As part of the deployment, we had to centrally validate the following at each department.

  1. Communication (IP) and name resolution (WINS and DNS) was working among the public namespace (.gov), the Active Directory private namespace (.lcl) and all Exchange servers, Active Directory domain controllers, and any required DCs in trusted domains. This was tested with PING and NSLookup.
  2. An account to run the ADC had been created and had the appropriate permissions to Active Directory (in this case, domain admin) and Exchange 5.5 (permissions admin or greater at the org, site and configuration containers). In some cases, this functon was split into three accounts to accommodate bureaucracy (one for the ADC service, one to contact AD and one to contact Exchange 5.5).
  3. Exchange 2003 DomainPrep had been run at the agency and the appropriate groups were created.
  4. The domain in question had been converted to at least Windows 2000 native mode to accommodate universal groups (the ADC would create these from DLs in Exchange 5.5)
  5. Enterprise admins had not been removed from the built-in administrators group on the global catalog server where the ADC would be hosted.

In spite of all these precautions, we uncovered some interesting lessons-learned during the process of installing this tool nearly 30 times.

  1. Windows 2000 domains could not install the ADC using the RunAs feature. The enterprise admin had to log on locally to the machine to perform the install. On Windows 2003 child domains, RunAs worked fine.
  2. The account provided for the ADC was added to the "Exchange Services" group in the user container, and only members of that group could actually launch the ADC utility.
  3. The ADC installation would fail if any of the following conditions were met:
    1. Active Directory Users and Computers was running during the install.
    2. Name resolution to the forest root was invalid (this occurred primarily if the DNS settings on the server did not automatically append the parent domain suffix in attempts to resolve single-label domains. It could also be corrected by adding a WINS entry for the root servers, but this was only done for short term work-around purposes).
    3. Security templates had been applied to the global catalog server where the ADC was being installed.

The individual connection agreements were setup as follows:

  1. All connection agreements were 2-way, primary for both Windows and Exchange (unless multiple recipient CAs were required).
  2. Deletion was disabled on both sides to prevent inadvertent mailbox deletions (many agencies were going to see disabled users mapped to mailboxes)
  3. Schedule was set to Never so that we could turn all CAs on simultaneously using the "Activate_CAs.vbs" script included with the ADC install.
  4. The FROM EXCHANGE tab pointed to the 5.5 site of the agency, included all object types, and the default container in Active Directory was always labeled _ADC {insert whatever} for troubleshooting.
  5. The FROM WINDOWS tab pointed to the child domain and a new recipients container in Exchange 5.5 was created called "ADC {whatever}" for objects that needed to be created.
  6. In the details tab, we had every agency put in the names and contact information for at least two responsible parties for troubleshooting purposes.

The CAs were activated centrally using the activate_cas.vbs script. This script essentially talks to the configuration naming context, finds all CAs configured, and changes the schedule from never to always. The process was done on a Friday night and was 99% complete inside of three hours. As the ADC updated information in Active Directory from Exchange and the child domains replicated with the forest root, nearly 9.5 GB of data transferred in and out of the root. The overall size of the Active Directory database grew by 40%.

However, the design was not without its problems for folks concerned about their neighbors. One of the features of the Active Directory Connector is that a new Security Group, called "Exchange Services" is created in every domain where an ADC is installed. This group, by default, is granted full control to the newly created Exchange 2003 Organization (in the configuration naming context of AD). You can probably see where I am going with this. Any one agency could now take control over the entire Exchange Org by simply adding their user account to this group (in their own domain). Unfortunately, because agencies were unwilling to allow for a centralized rollout of Exchange, this increased risk had to be mitigated through policy (what auditors call detective and corrective controls). 

Although this model works technically, it may not be politically popular in your own federations. As with any Enterprise Application, it must always be assumed that those to whom domain admin access is granted are trusted by all. In reality, most of the folks in our federation trusted each other (on malice issues), but feared most of all incompetence of their peers, which could lead to a situation where one agency could inadvertently gain full control of the messaging system and do something crazy to bring the whole thing down.

Note to self: be persistent when recommending a centralized rollout of the ADC.