Scaling the Infrastructure Components
Applies To: Windows Server 2008, Windows Server 2008 R2
Since all of the AD RMS infrastructure components participate actively in the licensing process, it is important to consider how each of these components is scaled to support the load imposed on them.
Before scaling your AD RMS infrastructure, you must estimate the maximum load and the average load the servers must handle. Specifically, this is the maximum certification and licensing operations per second, and the average operations per second. Obtaining the average load can be difficult before you implement AD RMS, because the number of users protecting content and opening content during a specific time period can be difficult to estimate. However, you can make an estimate by obtaining the current e-mail volume in your organization and dividing it by the average number of messages that will be protected. This works in most scenarios where e-mail represents the largest fraction of protected content.
Estimating peak load can be difficult as well, especially when you consider intraday fluctuations in the load, and particular onetime events, such as a large volume of protected documents created and opened during a short time period. In this respect, a technique that is relatively easy to apply that leads to good results is to estimate a worst case. For AD RMS, this is typically a situation where a protected e-mail message with a few protected attachments is sent to all the users in the organization. If you are using Exchange pre-licensing, all the messages and documents are licensed within a short time period: typically less than one hour. In a severe worst-case scenario, there could be a few reply messages to the whole audience within the same hour. So, a reasonable worst-case representative for many organizations would include a few protected documents (e-mails and attachments) being licensed to the whole company within one hour. If your company has ten thousand employees, and the scenario described can be considered realistic, you can estimate that the servers must support a load equivalent to approximately a hundred thousand licenses in an hour, or thirty messages per second. Sizing for such an event ensures that your infrastructure can support high-load events without trouble.
For organizations where a large percentage of e-mail messages will be protected, or where a significant fraction of the expected load is not represented by e-mail, but by standalone documents, you can use similar calculation methods to estimate the load.
Performance Considerations for the AD RMS Server
The first role to consider for scalability is that of the AD RMS licensing servers. Licensing servers are directly involved in the issuance of every content publishing or access license, as well as in the creation of the user’s Rights Accounts Certificates (RACs). So the load they have to handle is proportional to the size of the deployment, the number of documents to protect, and their consumption rate.
Since AD RMS is processor intensive, because it uses cryptography when certificates are generated and validated (involving complex calculations), the AD RMS server’s capabilities are, in general, processor bound.
An AD RMS server has close to a linear performance increase, up to four cores, and good scalability up to sixteen cores. Eight cores provide a very good cost/performance ratio.
Since each environment and situation is different, it is not possible to provide generic guidance for sizing AD RMS servers, but Microsoft laboratory tests and production deployments indicate that each core in a typical server CPU can handle 100,000 license requests per hour at maximum load, which may occur in a large organization. However, typically, 50% of maximum load should be targeted in order to provide some room for fluctuations. In this case, 50,000 requests per hour should be the maximum load for each CPU core.
RAM in AD RMS servers is used mainly to cache certificates, to store requests while they are being processed and to manage the queue of outbound requests to SQL Server. As such, increasing the memory in each AD RMS server provides some reduction in CPU load and increases the ability to cope with peak demand. Nevertheless, a typical configuration of 2 GB of RAM is able to provide a significant amount of caching and is considered sufficient for most cases. 4 GB of RAM or more is necessary only for servers with a high number of processors.
Local disk in the AD RMS servers is not used intensively unless you use a local installation of SQL Server or SQL Server express (a configuration not recommended in high load environments). Disk, however, has to be fault tolerant in order to provide the availability desired in most environments. Thus, a two drive RAID array is the minimum recommended for a production server, a configuration that should cover most scenarios well.
Network bandwidth can be a limiting factor for large servers, since each licensing request consumes between 24KB and 30KB, so a server receiving 200,000 license requests per hour (this may occur in a scenario where a protected e-mail is sent to a large organization, and many users attempt to read it at the same time) consumes in excess of 10Mbps on the network. A high-end server can easily tax a 100Mbps link under peak load. Additionally, when complex licenses with many individual rights expressed are issued, each license can grow as large as 1MB. In these cases, a large number of licenses can fill the capacity of a 1 Gbps network. What this means is that even if AD RMS can scale up to a large number of processor cores per computer, the network interface can become a bottleneck for very large systems. Staying near the recommended configuration of four to eight cores per server, and adding more individual servers when necessary, is recommended.
All of this discussion indicates that if multiple servers, each containing four CPU cores, 2 GB of RAM, Gigabit NICs and two sufficiently large hard drives, are deployed to support the AD RMS services in a root certification cluster, a peak of about 200,000 licenses can be issued per hour, per server. Since licensing load is not be uniform thorough the day, in most scenarios, the peak load per hour will probably be much higher than the daily average. This must be taken into account when considering server performance for AD RMS.
If a higher number of licenses is expected during peak hours, adding more CPUs (up to eight cores) or adding more servers are both valid approaches. A minimum of two servers is necessary for high availability purposes, so using more than four cores per server should be necessary only for loads higher than 400,000 licenses per peak hour. Above that, scaling through additional processors is recommended, up to eight cores (for 320,000 maximum license requests per hour in a two server installation). For still higher performance, adding extra servers is the most cost effective measure.
However, you must consider that each environment is different and servers vary in their performance capabilities, so these numbers should be taken only as approximations. You should take a real measurement for each environment before committing to a specific server configuration.
When more than one server in a cluster is used to support additional load, the servers should be load balanced through a network load-balancing solution. While any TCP/IP based load- balancing system should be able to provide the desired load distribution, Windows Server 2008 provides Network Load Balancing Services, a feature that allows for several servers (up to 32) to be configured, forming a load balanced cluster. As discussed earlier, AD RMS is a stateless service, and each request can be processed independently of others, other than those related to the creation and consumption of the same document (however, such requests, in practice, are usually separated by a significant amount of time). So no form of client affinity, that is, the ability to route requests from each client to a specific server in a persistent way, is necessary. NLBS does not take into account the level of load for each server before delivering requests. But since there isn’t a significant variation in the processor load generated by requests; identical servers tend to be uniformly loaded when processing a large number of requests (even without intelligent request routing). Network Load Balancing Services enables the ability to provide asymmetric loads to unevenly distribute the requests, in the case where you have servers of different configurations.
One important consideration, when using a load-balanced cluster, is that NLBS balances requests from different clients. However, for some protocols, successive requests from one client can end up being processed all by a single client. One such case is when IPSec is being used to encrypt traffic between the client and the server. While normally, an individual client isn’t responsible for a significant portion of the traffic, some servers that act as proxy for licensing requests (such as Exchange servers, acting as Hub Transport servers, doing AD RMS pre-licensing) can represent a significant portion of the licensing volume. In such cases, when a small number of systems are generating a large percentage of the requests, it is advised to disable IPSec between those servers and the AD RMS cluster in order to insure adequate balance of the traffic among the different nodes.
In some circumstances, it might be necessary to deploy one or more licensing servers that are not members of the root certification cluster. This is typically done to support departments that require direct control over issuing both publishing and use licenses, such as a legal department with security requirements that necessitate department-level control, or distributed environments with unreliable or costly network links.
Sub-enrolled clusters can issue client licensor certificates (CLCs), publishing licenses for on-line publishing, and end-user licenses (EULs), but they cannot sub enroll other clusters. This implies that an AD RMS hierarchy can only be two levels deep.
For these sub-licensing clusters, similar sizing and scaling considerations apply as for the root certification cluster.
Directory and Naming Services Scaling
It is not the objective of this section to define how a directory services platform should be sized or scaled, but it is necessary to point out that directory services are a critical part of the AD RMS infrastructure. Furthermore, directory services servers can be significantly stressed by AD RMS operations, in some circumstances. One particular operation that introduces stress in a directory services infrastructure is group expansion. If an e-mail message is protected, sent to a large distribution list (for example, one containing ten thousand users), and each user opens the message, a group expansion operation of the DL, in order to determine membership and rights for the user, is performed with each open. The directory services cache in the database server helps to significantly reduce the load imposed on the directory services, but the first expansion operation is performed by the directory servers.
This is a commonly neglected aspect of an AD RMS infrastructure design, and sizing and placement of directory servers must be evaluated when designing a rights management platform.
Scaling the Database Server
The other core component of an AD RMS solution is the database server. A database server in an AD RMS solution provides storage and retrieval capabilities to the AD RMS service in three forms:
It stores the AD RMS cluster configuration, so all the servers in a cluster act as a single licensing unit.
It stores a cache of directory service data used by AD RMS.
It stores the AD RMS service logs.
The database storing the cluster configuration does not normally present scaling problems, as it is accessed infrequently and, even in the most demanding scenarios, rarely produces a significant load on the database server hardware.
The directory services cache can have significant load under specific circumstances, such as when a license is issued that requires expansion of the membership of large distribution lists. Nevertheless, even under those circumstances, it is infrequent that the database has a significant load for a long time period. Since the data cached by the database server rarely represents a significant volume (by typical database standards), even the most demanding operations are normally resolved quickly by the database server.
The service logs, on the other hand, can represent a significant volume of information. While the type of processing done by this service is very light conceptually, consisting of a continuous stream of append operations to the log tables, it can impose a significant load on the database server-disk subsystem.
A typical database server configuration for AD RMS, such as one with four CPU cores and 4 GB of RAM, is sufficient for most environments. Only very high volume environments require a database server with a higher performing configuration. However, the disk subsystem requires more consideration.
Specific information on database volumes, load and contents is included in the next section.