Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments
This topic provides the Microsoft support policies for running currently supported versions of Microsoft Exchange Server in production in a hardware virtualization environment. This topic also provides recommendations for running Exchange Server in production in a hardware virtualization environment.
Hardware virtualization software enables you to run multiple, separate operating systems concurrently on a single physical machine. Microsoft has the following software offerings that provide hardware virtualization functionality:
Windows Server 2008 Hyper-V Technology/Microsoft Hyper-V Server Windows Server 2008 provides 64-bit virtualization technology called Hyper-V. Hyper-V is a hypervisor: a layer of software that sits just above the hardware and beneath one or more operating systems. For more information about Hyper-V, see Virtualization and Consolidation with Hyper-V.
Windows Server 2008 R2 Hyper-V Technology/Microsoft Hyper-V Server Windows Server 2008 R2 Hyper-V builds on the architecture and functions of Windows Server 2008 Hyper-V by adding multiple new features that enhance product flexibility. For more information about the key features of and core scenarios for Windows Server 2008 R2 Hyper-V, see Virtualization with Hyper-V: Overview.
Microsoft Virtual Server Virtual Server is software that provides server virtualization technology that was engineered for the Windows Server System platform (Windows Server 2003 and Windows Server 2003 R2). For more information about Microsoft Virtual Server, see the Virtual Server 2005 R2 SP1 Product Overview.
Microsoft Virtual PC Virtual PC is software that lets you create separate virtual machines on your Microsoft Windows desktop, each of which virtualizes the hardware of a physical computer. For more information about Virtual PC, see Microsoft Virtual PC 2007 Product Information.
Third parties also provide hardware virtualization functionality. For details about the Microsoft support policy for third-party hardware virtualization software, see:
For design and sizing information, recommendations, and best practices for running Exchange Server in production in supported non-Microsoft hardware virtualization environments, check with your virtualization software manufacturer.
Terminology Used in this Topic
The following terms are used throughout this topic:
A layer of software that sits just above the hardware and beneath one or more operating systems.
The physical machine that is running the hardware virtualization software. In some hardware virtualization environments, this machine is also referred to as the parent or host machine.
A virtual machine that is running as a child machine of a hardware virtualization environment. The virtual machine typically runs at a second or third level above the hardware in the host machine.
Support Policy and Recommendations for Exchange Server 2007
Microsoft supports Exchange Server 2007 in production on hardware virtualization software only when all the following conditions are true:
The hardware virtualization software is Windows Server 2008 with Hyper-V technology, Microsoft Hyper-V Server, or any third-party hypervisor that has been validated under the Windows Server Virtualization Validation Program.
Deployment of production Exchange servers on Windows Azure virtual machines is not supported.
The Exchange Server guest virtual machine has the following conditions:
It is running Microsoft Exchange Server 2007 with Service Pack 1 (SP1) or a later version.
It is deployed on the Windows Server 2008 64-bit operating system.
It does not have the Unified Messaging server role installed. All Exchange 2007 server roles, except for the Unified Messaging role, are supported in a virtualization environment.
It meets all the requirements that are set forth in the Exchange 2007 System Requirements.
The storage that is used by the Exchange Server guest machine for the storage of Exchange data (for example, mailbox databases or hub transport queues) can be virtual storage of a fixed size (for example, fixed virtual hard drives [VHD] in a Hyper-V environment), SCSI pass-through storage, or Internet SCSI (iSCSI) storage. Pass-through storage is storage that is configured at the host level and that is dedicated to one guest machine. All storage that is used by an Exchange guest machine for the storage of Exchange data must be block-level storage. Exchange 2010 does not support using network attached storage (NAS) volumes. NAS storage that is presented to the guest as block-level storage by using the hypervisor is not supported. Pass-through volumes must be presented as block-level storage to the hardware virtualization software. This is because Exchange 2010 does not support using network attached storage (NAS) volumes. The following virtual disk requirements apply to volumes that are used to store Exchange data.
In a Hyper-V environment, each fixed VHD must be smaller than 2,040 gigabytes (GB). For supported third-party hypervisors, check with the manufacturer to see whether any disk size limitations exist.
Virtual disks that dynamically expand are not supported by Exchange.
Virtual disks that use differencing or delta mechanisms (such as Hyper-V's differencing VHDs or snapshots) are not supported.
Only management software (for example, antivirus software, backup software, virtual machine management software, etc.) can be deployed on the physical root machine. No other server-based applications (for example, Exchange Server, SQL Server, Active Directory, or SAP) should be installed on the root machine. The root machine should be dedicated to running guest virtual machines.
Microsoft supports Exchange cluster continuous replication (CCR) and single copy clusters (SCC) in hardware virtualization environments, provided that the virtualization environment does not include any hypervisor-based clustering or migration solutions (for example, Hyper-V quick migration or VMware ESX vMotion) that are configured to automatically failover or move clustered mailbox servers that are running as guest machines between root servers.
Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it is running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots are not application-aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange Server. As a result, making virtual machine snapshots of an Exchange guest virtual machine is not supported.
Many hardware virtualization products allow you to specify the number of virtual processors that should be allocated to each guest virtual machine. The virtual processors located in the guest virtual machine share a fixed number of logical processors in the physical system. Exchange supports a virtual processor-to-logical processor ratio no greater than 2:1. For example, a dual processor system using quad core processors contains a total of 8 logical processors in the host system. On a system with this configuration, do not allocate more than a total of 16 virtual processors to all guest virtual machines combined.
Performance and Scalability Considerations
Running Exchange 2007 SP1 in a guest virtual machine does not change the Exchange Server design requirements from an application perspective. The Exchange Server guest virtual machine must still be sized appropriately to handle the workload. You take the same approach to sizing a virtualized Exchange server that you would take for sizing a non-virtualized Exchange server. Mailbox, Client Access, and transport server roles must still be designed for performance, capacity, and reliability. In addition, they must be allocated resources that are sufficient to handle the load on the system, based on the usage profiles of the system. For details and guidance about sizing Exchange server roles, see:
Dynamic Memory Allocation Considerations
Many hypervisors include features to dynamically adjust the amount of RAM that is available to one or more virtual machines. This functionality lets the hypervisor allocate RAM to virtual machines based on the current perceived RAM requirements of the particular virtual machines.
Generally, this functionality is appropriate for virtual machine workloads that use a lot of memory for brief periods and then resume typical operations. In this scenario, the hypervisor can allocate memory to meet the needs of the particular workload, and then retrieve the memory for other virtual machines. However, this functionality may not be suitable for workloads that are designed to use a particular memory pool on an ongoing basis.
Many of the performance improvements in recent versions of Exchange are based on the efficient use of an appropriately-sized RAM allocation. This is particularly true of improvements that are related to reductions in I/O operations. The performance optimizations rely on Exchange caching data in RAM. When RAM is dynamically reduced, the expected performance of the system cannot be achieved. In this scenario, Exchange may exhibit reduced performance, or end-users may experience reduced performance when connecting to Exchange. Therefore, for virtual machines that are running Exchange in a production environment, it is best to turn off memory oversubscription or dynamic memory allocation. Instead, configure a static memory size that is based on the appropriate values for Exchange 2007.
For more information memory considerations, see Planning Memory Configurations. For more information about dynamic memory allocation, see the "Application Considerations" section of the Hyper-V team whitepaper, Implementing and Configuring Dynamic Memory.
Virtual Machine Considerations
Following are considerations for Exchange guest virtual machines:
When calculating the total number of allocated virtual processors, you must include the requirements of the root operating system. Many hardware virtualization products (for example, Hyper-V) allocate virtual processors to the root machine's operating system. The number of available virtual processors assigned to the root machine will vary, based on the number of physical processors and cores in the root machine and other configuration settings. In the case of Hyper-V, the number of virtual processors assigned to the root machine will equal the number of physical processor cores in the root machine. This number often represents a value that is larger than is generally required by most Exchange configurations.
When calculating the total number of virtual processors required by the root machine, you must also account for both I/O and operating system requirements. In most cases, the equivalent number of virtual processors required in the root operating system for a system hosting Exchange virtual machines is 2. This value should be used as a baseline for the root operating system virtual processor when calculating the overall ratio of physical cores to virtual processors. If performance monitoring of the root operating system indicates you are consuming more processor utilization than the equivalent of 2 processors, you should reduce the count of virtual processors assigned to guest virtual machines accordingly and verify that the overall virtual processor-to-physical core ratio is no greater than 2:1.
The Exchange server guest machine's storage and network design requires additional considerations for the root machine, specifically, the impact to the CPUs on the root machine. In some hardware virtualization environments (such as Hyper-V), all I/O requests that are made by guest virtual machines are serviced through the root machine. In these environments, we recommend that no other I/O intensive applications (for example, Microsoft SQL Server) be deployed on guest machines that are hosted on the same root machine as Exchange server guest machines.
The addition of a virtualization layer to an Exchange server (where Exchange is running in a guest virtual machine) means that there are additional components that must be monitored for performance and availability. Additional considerations when monitoring Exchange in a virtual environment are:
CPU cycles inside a guest virtual machine occur relative to CPU time slices on the root machine. This behavior causes the values for CPU-related performance counters inside a guest virtual machine to be different from the values reported by the root machine. However, both of the values that are reported by each system are correct because they are based on the system's point-of-view and the way in which processor resources are shared among the root and guest machines. For information about this issue in a Hyper-V environment, see Hyper-V: Clocks lie... which performance counters can you trust?
Exchange-specific performance counters are available only in the guest machine. The root machine only publishes performance data for resources it is using directly and counters specific to the host environment (for example, Hyper-V's performance counters). The root machine does not publish any Exchange-specific performance data.
Guest virtual machines on some hypervisors will exhibit processor core scalability trends that are different from processor scalability trends exhibited by physical machines. It is important that you thoroughly stress test the configured guest virtual machine prior to putting it into production.
Some hypervisors include resource control features that enable you to balance resources among guest machines. For example, in a Hyper-V environment, you can specify a percentage of processor resources that are reserved for each guest machine. This is referred to as the virtual machine reserve. You can also specify a maximum percentage of processor resources that can be used by each guest machine. This is referred to as the virtual machine limit. In addition, you can assign a relative weight to each guest machine to dictate how resources should be allocated when multiple guest machines are running and competing for resources. In most environments, the hypervisor's resource control settings will not need to be modified from their default settings. We recommend that you check with your hypervisor manufacturer for configuration and tuning information.
Supporting large mailboxes (for example, 1 GB and larger) requires the use of cluster continuous replication or hardware-based VSS solutions. Using hardware-based VSS is not possible in a hardware virtualization environment.
High Availability and Disaster Recovery Considerations
Exchange 2007 includes a number of high availability and disaster recovery features, such as local continuous replication (LCR), cluster continuous replication (CCR), standby continuous replication (SCR), and single copy clusters (SCC). All four configurations are supported in a virtualized environment.
Some hardware virtualization software includes features that support the clustering or portability of guest virtual machines across multiple physical root machines. For example, Hyper-V includes a clustered solution called quick migration, which combines Hyper-V host machines with Windows failover clustering. For more details about quick migration, you can download the Quick Migration with Hyper-V White Paper. With the Windows Server 2008 Enterprise and Windows Server 2008 Datacenter operating systems, you can run each server that provides client services as a guest virtual machine on a physical server and configure the physical server as a node in a failover cluster (a group of connected computers that work together to provide redundancy for services). In this configuration, other physical servers in the cluster are ready to support the guest virtual machines when needed through quick migration. The impact of quick migration on a guest machine depends on the nature of the outage:
Scheduled Outages A scheduled outage occurs when the administrator manually moves clustered resources to another node in the cluster. In this scenario, the guest machine state is suspended and saved, the resources are transferred to the specified node, and the guest machine is resumed from the saved state on the specified node. From an Exchange perspective, the server generally only experiences dropped TCP connections. Clients will experience an interruption in service during the migration process: Microsoft Office Outlook users in Cached Exchange Mode, as well as Exchange ActiveSync clients, will briefly be offline, and Outlook users in Online Mode, as well as Office Outlook Web Access, POP3, and IMAP4 users will be unable to access their mailboxes during the migration process. The length of the outage will depend on how long it takes to suspend, move, and resume the virtual machine and is heavily influenced by storage connectivity and the memory size of the virtual machine.
Unscheduled Outages An unscheduled outage occurs when a failure affects an active node to the degree of triggering cluster failover policies. For example, if the active node loses power or experiences a catastrophic software or hardware failure. In this scenario, the guest machine experiences an unexpected power loss. On an Exchange guest machine that hosts the Mailbox server role, the unexpected shutdown will put the databases in a dirty shutdown state. When the Exchange guest is restarted, Exchange will perform its built-in crash recovery processes and replay all log files for all databases. The amount of time it takes for recovery to complete depends on how many log files must be replayed; all log files generated after the checkpoint must be replayed. Generally, you can expect to achieve a replay rate of at least two log files per second.
Choosing a High Availability Solution for a Virtualized Exchange Server
We recommend using the built-in Exchange Server high availability solutions for virtualized Exchange servers instead of hypervisor-provided clustering or portability solutions (such as Hyper-V's quick migration feature). The features found in Exchange Server (in particular, cluster continuous replication (CCR)) provide greater benefits than those found in hypervisor solutions that move virtual machines between physical root machines.
The built-in data replication mechanism in Exchange (continuous replication) can be combined with Windows failover clustering in a configuration known as CCR. CCR can be deployed in a hardware virtualization environment, thereby providing you with a service and data availability solution for Exchange 2007 in a virtualized environment. In a virtualized CCR (or SCC) environment, each guest machine that is a node in the cluster must be hosted on separate physical root machines to provide true redundancy and high availability.
It is supported to deploy CCR or SCC using a combination of physical nodes and virtual nodes. As with all Exchange high availability configurations, you must ensure that all nodes are sized appropriately to handle the full workload during scheduled or unscheduled outages.
We do not recommend using hypervisor-based virtual machine migration (such as Hyper-V's quick migration) for virtualized Exchange servers. In a virtual machine migration configuration, an unscheduled outage can result in data loss. In a CCR environment, this type of data loss is largely mitigated by a feature called transport dumpster. The transport dumpster takes advantage of the redundancy in the environment to reclaim some of the data affected by the failover. For more information, see Cluster Continuous Replication.
The differences between virtual machine migration solutions and CCR (when deployed in a hardware virtualization environment) are listed in the following table:
Virtual machine migration vs. cluster continuous replication
|Virtual machine migration||Cluster continuous replication|
Operating system heartbeat detection
Exchange server heartbeat detection
Copies of Exchange data
Requires shared storage
Supports Exchange-aware backup from passive node
Backup and Restore Considerations
Exchange Server has significant I/O requirements. When deploying large Exchange servers in guest virtual machines, we recommend you use pass-through disks for data storage. The current implementation of VSS in Hyper-V does not support root-based backups for pass-through disks or iSCSI disks that are connected to an iSCSI initiator inside the guest virtual machine. As a result, VSS backups of an Exchange guest virtual machine that are performed within the root machine are not supported for pass-through or iSCSI disks connected within the guest virtual machine.
To perform supported backups of a virtualized Exchange server using either of these storage types, you must perform backups from within the guest virtual machine. You can use backup software that supports the ESE streaming backup APIs or an Exchange-aware software-based VSS solution (for example, Microsoft System Center Data Protection Manager).
VSS backup of an Exchange guest virtual machine from within the root machine is supported when Hyper-V virtual hard disks (VHDs) are used.
Some storage solutions also include storage vendor-supported methods for directly taking hardware-based VSS backups of the storage volumes. Support for these backup methods is provided by the storage vendor and not by Microsoft.
We recommend that you use separate LUNs that are protected using redundant array of independent disks (RAID) arrays for the host operating system, each guest operating system disk, and all virtual machine storage. LUNs for databases and log files should be isolated in accordance with Exchange 2007 storage best practices. All Exchange 2007 storage requirements and best practices apply to Exchange servers running in a hardware virtualization environment. For more information about storage requirements, recommendations, and best practices for Exchange 2007, see Planning Storage Configurations.
The following figure illustrates the storage configuration of Exchange 2007 in a Hyper-V environment.
Storage configuration of Exchange 2007 in a Hyper-V environment
Guest Virtual Machine Storage Requirements
The operating system for an Exchange guest machine must use a disk that is at least 15 GB plus the size of the virtual memory that is allocated to the guest machine. This requirement is necessary to account for the operating system and paging file disk requirements. For example, if the guest machine is allocated 16 GB of memory, the minimum disk space needed for the guest operating system disk is 31 GB.
In addition, it's possible that guest virtual machines may be prevented from directly communicating with fibre channel or SCSI host bus adapters (HBAs) installed in the root machine. In this event, you must configure the adapters in the root machine's operating system and present the LUNs to guest virtual machines as either a virtual disk or as a pass-through disk.
Root Machine Storage Requirements
Each root machine has minimum disk space requirements that must be met:
Root machines in some hardware virtualization applications may require storage space for an operating system and its components. For example, when running Windows Server 2008 with Hyper-V, you will need a minimum of 10 GB to meet the Windows Server 2008 System Requirements for the operating system. Additional storage space will also be required to support the operating system's paging file, management software, and crash recovery (dump) files.
Some hypervisors maintain files on the root machine that are unique to each guest virtual machine. For example, in a Hyper-V environment, a temporary memory storage file (BIN file) is created and maintained for each guest machine. The size of each BIN file is equal to the amount of memory allocated to the guest machine. In addition, other files may also be created and maintained on the host machine for each guest machine.
Exchange Server Storage Requirements and Recommendations
Following are the requirements and recommendations for storage connected to a virtualized Exchange server:
Each Exchange Server guest machine must be allocated sufficient storage space on the root machine for the disk that contains the guest's operating system, any temporary memory storage files in use, and related virtual machine files that are hosted on the host machine. In addition, for each Exchange guest machine, you must also allocate sufficient storage for the message queues on Hub Transport and Edge Transport servers and sufficient storage for the databases and log files on Mailbox servers.
Storage used by Exchange should be hosted in disk spindles that are separate from the storage that is hosting the guest virtual machine's operating system.
Virtual disks may not perform as well as other disk types. You should consult the performance and scalability documentation from your hypervisor manufacturer to understand how Exchange I/O may be affected by the use of different storage options.
We recommend using SCSI pass-through storage to host transport and mailbox databases and transaction log files. Although using pass-through disks limits the portability of virtual machines, this configuration has shown to provide the best performance of all storage options for a virtualized Exchange server.
When using iSCSI storage, the best performance is achieved when the iSCSI Initiator component is configured on the host machine, and the disks are presented to the guest machine as pass-through disks. We recommend using Gigabit Ethernet speeds or faster for the iSCSI storage and isolating the iSCSI storage network from all other traffic. We also recommend using dedicated physical network interface cards for iSCSI network traffic. In a Hyper-V environment, we further recommend that the dedicated physical iSCSI network cards be configured to use jumbo frames and not be bound to any Virtual Network Switch.
Configuring iSCSI storage to use an iSCSI initiator inside an Exchange guest virtual machine is supported. However, there will be reduced performance in this configuration because the network stack inside a virtual machine is not as full-featured as a non-virtualized network stack (for example, a virtual network stack does not support jumbo frames). However, because the iSCSI storage is connected directly to the guest's iSCSI initiator and not configured as pass-through disks, the virtual machine has greater portability.
We recommend specific network configurations when running Exchange 2007 in a hardware virtualization environment. These configurations are based on whether Exchange is deployed for high availability.
For environments that are not deployed for high availability (for example, no CCR, no SCC, no Quick Migration), we recommend that you follow the planning and deployment guidance provided by your hypervisor vendor. For example, in the case of Hyper-V, we recommend that you follow the guidance set forth in the Hyper-V Planning and Deployment Guide and the Microsoft Hyper-V Server 2008 Configuration Guide.
For environments that are deployed for high availability (for example, CCR or SCC), we recommend having at least two physical network interface cards (NICs) in the root machine. One NIC should be dedicated to the hypervisor root machine, and the other NIC should be dedicated for the guest virtual machines. Additional separate NICs should be used for any iSCSI storage that is in use by the root or guest machines.
If you are deploying Client Access or Hub Transport guest machines in Hyper-V that are also configured for high availability by using Network Load Balancing (NLB), then you must install the hotfix from Microsoft Knowledge Base article 953828, "The NLB host does not converge as expected on Windows Server 2008 Hyper-V virtual machines". For more information about how to install and configure NLB, see the Network Load Balancing Deployment Guide.
Support Policy and Recommendations for Exchange Server 2003
Microsoft supports Exchange Server 2003 in production on hardware virtualization software (virtual machines) only when all the following conditions are true:
The hardware virtualization software is Microsoft Virtual Server 2005 R2 or any later version of Microsoft Virtual Server.
The version of Exchange Server that is running on the virtual machine is Microsoft Exchange Server 2003 with Service Pack 2 (SP2) or later.
The Microsoft Virtual Server 2005 R2 Virtual Machine Additions are installed on the guest operating system.
Exchange Server 2003 is configured as a stand-alone server and not as part of a Windows failover cluster.
The SCSI driver that is installed on the guest operating system is the Microsoft Virtual Machine PCI SCSI Controller driver.
The virtual hard disk Undo feature is not enabled for the Exchange virtual machine.
When a Microsoft Virtual Server SCSI adaptor is added to a virtual machine after the Virtual Machine Additions have been installed, the guest operating system detects and installs a generic Adaptec SCSI driver. In this case, the Virtual Machine Additions must be removed and then reinstalled for the correct SCSI driver to be installed on the guest operating system.
To verify the SCSI driver that is installed on the guest operating system, follow these steps:
On the guest operating system, right-click My Computer and then click Manage.
Under System Tools, click Device Manager.
Under SCSI and RAID controllers, verify that Microsoft Virtual Machine PCI SCSI Controller is listed. If you see a different driver listed, you must reinstall the Virtual Machine Additions. For example, if you see Adaptec listed, you must reinstall the Virtual Machine Additions.
If the virtual machine is configured to use an IDE controller only, no action is required.
Performance and Scalability Considerations
When you plan to deploy Exchange Server 2003 SP2 in a virtualized environment, the same performance and scalability aspects that are described in the Exchange Server 2003 Performance and Scalability Guide apply when you size each virtual machine for Exchange Server 2003.
However, some factors directly affect the performance and scalability of Exchange Server 2003 when it is running on Virtual Server 2005 R2. These factors should be considered when you size both the host configurations and the guest configurations.
Following are the factors to consider about the virtual machine configuration:
Each virtual machine can have only one CPU. This limits the processing power of the virtualized Exchange installation. The server should be sized in such a way that a single CPU can handle the estimated load on the server. Also, the number of virtual machines that are running at the same time on the host computer will affect the overall performance of the whole system.
When you size the disk capacity of the virtual machine, the time that is required to perform a full online backup of the Exchange Server data over the network should be considered. Consider adding a dedicated virtual network adaptor for Exchange Server backups.
Although you can perform an offline backup of the virtual hard disk (.VHD) files at the host level, this does not replace the need for performing a regular Exchange Server backup. For more information about Exchange Server backup and restore processes, see the Exchange Server 2003 Disaster Recovery Operations Guide.
Create separate fixed-size virtual disks for Exchange Server databases and log files and store them on separate physical drives on the host server.
Exchange Server performance should be validated before production by using the Exchange Server 2003 Performance Tools. For more information about these tools, see the Exchange Server 2003 Performance and Scalability Guide.
Following are the factors to consider about the host configuration:
Make sure that the server that is running Virtual Server 2005 R2 is sized correctly to handle the number of virtual machines that you plan to deploy. This estimate should include CPU, memory, network adaptors, and disk configuration.
Use a hard disk solution that enables fast access. You can use a SCSI hard disk, a redundant array of independent disks (RAID), or storage area network (SAN) to store the .VHD files that are used by Exchange Server data.
If an antivirus program is installed on the host, the antivirus program should be configured not to scan .VHD files.
Support Policy for Versions of Exchange Server Earlier than Exchange Server 2003
Microsoft does not support any version of Microsoft Exchange Server earlier than Exchange Server 2003 in production in hardware virtualization environments. This policy applies to Exchange 2000 Server, Exchange 2000 Conferencing Server, Exchange Server 5.5, and all earlier versions of Exchange Server.