Application Center 2000 at Microsoft - Managing the Availability of Microsoft's Web Properties

White Paper

Executive Summary

Intranet sites in many Fortune 500 companies have replaced traditional desktop applications for basic tasks such as changing employee personnel data in human resource data systems. At Microsoft, as in many companies today, employees can update their personal data themselves through a web application on the company intranet, rather than relying on human resources department employees to process simple changes. This revolution in empowerment arises from an extensible development and deployment platform for corporate intranet solutions, enabling employees to access, prioritize and act on critical business information and applications.

Because of its primary mission of building great software, Microsoft has many unique needs. However, managing faster platform deployments and improving scalability, reliability, and manageability are common needs among global, Web-enabled enterprises.

Application Center 2000 is designed to reduce the operational complexity of managing multiple Web servers and related applications to the level of operating a single Web server.

This white paper explores the production use of Application Center 2000 in three different scenarios internally at Microsoft. Microsoft's Information technology Group (ITG) is sharing its experiences with customers to illustrate a diverse set of real-world examples of the business benefits of Application Center 2000.

On This Page

Introduction
Envisioning
Planning
Deployment
Operation
Lessons Learned
Conclusion
For Further Information

Introduction

During an extensive series of customer site visits in recent years, Microsoft learned that customers running Internet Information Server as a critical piece of their Web platforms were simultaneously deploying applications on multiple servers and using a load- balancing mechanism to distribute incoming HTTP requests to the cluster servers - a process known as software scaling. At the same time, business units inside Microsoft were doing the same thing.

Microsoft's corporate Internet Web site, https://www.microsoft.com, is one example of a site that uses software scaling. Requests to https://www.microsoft.com are routed to one of more than 150 servers that process the request and return an HTML file to a user's browser. Each server is capable of handling more than 1,000 concurrent users, which means the system's capacity is roughly 150,000 concurrent user requests. Using software scaling, Microsoft has been able to keep up with growing traffic by adding new servers to handle the increasing load as demand drives it. Users will not notice if one server is taken offline for any reason because Network Load Balancing (NLB) and Microsoft® Application Center 2000 are in place.

Software scaling provides two important benefits:

• Scalability — Companies can increase capacity simply by replicating the application onto a new server and to cluster nodes as needed.

• High availability — Because each server in the cluster is running identical applications, individual servers can fail without interrupting service to the users of an application. In the event of a server failure, workload simply shifts to the remaining servers in the cluster.

The benefits of software scaling can be offset, however, by the potentially high operational costs accompanying them.

Before the development of Application Center 2000, moving systems to a software-scaled architecture was a complex, labor-intensive project for most organizations. In addition, managing and monitoring the operations of the clusters was viewed as more difficult than managing individual servers.

Application Center 2000 is designed to simplify on-demand Web site scalability and offer world-class availability, while reducing operational complexity and costs.

The Application Center 2000 development team group focused on three core criteria to enable agility in Web site management:

• Simplified application management – In the past, building, deploying and managing multi-tier applications in clustered environments were exceedingly complex and unapproachable tasks. A fundamental design philosophy at Microsoft is to deliver state-of-the-art technology in a way that it is easier for developers and IT managers to use. The Microsoft .NET Enterprise Servers are designed to help developers and operations to easily build, deploy and manage reliable applications that scale to the Web tier, application tier, and data tier.

• Easier Web scaling - Web server clusters divide the load of an application or system across multiple, inexpensive, off-the-shelf servers instead of addressing scalability with increasingly powerful and expensive hardware.

• Increased application availability - Downtime is inherently reduced with a software scaling approach because most, if not all, single points of failure are removed. Should a server go offline, other servers can dynamically handle the load, enabling the application to continue servicing clients. Furthermore, integrated tools simplify management of all components and services across the platform, reducing management downtime and maximizing availability.

Using clusters with Application Center 2000 is no more difficult than operating a single Web server because cluster-wide configuration efforts are handled from a single console. Every cluster has one server designated as the cluster controller. This server, which can optionally also serve requests in addition to its controller duties, is the source for all cluster synchronization, which includes content, components, and configuration. If the cluster controller is not available, the member servers continue serving HTTP requests. Administrators can dynamically designate any node in the cluster to be the controller as needed.

As of this writing, Microsoft's Information Technology Group (ITG) has nearly completed the process of implementing Application Center 2000 throughout its internal and external Web sites to empower site administrators with more robust management tools for hundreds of clustered Web servers. ITG is taking advantage of Application Center 2000 to reduce the time-consuming, repetitive tasks that Web administrators deal with every day, such as rebuilding software configurations on Web servers, moving applications from a test server and sequentially moving them to a staging server, changing software configurations on multiple production Web servers, creating clusters, and so on.

This white paper describes the stages ITG undertook in implementing Application Center 2000 at Microsoft, from envisioning, to planning, to deployment, to operation. Specifically, this paper explains the activities associated with the deployment of Application Center 2000 on three major Web properties:

• MSWeb, Microsoft's primary intranet portal for its 50,000+ internal employees, contractors, and on-site partners, who use this site as the reliable channel for internal communications and strategic information relevant to Microsoft employees.

• https://www.betaplace.com, (referred to as Betaplace.com for the remainder of this paper) Microsoft's technical Internet beta Web community that was designed as a tool for Microsoft's product development groups and technical beta testers to interact within a consistent and secured environment. Betaplace.com supports roughly 30,000 users a day.

• https://www.microsoft.com (referred to as Microsoft.com for the remainder of this paper), Microsoft's primary corporate Internet Web site, which handles millions of HTTP requests each month.

These Web sites represent three key scenarios that Application Center 2000 addresses:

• A single Web server handling root domain HTTP requests needing improved availability and management.

• A Web site operating with multiple Web servers migrating to Application Center 2000 from an existing solution. The scenario explores complete migration from an in-place cluster load balancing solution.

• An enterprise- sized Web site integrating with, and migrating complex infrastructure tools and processes to, Application Center 2000.

This white paper is intended for Web and Microsoft Windows® operating system administrators, operations and engineering staff, IT architects, and consultants at the enterprise level. It is not intended to serve as a procedural guide. Each enterprise environment presents unique circumstances; therefore, each organization should adapt the plans and "lessons learned" described in this paper to meet its specific needs.

Envisioning

The first stage of this kind of migration process is to develop business goals and determine the scope of the project. Identifying a solution concept, analyzing risk, and determining the project structure are the first things that any team typically does to determine how much effort a project and its related technology require.

Microsoft Operations Framework

Microsoft Operations Framework (MOF) is a collection of best practices, principles, and models, providing comprehensive technical guidance for achieving mission-critical production system reliability, availability, supportability, and manageability on Microsoft's products and technologies. ITG was and is a contributor to the MOF principles. Today ITG uses MOF formally and informally throughout its operations.

By reviewing the three cases—MSWeb, Betaplace.com, and Microsoft.com—the following sections of this paper show how the MOF framework was applied to MSWeb, Betaplace.com, and Microsoft.com in varying degrees, with small and large deployments alike.

The MOF process model describes a life cycle that can be applied to releases of any size, relating to any service solution. The following diagram illustrates the life cycle, including four quadrants and four review activities.

The Changing quadrant introduces new service solutions, technologies, systems, applications, hardware, and processes. In the Operating quadrant, day-to-day tasks are defined to establish an efficient system that can be supported. Problems and resolutions are addressed in Supporting. Optimizing efforts to reduce operating costs, improve performance, capacity, and availability is addressed once daily operations and support mechanisms are in place.

Figure 1: General MOF Process Model

The Team Model

Business units, ITG, and Web operations at Microsoft typically follow the MOF Team Model for Operations, which offers guidelines for IT service management based on a set of consistent quality goals. The guidelines are best practices gathered from the evolution of the Microsoft Solutions Framework, Microsoft Consulting Services' own practices, Microsoft's IT Group, IT Infrastructure Library (ITIL), and Microsoft's business partners. Essentially, the MOF Team Model offers a set of successful organizational frameworks that are tied with processes such as software deployment, development, and other functions surrounding IT operations.

Virtual teaming is a common way of working at Microsoft, and this is especially true of technology deployments. Virtual teams are made up of people from across Microsoft that communicate with each other largely through e-mail and other collaborative technologies, such as SharePoint™ Team Services and SharePoint Portal Server.

Teams organized under the MOF team model are small, multidisciplinary virtual teams in which the members share responsibilities and balance each other's competencies to focus tightly on the project at hand. They share a common project vision, a focus on deploying the project, and ensuring high standards for quality. The members work together as a team of peers, with each member having his or her own defined role or roles, and with each role taking the focus at different points in the process.

The size and depth of the teams vary with the magnitude of the operation. For example, the MSWeb team is essentially composed of one person for each of the strategic and tactical positions. On the other hand, the Microsoft.com team requires several strategic managers and groups of tactical operations members.

Technology Integration and Planning (TIP) is an ITG department with the mission to enable ITG to be Microsoft's first and best customer by leading and coordinating enterprise technology adoption, and by forming strategic partnership alliances with product development groups. TIP drives the strategic collaborative partnership between ITG, business units with IT staff, and the Microsoft product development groups by:

• Establishing IT's strategies on new Microsoft and third-party technologies.

• Interfacing with executive product development groups and sales management.

• Program managing the deployment of enterprise applications (by working with ITG service managers, business unit IT teams, and product development groups).

• Enhancing product quality through expert feedback.

The virtual teams that worked on the deployment of Application Center 2000 in the three Microsoft Web sites used basic MOF principles to address the following areas:

Release: Program Management

In most cases, there are two sets of project managers involved with deployments at Microsoft. ITG's Technology & Integration and Planning (TIP) department works with the product development groups, ITG data center Service Managers, and business unit Program Managers to plan releases based on development phases and business unit project timing.

Operations: Program Management

The program manager is the general project manager who works with operations team members to identify action items for improvement in the product and in the production environment, estimate the impact of such problems on the project, identify how the functionality of the product is affected, and developing and executing resolution plans.

Infrastructure: Infrastructure Engineering

Infrastructure engineers and Operations Analysts provide the knowledge and capability to operate the computing platforms; they also draw application configuration plans based on practical experience from the test and pilot stages of the deployment.

Product Development

At least one team member must be an expert in the internal software design of Application Center. The virtual deployment team included at least one representative of the Application Center 2000 development team to provide expertise in Application Center 2000 internal software design.

Product Support

Early deployments are perfect opportunities to develop product support skills. Microsoft Product Support Service's Engineers work as part of the virtual team, developing intimate knowledge of the product and assisting with debugging. Lastly, the engineer also relays their new expertise to other Application Center 2000 support engineers in a "train the experts" fashion.

ITG services management includes Service Managers, Helpdesk staff, and technicians who manage day-to-day operations of servers in the data center.

Additionally relevant to this paper, Microsoft.com also manages its own site support with a dedicated staff of engineers. These engineers work with ITG and the product development group when new products are introduced.

The roles and their functions in the overall service management life cycle align with the MOF process model by quadrant. The process model quadrants are parallel, not linear, and therefore multiple roles can be and often are involved in each quadrant depending on the team and the system. Each role also can take part in more than one quadrant at the same time if that role is involved in the service management of multiple systems. The following diagram shows how roles aligned with the four quadrants within the MOF process model:

Figure 2: MOF Process and Team Models

ITG Structure

Microsoft's primary business is the design, development, sales, and marketing of computer software. Consequently, ITG has business drivers that are unique among global enterprises. In addition to providing the enterprise IT utility, ITG and IT professionals in business units play a strategic role as some of Microsoft's early technology adopters. Today they test and deploy Microsoft software before its release to customers, a process commonly known within Microsoft as "eating our own dogfood." The product development groups rely on the Microsoft IT and user community as one of the largest sources of product-improvement suggestions before release.

MSWeb Team Structure

MSWeb is Microsoft's enterprise portal focused on general-purpose information that employees need to be effective members of the corporate community. It is an information solution for the global community, providing key industry publications, in-depth information about strategic initiatives, and a search environment to the most authoritative mixed-media sources inside the firewall. With only one server dedicated to the core Web services, the MSWeb team was using fairly mechanical processes to manage its site.

The MSWeb team is made up of one or two people per job duty, as described in the previous section in this paper. Prior to Application Center 2000 deployment, the team maintained one production Web server, two Web crawlers used to search data on other Web sites for user content requests, and Microsoft SQL Server™ 2000 for miscellaneous data driven applications on the site.

The MSWeb team assigns versions to its Web site as if it were a prepackaged software product. For example, at the time of this writing, the site is operating at version 4.5 and is nearing the process of upgrading to version 5.0. The team increments the version number based on feature improvements to the site much like that of prepackaged software. The MSWeb team recognized that using Application Center 2000 to create member cluster nodes would significantly decrease hands-on deployment time, thereby giving more time for solution development as well as expanded service offerings.

To operate and manage the MSWeb site, the team uses Microsoft Windows 2000, IIS, SQL Server 2000, Site Server's Internet crawling search service of Microsoft Site Server, and other (prerelease) software, such as Microsoft OfficeXP and Microsoft SharePoint Portal Server 2001.

MSWeb servers are Intel-based, quad-processor, Pentium 600 MHz CPUs with 1 GB of RAM. The servers use RAID mirroring as described in the system requirements section. As with all ITG Web sites, the amount of site content determines the amount of hard drive usage. As the site grows, so does the hard drive space.

Betaplace Team Structure

The Betaplace.com team houses the servers for the Betaplace.com Web site in an outsourced data center. Prior to Application Center 2000 deployment, the team managed one Web server and one SQL Server 2000 server. Their original plan involved two or three Web servers and a Windows 2000 cluster of two instances of SQL Server 2000 as required by end user demand. The team had been actively researching the market for a product to help them get to their desired infrastructure. A third-party load balancing hardware and software bundled solution was in place in the outsourced data center. They also used NetIQ's AppManager and Freshwater's SiteScope, along with standard Windows 2000 management utilities, to do as much of the troubleshooting and remote administration as possible.

Figure 3: Pre-deployment diagram of betaplace.com

The betaplace.com hardware configurations were quad-processor Intel Pentium III servers with 640 MB of RAM. Hard drives were configured as described in the systems requirements section of this paper.

Microsoft.com Team Structure

Operating and managing the more than 350+ lab, development, and production servers at microsoft.com is a formidable task that requires an extensive staff with specific skills. The staff includes:

• a test team

• a development team

• a program management team

• an operational team

• a content delivery team

• a global business team

• outside contributors who provide additional content

The Microsoft.com Web site operates with fifteen production Web clusters; each cluster comprising seven to ten servers supplying Web content requests. Some clusters respond to executions of Active Server Page (ASP) application executions and other clusters serve static content. The root domain, Microsoft.com, comprises four clusters containing ten nodes per cluster. The https://www.msdn.microsoft.com cluster consists of eight servers. The https://support.microsoft.com/directory/ cluster is a set of eight servers.

More than 300 writers and developers working in more than 51 locations around the world provide information for the site. These content providers are able to update their sites within https://www.microsoft.com around the clock, refreshing five to six percent of the site every day. Each of the more than 110 content servers contains over 420,000 pages of information.

Microsoft.com is one of the largest business Web sites on the Internet today. Traffic ranges around 60 million page views a day, 300 million hits a day, 4.1 million users a day.

The physical architecture behind https://www.microsoft.com may seem surprisingly modest. Thirty-five servers host general Web content; 12 respond to site searches; 16 service download requests along with another 30 in distributed data centers; three host FTP content; and 45 servers host SQL Server data. Additional servers overseas handle some of the international load. They are set to handle a maximum of 6,000 concurrent users and operate comfortably with an average of 1,000 to 1,500 users per server. The processors typically run at between 40 percent to 50 percent utilization.

Servers at microsoft.com are quad-Intel Pentium III processors configured with 512 MB of RAM each. Hard drives are mirrored within each system. As hardware configurations improve, microsoft.com servers follow suit in a timely manner.

Figure 4: Microsoft.com Web infrastructure

The test team is one of the most resource-burdened teams at Microsoft.com. They build, configure, test, and rebuild several servers every day. This nonstop process consumes hundreds of working hours each month. Application Center 2000 eliminates the repetitive configuration tasks in the lab, saving several hours a week. Production Web servers at Microsoft.com face a similar situation, and the Application Center 2000 ability to maintain an application image for test and production servers is proving invaluable.

Application Center 2000 can replicate Internet Information Server metadata and settings or entire software system configurations. The operations team at Microsoft.com viewed this as a subtle but incredible benefit to them for their production servers. Reducing the time it takes to roll updates to Web servers by as little as 10 percent would save several hours, and consequently it would keep more servers online and available to handle HTTP requests. Naturally, more servers online at Microsoft.com means more capacity in a setting where a traffic spike of 1,000–users or more can happen at any time.

Proof-of-Concept

"Proof-of-concept" refers to activities intended to prove that a documented design or other plan will work in production. The goal of each proof-of-concept is to validate that a proposed design will work in a scaled-down test environment that closely matches the software configurations of the production environment. The Application Center 2000 product development team and ITG's Technology Integration and Planning department introduced and demonstrated Application Center 2000 to as many of the departments within Microsoft that operate intranet and Internet sites as possible. Each department developed relative tests for their infrastructure to check deployment plans before deciding on a baseline release plan.

Most business units tested an early beta release of Application Center 2000 in the product development group's test laboratory, with the business units' Web application content and configurations set just as they were on production Web servers. The business units performed the proof-of-concept testing in the lab long enough to verify and confirm that the proposed changes would improve day-to-day operations of the production environment. Site performance and manageability were key metrics for everyone. Proof-of-concept was especially important in this case, because Application Center 2000 documentation was not fully available when the beta deployment began.

The product development group used other proof-of-concept tests on processes that they it would later use to remove third-party solutions from clusters, install Application Center 2000, and remotely manage the arrays by using the Microsoft Management Console (MMC) administration tools included in the product.

MSWeb

Because MSWeb was successfully operating an intranet site using a single server, the proof-of-concept for them was confirming that Application Center 2000 would make it easy to operate two production Web servers and the development server, with the same content on each server at all times. The team was at the point where it needed tool improvements that would ease software change configuration management procedures and help ensure availability. An operations analyst assessed the value of using Application Center 2000 during initial development. The Application Center 2000 development group viewed MSWeb as a quintessential proof-of-concept opportunity, demonstrating that single-server sites would appreciate the ease of increasing site availability with an additional server and Application Center 2000.

Betaplace.com

Betaplace.com had the capacity to support its user base with the hardware already in place. Operating plans for the site originally included clustered Web services to support the thousands of Microsoft beta software testers around the world during their trial periods. In fact, the Betaplace.com team was looking for a solution that would provide hardware fault tolerance with software failover in the advent of system failure. Application Center 2000 was available as prerelease product to Microsoft internal groups. It provided failure management software and the additional capability to synchronize our software across the cluster automatically.

To the Application Center 2000 development group, the betaplace.com site was valuable as a proof-of-concept demonstrating that a complete deployment of Application Center 2000 can replace hardware and software solutions on the market that cost several thousand dollars more.

Microsoft.com

Microsoft's own corporate Web site presented the largest proof-of-concept opportunity. The Microsoft.com operation supports over more than 4 million unique users a day and over 100 servers operating 24 hours a day, every day of the year. Deploying beta versions of software in a live environment such as this truly puts the software "under the microscope" in a real-world setting In fact, the product development group proactively sought out the successful deployment and use of Application Center 2000 in the Microsoft.com operation as part of release-to-manufacturing criteria.

Like many enterprise corporate Web sites operations, Microsoft.com developed its own tools for reporting cluster availability, tools for replication management within NLB clusters, and so on. Deploying Application Center 2000 here would test the product in a tool integration and migration scenario in the most extreme capacity management settings that Microsoft could find prior to public release.

Application Center Technologies at Microsoft

This white paper discusses how features of Application Center 2000 were evaluated, tested, and used on three unique Web sites at Microsoft. Several other Microsoft intranet sites use additional features and tools along with Application Center 2000 —such as Microsoft FrontPage® web site creation and management tool, FrontPage Server Extensions and Component load balancing —that are not detailed in this paper.

The sections that follow describe some of the features of Application Center 2000 that are highlighted in the scenarios discussed. For a complete review of all features of Application Center 2000, refer to the Application Center 2000 Evaluator's Guide. For a comprehensive review of Application Center 2000 functions and technologies, refer to the Application Center 2000 Evaluators Guide and the Application Center 2000 Resource Kit. For URL references for the Evaluator's Guide and the Resource Kit, refer to the For Further Information Section of this paper.

Single Application Image

In the context of Application Center 2000, an application is defined as a collection of software resources. Such an application can consist of Web site content, components, and configuration settings. A system administrator defines which resources constitute an application. This approach allows administrators to manage these sites as logical groupings.

Application Center 2000 provides a central console to manage all elements of Web applications (IIS Web sites, HTML and image files, ASP pages, COM+ components and applications, database connections, security settings, registry settings, and so on). Application Center 2000 provides a central console to manage a collection of these resources for each application, allowing a system administrator to view the application as a single image, even though it spans several servers. Updates and changes to an application can be made from a single console and then automatically synchronized across large server arrays.

Simplified Load Balancing

Load balancing is one aspect of computer clustering. It distributes application component processing load evenly across multiple servers.

Application Center 2000 simplifies the configuration and management of Microsoft's and third-party Web load balancing applications. The essence of Microsoft's Network Load Balancing (NLB) is a mapping of a shared virtual IP address (VIP) to the real IP addresses of the servers that are part of the load balancing scheme. NLB is a network driver interface specification (NDIS) packet filter driver that sits above the network adapter's NDIS driver and below the TCP/IP stack. Each server receives every packet for the VIP, and NLB decides on a packet-by-packet basis which packets should be processed by a given server. If another server processes a packet, the server running NLB discards the packet. If the server determines that the packet should be processed locally, the packet is passed up to the TCP/IP stack.

Fault Tolerance

Session state is a temporary store for user session information such as user preferences, shopping baskets, and other user-specific information. One of the biggest challenges faced by web application developers is maintaining session state or coherency, ensuring that client session information is not lost. The Application Center 2000 Request Forwarder is a mechanism designed to resolve these session state issues when handling HTTP client requests. Request forwarding preserves session state connections with clustered Web servers without major modifications.

Self-monitoring environments isolate point failures of hardware, networks, and applications without interrupting application service. Application Center 2000 provides a self-healing environment that can proactively repair software faults. Application Center 2000 also enables high availability because there is no single point of failure. Every aspect of the Application Center 2000 solution has been designed so that the application will continue serving requests in case of hardware or software failure. If the cluster controller is not available, the cluster is still available. This allows the application to gracefully handle failures, without affecting the customer experience.

Application Staging and Deployment

Application Center 2000 allows administrators to stage and deploy entire applications—including all files, components, and configuration settings—across multiple servers. Application Center 2000 simplifies the process of deploying Web and COM+ applications through the development, test, and production lifecycle. It also allows seamless incremental updates and application upgrades without requiring system down-time. One of its primary functions is to coordinate the deployment of one or more Single Application Images to one or more servers. Because the application has already been defined, administrators can use a wizard to easily define which application(s) to deploy to which server(s).

Application Replication

Application Center 2000 provides a core replication engine and drivers to move applications between cluster nodes. It keeps Web applications synchronized across arrays of multiple servers.

Real-Time Performance and Health Monitoring

Application Center 2000 uses Windows Management Instrumentation (WMI). WMI is the Microsoft implementation of Web-Based Enterprise Management (WBEM), an industry initiative to develop a standard technology for accessing management information in an enterprise environment.

Application Center 2000 provides an integrated view of application performance and health, combining all performance, event, and log data from multiple servers into a common view across a cluster. Default monitors handle such items as Web services, Application Center 2000 services, and system events. Additional monitors can also be created by adding thresholds to existing performance data collectors of performance data.

Thresholds are monitoring rules that evaluate a single property of a data collector. When these rules are met, the state of the data collector changes, such as from normal to critical. For example, a CPU threshold can be defined to enter a critical state if the processor utilization collector is greater than 75 percent for 30 seconds, measured every ten seconds. Using thresholds, administrators can define not only which areas of the system they want to monitor, but also the conditions under which these areas become a cause for concern.

The monitors for a cluster are synchronized configuration settings, so the administrator is assured that the proper monitors are running on each server in the cluster.

Planning

In the planning stage, a draft functional specification and a draft master project plan are created. The functional specification includes information such as design, usability, and deployment goals; the solution design; component specification; project risks; timing of milestones, and project standards. The master project plan includes the approach, dependencies, and assumptions of the project. The master project plan also includes other plans, such as a deployment plan, a pilot plan, a purchasing and facilities plan, a test plan, a capacity plan, a training plan, and a security plan.

Business Assessment

Moving to a new software solution only happens only when a business review indicates the need for an improvement. Each scenario has its own needs and motivators driving it to new systems. This section of the paper reviews the business factors that lead each team to plan an Application Center 2000 deployment.

MSWeb

The MSWeb team usually expects 10,000 users of its Web site each day, with the majority of users accessing the site during Pacific Standard Time business hours. In the past, Web site downtime was scheduled during weekends, when user activity was lowest. MSWeb service failures are noticed almost immediately due to its wide presence inside Microsoft.

Before Application Center 2000 was deployed on the MSWeb servers, operations managers stored original content on a server acting as a staging server for final Web content by using Microsoft Visual SourceSafe™. Next, Visual SourceSafe and a basic script file were used to move content to the Web server. The Windows 2000 Server Resource Kit utility, RoboCopy, was used in instances where large amounts of data needed to be replicated from a development server to the production server. This project was small in scope considering that it involved two servers; however, any negative service-level impact would be noticed company wide.

Microsoft's on-campus data center hosts roughly 1,400 servers. Thousands more servers are in off-campus data centers within a few miles of Microsoft's corporate headquarters. Being involved in an enterprise data center necessitates more operating procedures than smaller IT departments. With thousands of people and millions of dollars in assets involved, ITG needs to ensure accuracy and accountability for its services. To change system settings, clients submit work orders through an ITG intranet site.

The MSWeb team estimated that using Application Center 2000 for remote hosted server management alone would save roughly 40 minutes of work-order ticket generation effort each time the team wanted to change configurations or settings on its server. In a month they could easily save eight hours of effort.

Betaplace.com

The Betaplace.com team was looking for improvements in its cluster management. Its existing third-party solution lacked the ability to distribute COM+ objects, and would not propagate shared files from development servers to the production server or server(s). Before the team could set up a cluster of two Web servers in-place, they needed to have a software tool to manage the continuity of applications and files on a remote Web cluster

Microsoft.com

The microsoft.com team includes an extended staff of in-house developers. These developers had created many tools to manage aspects of the clusters on the microsoft.com site, (including replication from test servers to staging servers to production servers), and to create availability reports. The microsoft.com Web site provides a unique test environment for new Microsoft products because the management and operations staffs are in the same location as the development staff, and because the site handles so much capacity each day.

Planning Considerations

Planning infrastructure changes to a Web platform is a major undertaking. Without involving all functional areas in the planning process, site operators risk project failure and possibly entire site failure. The areas included in deployment planning werer

• Development Preparation

Business unit teams worked with ITG, the Application Center 2000 product development group, and support engineers to develop appropriate laboratory tests, review deployment plans, confirm the support plan, and confirm the escalation procedure during beta deployment.

• Server and Systems Preparation

Generally speaking, Application Center 2000 does not require alterations to the server platform. It can be installed on existing hardware that meets the detailed system requirements without affecting system performance.

Individual business units deploying intranet servers within ITG Data Centers usually select hardware based on ITG's standardized recommendations. The hardware configurations of servers in an Application Center 2000 managed cluster do not need to be exact duplicates of each other when starting the deployment. The servers do, however, need to have similar hard drive configurations. That is, if the first hard drive of the cluster controller is labeled as drive "C:" then all other cluster nodes must have servers with drive "C:" as their first hard drive. All other hard drives also need to maintain labeling continuity as well.

• Operations and Service Management Preparation

Each of the three Microsoft Web sites discussed in this paper had unique operations and service management circumstances. ITG manages the MSWeb servers housed in Microsoft's on-campus data center. Betaplace.com houses its servers in an off-site facility, which is managed by a third-party Internet Data Center. Microsoft.com uses a Microsoft data center near Microsoft headquarters. The operations team for Microsoft.com is on site 24 hours a day. Site Program Managers work from the headquarters campus.

Capacity Planning

Capacity planning is a top IT management concern. The lack of a proactive and continuous capacity planning procedure leads to performance and availability problems.

There is a direct relationship between capacity planning, performance, and stress. A stress test assesses performance. After performance has been assessed, capacity can be planned. Another way to plan capacity is by monitoring the performance of live Web applications and clusters and calculating projections.

The Microsoft Web Application Stress tool, available from the Microsoft.com Web site, provides capacity planning tools that help find the answers to current and projected capacity demands across a cluster. It contains the tools necessary to do modeling and prediction on a wide variety of system components. It also identifies real-time resource constraints by using a large set of heuristics that clearly illustrate the bottlenecked resources.

The pre-deployment testing data from Microsoft.com showed that nodes in the clusters were responding to requests with an average of only 7 MB per second. Clearly this was not a large amount of data; theoretically, one server can handle this amount. However, Microsoft.com has around 40,000 concurrent connections at any given time, and these are predominately slow, long-lasting connections, amounting to approximately 3,700 requests per second. The Web Application Stress tool incorporates a software router to accurately reproduce this behavior so that alternative hardware solutions can be reviewed in a test lab setting.

The groups deploying Application Server 2000 inside Microsoft generally had the available capacity to withstand the minor performance impact of Application Center 2000 replication processes to the cluster nodes and controller without upgrading hardware in response to the deployment during the beta timeframe.

Pilot Test

After meeting with the ITG Release Program Manager and the Application Center 2000 development group, business unit Web site operators replicated data from their production staging servers to servers in the product development group's test lab and ran through Application Center 2000 beta installation steps by using a brief installation guide. Deployment teams from the business units tested routine and non-routine activities with Application Center 2000 in the lab with servers configured similarly to production sites.

MSWeb

The MSWeb team's initial meeting with the product group was in the MSWeb team's test lab. For MSWeb, Application Center 2000 was simple to install, and the management features were so obvious at first look that the proof-of-concept was not only completed but improved upon. MSWeb accelerated planned improvements based on the evaluation. Within three hours of lab testing, Application Center 2000 was a catalyst that MSWeb needed to create a two-node Web cluster that would quickly expand services to include media event calendaring scheduling with a four-node Web cluster.

Betaplace.com

Like MSWeb, the Betaplace.com team testing was fairly straightforward as well. Their testing did not indicate a measurable performance impact on the cluster controller. Testing confirmed that the cluster controller broadcast traffic to member nodes was only 1.5 KB per second. The data center offers at least 100Mb per second of bandwidth on the LAN. It was evident that, even on a beta version that was not optimized for performance, Application Center 2000 was not going to negatively affect system performance. The cluster controller designation is assigned to any server in the Web cluster. It doesn't need to be a separate, non-production server.

Betaplace.com uses Microsoft Passport, which is a single sign-in service that allows the use of a single name and password across any Web site supporting Passport, among several other features. The Passport Manager component contains the server-side decryption code and the cookie and profile access mechanisms that are needed to implement the single sign-in service on a Web site. All Passport-related persistent and session cookies from a user's computer are deleted when the member signs out by clicking the sign-out link or by closing their browser. Betaplace.com would be the first Web site to use Passport along with NLB and Application Center 2000. In a way, it was a minor proof-of-concept test to successfully integrate Passport along with Application Center 2000.

Microsoft.com

The Microsoft.com team had more complexity to evaluate in a lab setting. They also maintain an extensive lab of their own with a dedicated testing staff. Beyond the standard installation and performance testing on their production server content, the Microsoft.com team needed to integrate Application Center 2000 with its own solutions that had been in place for about a year. This took a lot of planning work for the Microsoft.com development team and a site program manager.

Benefit/Cost Analysis

Application Center 2000 return on Investment analysis can be measured in several ways, including; faster Web server deployment, shortened application development cycles, higher application availability, increased site performance and scalability, and reduced management complexity. Software and hardware costs are, generally speaking, the smallest costs involved in deploying software solutions. The most substantial costs come from the personnel costs required to manage the solution after it is in place. In the cases reviewed here, software costs were negligible and not discussed.

Each Microsoft team had unique needs and motivators driving them to test and ultimately to integrate Application Center 2000 into their operations. The following sections frame each scenario and discusses what benefits each team expected once operating their Web sites with Application Center 2000.

MSWeb

For MSWeb, the benefits were obvious. In fact, the incredible efficiencies in day to day maintenance of a Web cluster's availability gained by using Application Center 2000 moved the MSWeb team's business plan forward.

The next project the team planned involved increasing the number of servers to four servers and supporting Microsoft's intranet Windows Streaming Media event calendar. MSWeb site traffic could potentially double by handling the calendar requests for Microsoft Windows Media™ Services. While Windows Media Services servers would still serve the streaming media files; MSWeb would handle the Web requests. The impact and responsibilities could increase by two fold or more, whereas operational management wouldn't need to increase at all. Removing the calendaring requests burden from the Windows streaming media team let the team focus efforts on hosting events —their primary service. MSWeb could manage the workload by using Application Center 2000 with the same resources as in the past.

In this scenario, one cluster of four servers could handle Web requests for MSWeb and the event calendar while the Windows streaming media team presents the event.

This is one of the best examples of how two separate groups serving overlapping business needs with segregated operations can cooperate to reduce efforts and expenses.

Application Center 2000 will significantly simplify business processes. Rather than directing data center operations personnel with common but critical server configuration management tasks like application installation, directory management, and system image alteration requests —each of which take an hour to initiate and confirm completion —an Operations Analyst will be able use Application Center 2000 to execute the procedures using Application Center in as little as 15 minutes.

Betaplace.com

The Betaplace.com team did a basic cost analysis of the amount of deployment time they would gain by using Application Center 2000. Because of the system needs of Passport, if the Betaplace.com team continued to use the existing third-party solution for its load balancing functionality, they would have needed separate COM+ application servers and separate Web servers. It would also have required twice as many Web and COM+ servers to attain the needed fault tolerance and scalability with the pre-existing infrastructure relying on the external load balancing technology.

The key benefit of moving to Application Center 2000 was the ability to avoid the deployment of additional servers in the future. Imaging features in Application Center 2000 make it is quick and easy to delete all software from a server, re-create the entire software configuration, and place the software in an Application Center cluster on the Web. It is it much faster to create a test server exactly as configured on the production server, test newly developed applications, evaluate overall performance and site functionality, delete the contents of a server, and retest.

According to a general survey of vendors, at the time of this writing, outsourced hosting of dedicated Web servers costs between $2,500 and $6,000 each month, per server. The incremental cost of maintaining Application Center 2000 in the outsourced data center is a fractional figure of the total cost of ownership. The process improvement in application testing and publishing should save betaplace.com more than three human resource hours a month in the lab alone.

Microsoft.com

Deploying Application Center 2000 on Microsoft.com became a product release criteria, so the Microsoft.com staff addressed this deployment from a point beyond cost/benefit analysis. However, a basic review of the product easily indicated two areas where the site could immediately benefit.

As with other Web sites, testing is the most resource expensive aspect of Web application deployment. Test teams are continually overburdened with such tasks as set up, testing, re-testing, tear down, and rebuilding. As the Microsoft.com team developed its release plan, it identified the strength of Application Center 2000 application management features and focused on the benefit gains there

Prior to the deployment of Application Center 2000, each month the Microsoft.com team spends approximately one-half hour reviewing events to confirm application and file replication. This was the best way to verify that all cluster nodes were correctly configured with the proper content, considering the number of servers being managed. The team expected Application Center would eliminate this task each month.

The microsoft.com team has made considerable engineering investments in building its own custom scripts and tools to do similar tasks to what Application Center 2000 was designed to address. They team anticipated that they could free up those development resources and in turn, focus on its core daily duties.

Release Plans

A Release Plan is an active electronic document that includes the milestones and hyperlink references to other documents, such as an operations plan, deployment project schedules, any separate test plans, diagrams, installation steps, and so on. The following sections will illustrate the diversity of Release Plans and how they affect future plans.

MSWeb

While the MSWeb was originally developed by using a variation of the Microsoft Solutions Framework and MOF for the Web site as a whole, projects like this (which only lasted a few days from pilot to completion) didn't require extensive documentation. Testing installations procedures, testing application usability, and defining procedural changes in the product development group's lab was sufficient.

There were separate Release Plans for increasing cluster node count and managing calendaring requests. These plans were altered after the successful use of Application Center 2000.

Betaplace.com

The betaplace.com team created a multi-page Release Plan that included detailed installation steps, including cluster creation, verifying IP addressing, installing Passport Manager and keys, application creation and management, and the steps for creating member servers. Installation variables such as cluster name, IP address, and so on were set for test and production servers.

The team used Microsoft Project to create a detailed chart of milestones, action items, and task owners.

Figure 5: betaplace.com post deployment infrastructure

Microsoft.com

The microsoft.com team created a multi-page Release Plan that included links to more than six deployment-related documents and diagrams. The plan divided the project into three phases:

1. Server Management Phase. The test team deployed Application Center 2000 to the initial production cluster, https://www.msdn.microsoft.com. The team deployed NLB, health monitoring, IIS metabase, and COM+ replication. The internally developed application replication tool, PubWiz, continued to move files to the cluster controller and members in this phase.

2. PubWiz Integration Phase. The additional management tools in Application Center 2000 justified the replacement of the existing Web application replication tool based on Microsoft Visual Basic®. The depth of use of PubWiz, however, meant a stepped migration from cluster to cluster.

3. Rollout phase. The test team deployed Application Center 2000 to all clusters under Microsoft.com. This phase also included subsequent upgrades to the beta build upgrades and final release installation responsibilities.

The phased approach was segmented on to specific clusters. As the deployment moved forward, the node count of the cluster increased. The first cluster addressed was the cluster of five servers supporting https://www.msdn.microsoft.com. Next was the cluster supporting https://support.microsoft.com/directory/. The last clusters to be addressed were at the root, https://www.microsoft.com. Because Application Center 2000 is a high-profile Web management tool, deploying onto clusters of the root (https://www.microsoft.com) nodes was a major step toward integrating it into general operating procedures for all administrators at microsoft.com.

Product development group members and product support engineers spent quite a bit of time assisting the Release Program Managers at microsoft.com with planning deployment steps to ensure success. The product support engineers also contributed with debugging in the lab, which led to clean deployments. This is a key way in which product functionality is improved before final release. It is also a way for Application Center 2000 support engineers to become deeply familiar with the product in its final form.

Deployment

During this stage, the MSWeb, betaplace.com, and microsoft.com teams deployed the core technology of Application Center 2000. They deployed site components, stabilized the deployment, and transitioned the project to normal operations and support staff (if different from the project staff). After the deployment, the teams conducted project reviews and sevaluated project milestones against planned results.

MSWeb

The deployment of Application Center 2000 on MSWeb production servers proceeded according to plan. The main Operations Analyst estimates that it took three hours to install Application Center 2000 on the original MSWeb server and on the new server. The original server was designated the cluster controller and the new server was a node of the newly formed NLB cluster. The installation occurred on a weekend to minimize any possible service impacts to site users.

Interestingly, Application Center 2000 exposed some IP infrastructure inconsistencies that had never been noticed before. MSWeb servers were sending IP packets across unnecessary routes based on the subnet and gateway settings of the NICs. The round-about traffic pattern problem was resolved and node communication resumed as designed.

Betaplace.com

The plan for deploying of Application Center 2000 on the betaplace.com Web site was to complete the installation without any downtime to the live site. A server was taken offline, and the beta version of Application Center 2000 beta was installed on one node while another node actively responded to HTTP requests. To manage this, the team switched its hardware/software IP controller to point to the one live node for the duration of the set up and configuration period. After the setup and configuration period was complete, the server was reconnected to the network, the IP controller was directed to the server with Application Server 2000 installed on it and the beta version of Application Center 2000 beta was installed on the server that was formerly handling the site requests.

Microsoft.com

While trying to simulate operations in the lab, issues surfaced that could only be discovered by installing Application Center 2000 in a complex infrastructure such as Microsoft.com. The beta version of Application Center had prerequisites and chronological installation steps that do not exist in the final release. The Microsoft.com team crafted a site Operations Guide to include Application Center 2000 into the infrastructure to reduce the impact of using beta software on production Web servers. The guide covers the following steps:

1. Synchronize node port rules

2. Confirm host ID's

3. Install Windows Service Pack files

4. Install Web applications

System Requirements

The basic system requirements for the deployment of Application Center 2000 are:

• Intel Pentium-compatible 400 MHz or higher processor.

• Microsoft Windows 2000 Server or Windows 2000 Advanced Server operating system.

• Microsoft Windows 2000 Service Pack 1 or later.

• 256 MB of RAM minimum recommended.

• 100 MB of available hard- disk space to install services; additional space required for site content and databases.

• CD-ROM drive.

• One Network Interface Card (two recommended); if using Windows 2000 Network Load Balancing, two Network Interface Card's are required.

• Windows 2000–-compatible video graphics adapter with 800x600 minimum resolution.

• Microsoft Mouse or compatible pointing device.

• Microsoft Internet Information Services 5.0 must be installed as part of Windows 2000 installation.

ITG builds standard server configurations based on server role, for example, the standard hard rive configuration for all Web servers, SQL Server application files, and SQL Server log files, is to implement RAID1. ITG uses RAID10 (0+1) for SQL Server database files. RAID5 is not a logical choice for Web servers because since each server maintains its own data on a local RAID stripe set, so load balancing among cluster nodes is the primary redundancy solution.

As mentioned previously, ITG supports and handles backup tapes as part of data center operations. As part of standard procedure, these were created prior to installing Application Center 2000 on production servers.

Operation

Operation addresses the people, process, technology, and management issues pertaining to complex, distributed IT environments. This section highlights the repeatable processes and procedures used to achieve mission-critical system reliability, availability, supportability, and manageability of Microsoft products and technologies.

Procedural Changes

Day-to-day operations should improve with the introduction of new Web management tools for service managers and Web operators. Service managers should expect improved reporting and stability; while Web operators should expect improved site management. As new operational procedures become standard, the teams will start to spend more time optimizing the system. The following section highlights the expected improvements that each team has experienced or expects once operating its Web sites with Application Center 2000.

MSWeb

Prior to Application Center 2000 deployment on the MSWeb Web site, the operations team managed one Web server and then managed the replication of files checked-in to Microsoft Visual SourceSafe files. After replication between servers was complete, an analyst took approximately 30 minutes a day to compare the production server's file time stamps with the staging server's file time stamps to confirm accuracy.

With Application Center 2000, MSWeb replication is managed from a remote desktop computer running the Application Center 2000 client software. Application Center 2000 uses Windows 2000 Event Logs to check for failed procedures. Automated alerting has reduced manual tasks. Application Center 2000 alerts Operations Analysts to specific issues and provides a report of what Application Center 2000 did to resolve the issues.

Betaplace.com

After Application Center 2000 was installed and clusters were fully operating, the pre-existing firewall and load balancing system was relegated to firewall duty for the cluster. Cluster management was handled via familiar Windows -based management tools such as the Microsoft Management Console MMC snap-ins.

Using Application Center 2000, the Betaplace.com team started aggregating all Web server availability performance and event data from cluster nodes into a single report representing the entire cluster. Additionally, the team configured the health monitoring tools with performance thresholds. Healthmon is used to diagnose issues as they arise.

Microsoft.com

Microsoft.com performed two availability measurements: Virtual IP (VIP) availability, which is the overall availability of a site such as microsoft.com or support.microsoft.com; and key node availability, which is the overall availability of the nodes on a VIP.

Application Center 2000 can report on availability of individual nodes in a cluster and the cluster as a whole. The root domain, https://www.microsoft.com, has four clusters—which also means four Application Center 2000 cluster controllers—on which administrators need to view and aggregate four separate cluster Healthmon reports. The advantage that Application Center 2000 offers in reporting is that it can create availability reports for the individual nodes in the cluster. This may help the microsoft.com team identify which nodes and which of the four clusters, if any, cost more to operate month over month.

Administration

Changes in administration techniques are driven by new Web and cluster management tools. This section covers the significant process changes that each team made because of Application Center 2000.

MSWeb

For MSWeb, avoiding downtime is so critical that past maintenance administration was done on weekends. During the business week, maintaining availability is the first priority. The simplified cluster configuration wizard in Application Center 2000 made the move to a two-node site approachable and manageable. Along with that, the speed of the application replication tool and monitoring tools in Healthmon gave the MSWeb team the resources to handle the high-availability demands of its clients. The team is using the Single Application Image feature to quickly build and re-build wholly configured cluster members that are ready for production. It speeds testing, which is one of the most resource -intensive aspects of running a valuable Web site. In the case of the MSWeb team, it freed the resources necessary to take on additional services for the Microsoft community.

The team estimates that it has shaved 80 percent of its day-to-day menial tasks with Application Center 2000 in place. By using the Application Center Microsoft Management Console MMC snap-in, they immediately eliminated the need for several scripts they had used to remotely manage content replication from staging servers to Web servers. Previously, an Operations Analyst managed site replication from staging servers to the Web server by using Microsoft Visual SourceSafe. The analyst confirmed content accuracy after replication by manually verifying file time stamps. With Application Center in place, if replication fails, a Windows 2000 Event Log entry is made that triggers an e-mail alert notification to operators.

Since the deployment of Application Center 2000, the MSWeb team has been able to maintain 99.8 percent of its planned site availability. Much of the benefit is derived from operating the two servers in a cluster versus using a single production Web server to support all Web requests. Application Center cut unnecessary downtime by making daily tasks easier.

One issue the team noticed during the pilot phase was that when Application Center 2000 took a server offline for various normal reasons, another Web management application attempted to restart the Web service. In this case, the solution was simple; suspending the Web service during the planned incident.

The MSWeb team also uses Healthmon in Application Center 2000 to display CPU performance histories in a report delivered to MSWeb managers. The report, which takes less time to generate than previously used reporting tools, shows performance and hardware utilization. This information lead the team to take on additional services, such as the internal Windows Media Services event Web -based calendaring services.

Betaplace.com

The Betaplace.com team's main priority was to put in place a stable and easy-to-manage cluster of Web servers. They weren't necessarily looking for additional Web management tools, but they found a valuable reporting feature in Application Center 2000 that they truly value beyond the original goal. With Healthmon preconfigured to actively monitor common performance issues the team can precisely locate problems as they arise.

Because the servers are hosted in a remote facility by a third party, there were occasions when Web site Program Managers weren't aware of issues with the Web site until after the 3rd party operations staff resolved the issues. It was difficult to analyze what aspects of Web operations were costing the team time and effort.

The Betaplace.com team continues to use other management tools such as Freshwater Software's SiteScope to monitor website availability, Webtrends for site historical statistics and Windows Terminal Services to remotely connect to and manage Windows 2000. Cluster management, application replication, and managing the site application images for lab and production use are all handled with Application Center 2000 tools.

Microsoft.com

Although the microsoft.com team has developed and uses some very good tools to synchronize IIS settings and content, Application Center 2000 does a great job of synchronizing registry keys and .dll registrations in addition to replicating content. On a site with as many application directories and settings as microsoft.com, the management of registry settings and application files is impossible without automating software. In fact, the microsoft.com team found that replicating small items and whole applications alike with Application Center 2000 is an incredible time saver that was originally underestimated. New Web servers are set up every day in the microsoft.com lab. Using Application Center 2000 to configure base IIS settings and the IIS metabase across all nodes in a cluster exponentially simplifies the effort based on the number of nodes.

Lessons Learned

In the early stages of Application Center 2000 deployment, the product development group, ITG, and the three teams discussed in this white paper encountered challenges that had to be addressed. Microsoft is sharing the following lessons learned so that customers can benefit from ITG's enterprise experiences.

Ensure proper network configuration

Application Center 2000 does not function properly when cluster nodes are in different TCP/IP subnet masks. Ensure that cluster node network interface cards are properly sending packets to other nodes in a cluster. During deployments, there were cases where nodes were on the same TCP/IP subnet but were going through different hardware switches that prevented RPC communication. In addition, two teams noticed that the TCP/IP subnet mask setting was different for the dedicated IP and the virtual IP on nodes within the same cluster. Although this was unrelated to the installation of Application Center 2000, in these cases, Application Center 2000 identified an infrastructure deployment issue that had been previously overlooked. Use Network Monitor to verify that packets are moving as desired.

Be aware that the installation of Application Center 2000 via Terminal Services will be interrupted as Application Center 2000 resets the NIC and causes the session to be disconnected.

Carefully Test Interoperability

When pilot testing, evaluate the effects of third-party or internally developed solutions, such as load balancing software and Web management tools, when run in conjunction with Application Center 2000. An example of this scenario involves the availability solution that the betaplace.com team previously used. During set-up, Application Center 2000 takes a server offline for parts of the cluster configuration. The availability management tools that betaplace.com used were designed to attempt to bring offline servers back online as soon as the management tool identified specific events related to availability. After identifying the cause on the node, the team simply suspended the service instead of stopping Internet Information Services.

Devise an anti-virus solution for Web servers without involving anti-virus software applications on production Web servers. Most anti-virus applications take additional time and CPU power to scan incoming files and compare them to an internal database of known files. This can become a bottleneck in the system performance, considering the amount of data that Application Center 2000 can replicate from controller to nodes. Pre-production Web servers may be appropriate servers to use anti-virus software.

Confirm Authentication

The Application Center 2000 cluster controller will replicate SSL certificates to the cluster nodes if the Web site is using SSL certificates. Prior to deployment, confirm that the certificate is exportable and verify that the name bound to the certificate is valid on all the nodes involved. Consider binding the certificate to the shared DNS name of the cluster.

Re-examine Fault Tolerance

Although Application Center 2000 Request Forwarder can handle HTTP client session state, it cannot make all Web client session state applications completely fault tolerant. To make them completely fault tolerant, a persistent method of maintaining state must be implemented. Be aware that some implementations of Web client session state are not fault tolerant at all. If the original Web server is no longer available to receive a forwarded request from another node in the cluster, state information can be lost.

Make the Most of New Reporting

Use Healthmon to measure the value of Application Center 2000 alerting and reporting information. Application Center 2000 can assist in the operations and administration reporting aspects of Web site management. Use Healthmon overall system availability reports to conduct a cost-benefit analysis of individual nodes compared with other nodes.

Conclusion

Application Center 2000 has been responsible for maintaining the availability of several Web sites in production inside Microsoft since before release to manufacturing. It reduces the duration of what are normally time-consuming tasks, which makes time for development testing. Thorough testing ensures more solid solutions. Application Center 2000 reduces and sometimes eliminates custom work, individual configuration settings, and scripts. Custom work usually escalates the amount of time and cost of a software solution.

Application Center 2000 gives Web deployment teams the ability to perform warm run-throughs of deployments on test platforms alongside the actual production servers. Now, developers can run trial deployments of production scenarios in the lab very quickly, by imaging actual production servers. The team can re-image a server and re-create an exact replica cluster member within minutes. Testers can repeatedly deploy software application updates on a server configured exactly as it will be on the production site in a fraction of the time it used to take without Application Center 2000.

All of the teams discussed in this white paper see bigger potential for the future. As they start using this tool in cross-team projects, it will have a positive impact on operations. Application Center 2000 will supply significant agility once it is fully used in standard operating procedures.

Application Center 2000 is reinforcing the .NET strategy of managing faster platform deployments. .NET Enterprise Servers are designed with mission-critical performance in mind. It provides scalability, reliability, and manageability for the global, Web-enabled enterprise, and are built for interoperability using open Web standards.

Application Center 2000, as well as the other .NET Enterprise Servers, is providing Microsoft with many benefits, including:

• Software scaling — Server clusters divide the load of an application or system across multiple, inexpensive, off-the-shelf PC servers instead of addressing scalability with more powerful and expensive hardware.

• Availability — Downtime is inherently reduced with a software scaling approach because most, if not all, single points of failure are removed. Should a server go offline, other servers can handle the load dynamically, enabling the application to continue servicing clients. Furthermore, integrated tools simplify management of all components and services across the platform, reducing management downtime and maximizing availability.

• Fast time to market— Through deep integration of all of the products and services making up the .NET Enterprise Servers, and with world-class development tools and support, developers need only focus on implementing business logic when building an application. The overhead associated with creating custom system services and infrastructure has been alleviated by including all of these technologies as standard components of the platform. This, along with a fundamental commitment to ease of use, allows developers to develop and deploy solutions more quickly than on any other platform.

• Ease of deployment, administration, and management— In the past, building multi-tier applications and deploying and managing those applications in clustered environments were exceedingly complex and unapproachable The .NET Enterprise Servers are designed to make it easy for developers to build reliable applications that scale out on the Web tier, application tier, and data tier — and are inherently more manageable than applications on any previous platform.

• Full exploitation of Windows 2000 Server— Windows 2000 Server includes an integrated, enterprise-class application server and provides an infrastructure that leverages state-of-the art hardware technologies, such as Storage Area Networks (SANs), large memory, and SMP architectures with up to 32 processors. The .NET Enterprise Servers build on this platform and use these services and capabilities for specific functions, such as data management and XML support.

For Further Information

To view additional IT Showcase material, please visit: https://www.microsoft.com/technet/itsolutions/msit/default.mspx

To view the Application Center 2000 page on Microsoft's Web site, please visit: https://www.microsoft.com/applicationcenter/

For Application Center 2000 Resource Kit chapter examples, please visit: https://www.microsoft.com//technet/prodtechnol/windows2000serv/reskit/deploy/dgbl_exc_elhg.asp

To review the MOF Executive Overview, please visit:

https://www.microsoft.com/mof/

For any questions, comments, or suggestions on about this document, or to obtain additional information about Microsoft IT Showcase, please send e-mail to: showcase@microsoft.com