Chapter 12 — Deploying Phase
Published: June 21, 2004 | Updated: October 29, 2004
On This Page
Introduction and Goals
Team Focus During the Deploying Phase
Completing Deployment Preparations
Deploying the Solution
Post Deployment Acceptance Test
Transferring Ownership to Operations
Closing the Deploying Phase
Introduction and Goals
The goal of the Deploying Phase is to ensure that all changes related to this migration project are deployed successfully into the production environment in the least disruptive manner possible. The rollout should be conducted as a series of activities. Although you can abbreviate or combine some of these activities, you cannot skip any of them completely without increasing the risks to the project's success. These activities are:
Complete migration preparations
Deploy the solution
Perform final acceptance testing
Transfer solution ownership to Operations
Nearly all of these activities are performed in conjunction with the Operations team. Close coordination and agreement between the two teams is now a critical success factor; strive to make all decisions by consensus to minimize risk.
Team Focus During the Deploying Phase
Table 12.1 is repeated from the UMPG to help project team members quickly identify which material in the Deploying Phase chapter they need to read. The primary team role driving this phase is Release Management.
Table 12.1: Role Cluster Focuses and Responsibilities in the Deploying Phase
Focus and Responsibility
Customer feedback, assessment, and sign-off
Solution/scope comparison; stabilization management
Problem resolution; escalation support
Training; training schedule management
Performance testing; problem identification, definition, resolution, and reporting
Site deployment management; change approval
Completing Deployment Preparations
The preparations for deployment started in the early phases of the migration project, and now they need to be completed. Note that these activities should be included as part of the migration schedule because the migration itself cannot begin until the activities are complete.
Source control of scripts and applications is an essential part of the migration process. Ideally, you should take a snapshot of the source code, library versions, and versions of any software you use to create the SQL Server environment so that you can recreate the exact installation at any time in the future. This information is useful following a fallback.
You can use tools such as Microsoft® Visual SourceSafe® to conduct source control activities.
If you are conducting a phased migration, you may have different versions of the source (scripts, application source code, and so forth) for the different phases. These should be captured as successive validation points and have snapshots taken accordingly.
You should implement mechanisms to log any issues that are uncovered during the deployment, to track their triage and resolution, and to enable status reporting. You can build an issues-tracking database to support this.
The fault management process should be as flexible as possible so that it can be kept up-to-date in a production crisis. The worst-case situation is when, during such a crisis, people do not keep track of the changes that they make to the system or do not give adequate time for the most recent change to take effect. The situation can rapidly deteriorate as more people become involved in the attempt to solve the problem.
All server administrators should be highly trained in resolving critical production issues. When a crisis occurs, priority should be given to the established process, which provides structure and control. Only highly-experienced administrators should be actively involved with this type of change.
For more information about tools for fault management and reporting, see Chapter 8, "Stabilizing Phase."
The validation of every phase of a migration is essential for its success. The absence of checks and balances could lead to difficulties in performing a deployment or executing the fallback strategy.
Of particular importance in a database deployment are:
Schema validation. This ensures the consistency of the database schema predeployment and post-deployment.
Security validation. This ensures that the level of security that you want has been implemented in the new system.
Service validation. This ensures that the service levels in the deployed application meet the needs of the users.
Schema validation helps ensure that the database source moved to production is actually the system that you want. The following items should be performed to validate the schema before the execution of the cutover process:
Confirmation of database scripts from source control. Each script that is modified in development should have a unique source control ID. This source control identification should be validated before migration.
Database objects count. Ensure that the number of objects that were promoted in development is identical to what will be promoted in production.
Server comparison. Compare the following to the development/staging server environment:
Windows Server 2003 environment setting. Compare the server configuration settings to make sure the performance obtained in development matches what will be promoted to production after deployment.
SQL Server 2000 configurations. Compare SQL Server configurations in the development and staging environments to ensure that it matches what will be promoted to production after deployment.
Test script execution. Execute the same scripts that were used in development to validate that the result set obtained in development is what is obtained in deployment.
Validation of all logins should be required to ensure that the level of security that you want has been implemented in the new migrated environment. To properly check that the migrated logins have the appropriate level of access, and that logins that you do not want do not have access, the following methods can be used:
Scripts. Create a series of test scripts to execute against the deployed database to check the level of access for each login.
Application testing. Using user acceptance testing, set a criteria for successful login authentication for all users.
If an additional level of security is required for the migrated database that is similar to that which the existing Sybase system provided, you can implement the SQL Server Security add-in.
For more information about SQL Server 2000 security, see the Microsoft SQL Server 2000 SP3 Security Features and Best Practices white paper at http://www.microsoft.com/sql/techinfo/administration/2000/security/securityWP.asp and the "Best Practices for Enterprise Security" article at http://www.microsoft.com/technet/archive/security/bestprac/bpent/bpentsec.mspx.
Although the migrated database application may perform adequately in the pilot and test environments, you should repeat tests on the migrated application to ensure that the service levels are maintained.
In particular, you should consider running the following test types:
Performance testing. Run test scripts to verify the database response times in the migrated configuration.
Scalability testing. Load-test the new implementation to determine how it will sustain multiple users and transactions.
Although these tests might make up part of the user acceptance testing, you should be prepared to run them before user acceptance testing commences. The deployment may be described as a failure even if the actual migration efforts are successful because of real-world performance characteristics of the migrated components or system. Testing the performance of the migrated database and applications against what is to be expected during formal acceptance testing is as important as the actual migration itself. Plan ahead and work with the team on these issues until each of them is fully resolved.
Deploying the Solution
For maximum efficiency, ensure that your change and deployment process is modular, which means that you should be able to restart the entire process, stop at any point and then continue, and identify exactly which changes have been made and which have not. You may decide to create individual scripts for each logical group of changes and step through them. By segregating the various tasks into discrete units, you can achieve greater control over deployment, more accurately measure your progress, and more appropriately designate budget distributions. These units will modularize your progress through the Deploying Phase. Modularization helps with troubleshooting and redirection if choices made during the planning effort need adjustment.
A good way to keep track of what you have done is to put auditing code in all of your implementation scripts, recording success or failure in a log. To make this process quicker, you can create a tool that executes the scripts in sequence (for example, you can create a table of scripts to run in SQL Server and loop through them in a script that uses xp_cmdshell procedure calls to run them through the isql utility). Alternatively, you can generate one large implementation script and include code that ends the implementation if certain errors occur. However, such a script is difficult to restart unless you also insert code for detecting where the previous run ended. If you choose this approach, think about the amount of auditing code you would need to include and then plan what you would need to do to find the error.
Build Core Deployment Package
Building the core deployment package is concerned with building the releases that are required for migrating and consolidating to SQL Server 2000.This process includes the following considerations:
Select the deployment mechanism. The following list describes some possible options:
Use an automated process where possible (for example, through use of scripted installation and configuration).
Use manual processes for some installation and configuration. These must have detailed, step-by-step instructions.
Consider using Microsoft Application Center to manage and distribute the application across the enterprise.
Create the deployment package, including:
SQL Server installation and configuration scripts.
User database migration scripts.
Database administration tasks and packages, such as maintenance plan, jobs, and backup and restore procedures.
Windows domain user accounts configuration scripts.
Scripts for configuring ODBC data sources, ADO, and OLE DB connection string.
Scripts for configuring COM+ components for database connection.
Acceptance test scripts and packages.
Create rollback scripts to remove changes introduced by the deployment. This is optional, but a good practice. It makes problems easier to fix during deployment, and it makes it easier to restore the original configuration during disaster recovery.
Add the deployment package to the configuration management database (CMDB). Maintain change control of the release. Where a CMDB does not exist, a simplified one should be created for release-tracking. The purpose of updating the CMDB is to ensure that accurate knowledge is stored regarding all of the components or configuration items that make up the releases.
Important If you must deviate from these scripts in the production environment, be sure to alter the scripts to reflect the change and enter a note in the script or documentation regarding what was altered, why, and whether you noticed any issues afterward.
Interim Milestone: Core Technology Components Deployed
At this stage, components such as the database servers, their supporting hardware, and communications infrastructure are deployed. They constitute the framework or backbone for the entire solution.
To avoid delays, core components may be reviewed and approved for deployment in advance of other parts of the solution that are still being stabilized. Operations staff must feel confident when making this commitment before the whole solution is proved to be stable.
Site Rollout Preparation
After the core technology has been deployed, you can plan the site rollout. This step is concerned with preparing for the actual rollout of the releases.
Rollout planning includes the following considerations:
Assemble the required resources:
Assemble the rollout team.
Assemble rollout documentation.
Conduct prerollout training and briefings.
Conduct a dry run through the deployment process. This should include all personnel who are connected with the release. Checklists are valuable for summarizing the steps and to ensure that any chance of missing a step is minimized.
Communicate the rollout plans:
Set communication lead time — provide a "drop-dead date" in the schedule. All resources must be available and ready by this agreed date or a new rollout window must be selected.
Inform help desk.
Inform the configuration management team.
Inform the Change Management team.
Inform all other related areas of service management, as well as functions such as facilities management.
Train support staff and administrators:
Ensure that the customer support staff is trained to support the deployment of the release and ongoing operations.
Ensure that onsite support during deployment is available, possibly from third-party suppliers, including Microsoft. This should include ongoing support contracts.
Prepare the production environment:
Back up current production database servers and user databases.
Back up necessary portion of current production environment.
Prepare for contingency:
- Always have a contingency plan for an emergency fix or disaster recovery. Think ahead about what could go wrong during your implementation. This risk analysis is vital to the success of your process and may change your deployment plan.
This phase of the process is concerned with the actual rollout of new servers and databases into the production environment.
User training should be straightforward for the majority of migration projects because users should already be aware of their job roles and how the existing database application supports their day-to-day activities. Ensure that users are consulted about their training needs and that a training schedule is devised to help users understand any new functionality or modes of operation and to inform and assure them about the effect of the migration on their working practices.
Rollout includes the following tasks:
Select the deployment group and assemble the personnel.
Conduct user training where this has not been completed already. For a consolidation project that does not affect the desktop, there should be no need for additional user training.
Determine the deployment cutover window. Negotiate a deployment and switchover window that is acceptable to all groups that are affected by the change. For a mission-critical system, consider providing backup databases for alternate access during shutdown of production server and databases.
Deploy the release into the production environment. Execute the deployment and consolidation plans. Maintain the configuration database through the release period.
Cut over to the new production database and applications. Back up the old production database and system.
Document any unplanned procedures and changes. Sometimes, despite all your plans, something unforeseen occurs during a production implementation. Regardless of the severity of the problem, you should make a note of it in your implementation script or documentation. Record everything, including what code you had to run and what action you had to perform.
Implement any emergency fixes. If you need to apply emergency fixes, be sure to conduct a risk analysis, at least informally, before proceeding. At no time should you introduce a high-risk change to an unstable environment without careful testing. Also, be sure to immediately document all the changes that you made. Document the urgency of the situation, what led you to your decisions, and which changes you believe were most effective.
A good practice during a production crisis is to try to make only one change at a time and record observations before and after the change. However, crisis situations sometimes require drastic action. If you do not have a failover server prepared, you may find yourself making multiple untested changes based on previous experience and expertise.
When the environment is stable again, document any remaining issues and enter the event into the incident management system. It is important that you prevent any further change from being made to the affected server until the configuration is verified against previous configuration documentation.
For more information about incident management, see the Incident Management Operations Guide at http://www.microsoft.com/technet/itsolutions/techguide/msm/smf/smfincmg.mspx.
Interim Milestone: Site Deployment Complete
At the completion of this milestone, all targeted users can access the solution. Each site owner has signed off that his or her site is operating, though perhaps with some issues. The training may not have gone well, or a part of the solution may have malfunctioned after the deployment team departed the site.
At this point, the deployment team makes a concentrated effort to finish deployment activities and close out the project.
Stabilizing the Deployment
After the cutover process has been completed, you must validate the stability of the environment. After all of the steps of the deployment have been performed and validated, consider having users access the new production environment to enter standard production queries using the new production toolsets to validate that the functionality of the migrated application and database environment is performing as expected.
Following the acceptance period, schedule a quiet period, during which no enhancements to the functionality of the database or software are made. This is to ensure that time is available for spotting bugs that may have crept in during the deployment process and to accommodate their subsequent resolution.
Post Deployment Acceptance Test
When the cutover is complete, you should validate the correct operation of the system on the production hardware. Frequently, this involves running at least a subset of the user acceptance tests again. However, time may be limited at this point, and the tests therefore cut back to a minimum set to ensure that it is basically operational.
After the transfer of ownership to operations, the system should be carefully monitored for some period, and end users of the system should be interviewed to ensure that it is operating correctly.
Transferring Ownership to Operations
To ensure the capabilities of the operations group, you may need to perform a readiness assessment. The MSF Readiness Management discipline incorporates an assessment step that covers:
Measuring existing knowledge, skills, and abilities.
Determining any knowledge gaps between existing skills and those that are required to support the new systems.
Creating learning plans to provide the knowledge that bridges these gaps.
The output of this assessment will be a gap analysis that describes what needs to be learned and a learning plan to define how to bring operations staff up to speed with the deployed database applications. For more information, see the MSF white paper MSF Readiness Management Discipline v.1.1 at http://www.microsoft.com/technet/itsolutions/msf/default.mspx.
In many larger projects, and some smaller projects, the team that will ultimately support production is not the same team that created the architecture and rolled out the installation. It is important to ensure that there is a clear handoff of responsibility from one group to the other. To sustain a project’s success, it is the responsibility of Release Management to ensure that the team who will handle the day-to-day support fully understands the technology and can operate, maintain, and support the new production environment.
The following information and documentation must be provided:
Hardware specification sheets and documentation
System installation and configuration guide
Diagram of network and database server topology
Software manuals and documentation
Support policies and procedures
Known issues and resolutions
System administrator’s account and password
Additional system documentation may also be necessary, including:
System architecture and database overview
System hardware: what is in there and why?
System software: what SQL servers and databases are installed? Where? What changes have been applied to user applications?
Operation and maintenance
Common failure scenarios and recovery procedures
Shutdown and restart process
Before the operations team will accept ownership, the transfer may require a post-implementation review, during which the operations team can determine whether the deployed database application is ready for the team to take responsibility. In particular, the operations team representative should, at a minimum:
Ensure that all required operations documentation, including backups, monitoring, maintenance, procedures, contacts, and escalation paths exist and are complete, adequate, and reasonable.
Ensure that the release met site standards for messaging operations, database operations, network administration, monitoring/metrics, and availability management.
Ensure operations staffing is in place and adequate (that is, there is sufficient staffing levels and skills) for the release, both for the initial release and an ongoing basis.
Ensure the reasonableness of service level agreements (SLAs) relative to the operations function, and ensure that any automated infrastructure support mechanisms work, are sufficient to meet SLAs, and have thresholds set correctly.
Ensure that a test of backup integrity (both system and database) is planned.
Ensure that failover and recovery, availability management, and security measures are adequate for meeting SLAs for the particular release.
Ensure that a clear definition exists of who will be responsible for resolving post-engagement questions regarding the transition.
You should conduct a walk-though inspection of the operating environment involving members of both the deployment team and the operations team. For more information about the post-implementation review, see "Release Readiness Review — Microsoft Operations Framework," at http://www.microsoft.com/technet/itsolutions/cits/mo/mof/rlsrdy10.mspx.
Interim Milestone: Deployment Stable
It can be difficult to determine when a deployment is "complete." The deployment team can find it difficult to close out the project because of the ongoing issues that surface after deployment. For this reason, the team needs to clearly define a completion milestone for the deployment instead of attempt to reach a point of absolute finality.
The definition of this milestone is dependent on reaching an agreed understanding of "stability" between the deployment team, the operations team, and the user representatives.
Closing the Deploying Phase
Following the successful completion of user acceptance testing and handover to operations, the Deploying Phase is nearly complete. Although there may be many other pressing activities that demand the attention of the deployment team, it is important that you finish the project in a way that ensures that you tie all loose ends and you learn lessons for the future.
Conducting a Closedown Review
After the deployment, a closedown review meeting should be held to evaluate the success of the entire project, not just the deployment. All aspects of the project should be examined, including the project plan, the design, development and testing, the deployment process, and the final outcome. As input to the review, you should compile a project review document that provides a summary of the project for upper management, as well as a solid base for ongoing operation and support. The project review document should include the following sections:
Schedule and execution:
Project objectives versus achievements
Project schedule versus actual summary
Project budget versus actual summary
High-level summary of changes performed
List of accomplishments
List of issues and resolutions
Benefits and results:
Performance: databases, applications
Ease of use and management: server, database, application, network
Ease of support, less maintenance
Financials: hardware, software, cost of operations, productivity
User experience and customer satisfaction
Key members of the project team should attend the review itself, which offers an opportunity to learn and document from successes achieved, issues overcome, and mistakes made during the deployment process. The following topics should be included:
Quality and ease of communication (or lack thereof) during the implementation.
Issues that arose during implementation, including, for each issue:
Whether it was predicted during risk analysis
Priority and severity
Situation at time of occurrence
Valid resolution alternatives
Action taken (explain reasons, if necessary)
Preferred action, in retrospect (if different from action taken)
Did every part of the implementation succeed?
Was a post-implementation test of functionality performed?
What work remains to bring the implementation to successful completion?
What work remains to reinstate administrative and support excellence?
Has the configuration management system been updated and all change control forms filed?
Document the minutes of the review and append them to the project review document.
The Quiet Period
The period in between the deployment stable milestone and deployment complete is sometimes referred to as a quiet period. Although the deployment team is no longer active, team resources will respond to issues that are escalated to them. Typical quiet periods are 15 to 30 days in length.
The purpose of the quiet period is to measure how well the solution is working in ordinary operation and to establish a baseline for understanding how much maintenance will be required to run the solution. Organizations using MOF will measure the number of incidents and downtime and will collect the performance metrics of the solution. This data will help form the assumptions used by the operations service level agreement(SLA) on expected yearly levels of service and performance. For more information about SLAs, see the MOF Operations Guide at http://www.microsoft.com/technet/itsolutions/cits/mo/mof/mofpm.mspx.
Only following the quiet period should you consider decommissioning the old database system, and this should take place only after you have made a full, verified backup.
Key Milestone: Deployment Complete
This milestone is based on a signoff by user representatives that they are satisfied with the deployed system and by operations managers that they are satisfied sufficiently to take on the operational management of the deployed system without the direct support of the deployment team. Clearly, the exact terms of this agreement will depend on the migration and your organization's acceptance requirements.