Inside SharePointMaintaining Security Account Credentials
Code download available at: ChernyInsideSharePoint2009_02.exe(413 KB)
Password Changes in a SharePoint Environment
Troubleshooting Farm Credential Issues
Frequently changing farm credentials and security account passwords are important measures for keeping a Microsoft Office SharePoint Server (MOSS) farm secure. Yet for administrators, these tasks are tedious and repetitive. The tools are difficult to use, the underlying processes are complex, good documentation is hard to find, and there is a certain risk of corrupting the server farm even when following all of the proper procedures.
Adding to the mix is problematic technical advice by Microsoft Product Support Services (PSS). On the one hand, there are useful Knowledge Base (KB) articles, such as "How to Change Service Accounts and Service Account Passwords in SharePoint Server 2007 and in Windows SharePoint Services 3.0." And on the other hand, there are disasters, such as "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption.'"
The above article states that you must create a new configuration database if the account credentials of a Web app can't be decrypted anymore. This can happen if you use a critical command switch at the wrong time or if the process of updating farm credentials completes unsuccessfully for any reason. This KB article did a fine job of instilling fear in admins that changing farm credentials can corrupt a server farm irreparably. Who can expect customers to maintain security in their SharePoint environments by frequently changing farm and security account passwords if they face that kind of risk?
Customers don't wipe out Active Directory to reset user accounts, and they should not be forced to wipe out their SharePoint configuration databases either just because a Web application password was lost. It isn't necessary to create a new configuration database. In most cases, it is sufficient to use the standard Stsadm.exe tool to overwrite corrupted credentials. And in all other cases, you can reset the passwords and retain your configuration database by using the tools included in the companion material, available in the February 2009 Code Download. Resetting passwords in Windows SharePoint Services (WSS) 3.0 and MOSS 2007 environments doesn't require access to old credential keys nor a new configuration database. But it does require better support tools from Microsoft and better documentation of the underlying processes that facilitate password changes.
This is the first column of a two-part series in which I review the standard procedures and tools for maintaining SharePoint security accounts, discuss their limitations, highlight the risks, and suggest an alternative approach to achieve better security, reduce administrative overhead, and therefore lower total cost of ownership (TCO) in SharePoint environments. This first part deals with the architectural details and the complicated process of accomplishing password changes.
It is vital to understand how SharePoint deals with security accounts and passwords whether you maintain security manually using scripts or by using a fully automated solution, such as the one I will introduce in next month's column. To keep my explanations realistic and practice oriented, I once again rely on a test environment. In this first part, a straightforward single-server installation is sufficient, as outlined in the companion worksheets. As always, my worksheets and companion tools are for test labs only and are not to be used in production environments. Use my tools at your own risk.
Password Changes in a SharePoint Environment
Changing the password of a SharePoint security account or the farm's configuration account is a remarkably complex undertaking. Among other things, you must apply changes in Active Directory and in the SharePoint server's local Security Account Manager (SAM) database, in the Service Control Manager (SCM) database, in the IIS metabase, in SQL Server, and, of course, in SharePoint content and configuration databases.
You also must replicate the changes to all other servers in the farm to apply the changes consistently. You might have to re-encrypt all the passwords of all SharePoint services and application pools for all servers in the farm if your changes affect the farm's credential key, which is the encryption key that SharePoint uses to protect the passwords of security accounts in the configuration database. When you change the farm's config account password, you implicitly change the farm's credential key. Figure 1 shows the process in a server farm with two front-end servers. Frankly, it is impressive that this process works as well as it does.
Figure 1 Changing the password of a SharePoint security account
All security account password changes begin in Active Directory, and from that moment forward, until you update the affected password in SharePoint, the farm is in an inconsistent state. The password is now outdated in IIS and elsewhere, but SharePoint is still operational.
This is good news for organizations with high-availability requirements. Password changes do not require system restarts. Windows services and IIS can continue to use the security token they obtained by logging on the affected security account with the old password when you last started the server. But don't restart IIS or the entire server during the transitional phase of password inconsistency.
IIS and other services would not be able to log on with the old password during the restart, and the affected application pool or service would not be able to come online anymore. In this situation, IIS writes a warning into the server's event log (see Figure 2). So don't wait too long with the password update in SharePoint following the password change in Active Directory.
Figure 2 IIS warns about outdated application pool credentials
In order to update the password of an application pool account, you should use SharePoint 3.0 Central Administration (_admin/FarmCredentialManagement.aspx) or the following Stsadm.exe command:
stsadm -o updateaccountpassword -userlogin <DOMAIN\USER> -password <PASSWORD> -noadmin
In response, SharePoint encrypts the new password by using the farm's credential key and overwrites the old encrypted password in the configuration database. Next, SharePoint updates the account information in the IIS metabase and in all other necessary places. Once this is accomplished, SharePoint generates a timer job of type SPContentAppPoolCredentialDeploymentJobDefinition to deploy the new credentials on the remaining servers in the farm, which it places into the configuration database.
As seen in Figure 1, SharePoint relies on timer jobs to apply administrative settings globally on all servers in the farm. The SharePoint Timer services on the remaining servers in the farm pick up the job and accordingly update the local security settings on their servers with the help of the WSS Administration (SPAdmin) service to bring the farm back into a consistent state.
This is the process for application pool accounts, but there are many other types of SharePoint services that use security accounts, such as the SharePoint Timer service itself, the WSS Help Search service, and possibly Shared Services Providers (SSPs), Office SharePoint Server Search service, and single sign-on (SSO) service. There may be even more services, depending upon the solutions installed in your farm, and each service type has different password update requirements. The article at support.microsoft.com/kb/934838 contains a list of commands that you must use for the service accounts in WSS 3.0 and MOSS 2007. Check the product documentation of any additional solutions that you use for further tools, commands, and update procedures.
This highly diversified and uncoordinated landscape of tools and commands is one of the downsides of the current SharePoint security architecture. It doesn't scale well and can drive up TCO depending upon the number of custom solutions in your server farm that use security credentials. In the second part of this series, I will show you how to bring this under control so you can use a single solution to accomplish all the necessary updates regardless of the underlying service types.
The most important update scenario concerns the farm credentials. The password of the farm account is special because it affects the farm's credential key, which is used to encrypt all passwords in the farm, as mentioned earlier. So, following a change of the farm account's password in Active Directory, you must update SharePoint using the command:
stsadm -o updatefarmcredentials -userlogin <DOMAIN\USER> -password <PASSWORD>
SharePoint then must re-encrypt all existing (encrypted) passwords in the configuration database, it must update the SharePoint Timer service account (which uses the farm account as its identity), and it must propagate these changes again to all servers in the farm by means of a timer job of type SPAdminAppPoolCredentialDeploymentJobDefinition.
Many things can go wrong during this phase. The timer job might get stuck in the queue, as illustrated in Figure 3, or the update process might fail unexpectedly, due perhaps to a sudden power outage, leaving behind old encrypted passwords that SharePoint now cannot decrypt anymore because the credential key has changed.
Figure 3 A credential deployment job stuck in the queue because the SharePoint Timer service is not running on all servers in the farm
In another scenario, you might update the passwords of application pool accounts before the new farm credentials have reached all servers in the farm, causing servers with the outdated credential key to fail because they cannot decrypt the updated password in the configuration database yet, as illustrated in Figure 4. This is an interesting scenario and the reason for the -noadmin switch in the updateaccountpassword command. If the farm account is also used as an application pool account (not recommended), then you should update the farm credentials first, wait until all servers in the farm have processed the timer job, and then update the application pools.
Figure 4 Hold application pool updates until the Administration Application Pool Credential Deployment job has been processed
Correspondingly, the updateaccountpassword command checks whether the specified security account is the farm account and informs you about the dependencies if this is the case without performing the update. By using the -noadmin switch, you disable this check and apply the changed password to the account in the application pool configuration, but it is difficult to automate these procedures in a script with appropriate lag time.
Troubleshooting Farm Credential Issues
Now let's take a closer look at the updatefarmcredentials command. It comes with a dangerous switch that can cause great pain, especially if it's used as described in the article "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption”. I'm referring to the switch called -local. The SharePoint developers introduced this switch for performing local farm credential updates manually. The idea is that you can delete a timer job from the queue in Central Administration (_admin/ServiceJobDefinitions.aspx) if the job is corrupted or not processed for any other reason and then perform the necessary update step directly by using the command:
stsadm -o updatefarmcredentials -userlogin <DOMAIN\USER> -password <PASSWORD> -local
The -local switch instructs the updatefarmcredentials command to apply the password change only on the local computer. It is important to recognize, however, that this update affects the credential key and the SharePoint Timer service, but not the application pools, Search service, SSPs, and so forth. It is assumed that you already ran the updatefarmcredentials command without the -local switch on another server in the farm and thus re-encrypted all the passwords in the configuration database. It isn't necessary to perform this re-encryption step again. But what if you used the -local switch without taking this step?
Using the -local switch without first running the updatefarmcredentials command without this switch causes trouble because the -local switch changes the credential key. The application pool passwords are encrypted with the old key in the config database, but this key is now overwritten.
Take a look at Figure 5. You cannot run the updatefarmcredentials command without the -local switch anymore because this requires re-encryption of passwords that can no longer be decrypted. When the command fails, you can find entries in the Application event log stating: "Error re-encrypting credential Id 022e607e-b49e-40e4-bd3f-f56a3c69f94d with owner Id 431b6897-16eb-4b9a-be65-60f1f603008d during deploying of administration application pool credentials, please recreate credential manually. Operation is not valid due to the current state of the object."
Figure 5 Unable to re-encrypt application pool passwords because the passwords can no longer be decrypted
If you just changed the farm account in a single-server deployment from the Network Service account to a domain account, you are in real trouble because you can't change back to the old credential key in this case. The Network Service doesn't use a password, and the credential key is therefore random.
In a search for help, you might encounter that "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption” article I mentioned earlier, and without further knowledge, your trouble worsens because you now learn that you must create a new configuration database to get rid of these passwords that can no longer be decrypted. It is an unbelievable constellation of unfortunate circumstances. You shouldn't be able to run the updatefarmcredentials command with the -local switch in the first place, or the command should create a backup copy of the old credential key so you can re-encrypt the passwords later on. Or it should simply detect that the passwords are not yet re-encrypted and re-encrypt them at this point.
Instead, the -local switch goes ahead and happily corrupts your SharePoint configuration database without any warnings, as shown in Figure 5. A support tool from Microsoft to reset those passwords that can no longer be decrypted would come in handy. And, of course, proper product documentation that warns you about the critical nature of certain command-line operations and that guides you out of this trouble would also be appreciated.
The good news is that the updateaccountpassword command can encrypt new passwords of application pool accounts without having to decrypt the old passwords. So, use this command to update all application pools that use domain accounts. This should take care of most, if not all, corrupted passwords. Unfortunately, you can't use this command to update application pools that use the Network Service account. This account doesn't require a password, so the updateaccountpassword command doesn't apply.
Interestingly, the Network Service account might be associated with password data in the configuration database. New application pools that use the Network Service have no passwords, yet if you ever change the application pool to use a domain account and then revert back, the Network Service inherits the domain account's password reference. SharePoint doesn't set the password to null (bad coding practice) and this rubbish data now causes the trouble.
It's an ironic twist that customers are supposed to throw away their configuration databases because rubbish and completely useless data can no longer be decrypted! If you are lucky, you can change the application pool configuration in Central Administration (_admin/FarmCredentialManagement.aspx) and specify a domain account. If you are unlucky, you encounter the encryption/decryption error displayed in Figure 6. You can't change the account in Central Administration, you can't use the updateaccountpassword command, you can't run the SharePoint Products and Technologies Configuration Wizard, and you can't update the farm credentials by using the updatefarmcredentials command. Now what do you do?
Figure 6 Trouble caused by Network Service password junk left behind in the configuration database
To solve this problem, you need a tool that goes straight into the configuration database and removes the junk, such as the Reset AppPool Password tool, illustrated in Figure 7 and included with source code in the companion material. This tool is very straightforward. It pulls the data of application pools that use accounts with associated encrypted passwords directly out of the configuration database and then uses the SharePoint object model to determine if the application pool password can be decrypted.
Figure 7 Resetting corrupted passwords to null
If accessing the password through the object model fails with an argument exception, the password is corrupted. Here, the tool gives you an opportunity to replace the password's array of encrypted byte values with a null reference and saves the changes back into the configuration database. Empty strings don't need to be decrypted and therefore don't cause an argument exception. Problem solved.
In order to complete the repair process, I recommend that you update the farm credentials and subsequently all application pool accounts by using Stsadm.exe and Central Administration. Your farm is back in a consistent state, and you didn't have to throw away your configuration database.
Despite the fact that changing farm credentials and security account passwords is a tedious and error-prone process, you need not fear that changing farm credentials will corrupt your server farm irreparably. The configuration database can be repaired even if you happen to lose your current credential key. You simply need to reset the affected passwords, and this is possible through the standard Stsadm.exe tool or a low-level database tool, such as Reset AppPool Password. So keep changing farm credentials and security accounts frequently, use strong passwords, and don't use the Network Service as a security account because it complicates the farm configuration and troubleshooting. Use dedicated domain accounts.
Now that you've addressed the risks of database corruption through password changes, you can focus your attention on the real issue in the SharePoint security architecture: the current architecture does not accommodate password changes very well. You need to apply too many commands in a more or less specific order depending on the service types you must update in your farm. Some commands have dangerous switches; some don't. Some commands can corrupt your configuration database; others are harmless. Some services need to be updated globally across the entire farm and others are local to a specific server.
In any case, administrative overhead is high due to the complexities involved, and security is generally low due to infrequent change schedules, weak passwords, and scripts that deal with passwords in clear text. In my next column, I'll show you how to tackle these issues and eliminate them—and for all service types, including those not yet developed, whatever their password update requirements. No more (manual) password changes!
Pav Cherny is an IT expert and author specializing in Microsoft technologies for collaboration and unified communication. His publications include white papers, product manuals, and books with a focus on IT operations and system administration. Pav is President of Biblioso Corporation, a company that specializes in managed documentation and localization services.