Narrowing down odd behavior in the Program Compatibility Assistant on Windows Vista

Recently, a fellow Microsoft employee found this blog post that I wrote a while back, and he contacted me with a question regarding some odd behavior he was seeing with the Program Compatibility Assistant (PCA) on Windows Vista.  He found that a specific EXE consistently triggered a PCA dialog when he closed it on his system, but nobody else on his team saw this behavior for this EXE on their systems.  What made this even more strange was that this EXE included an embedded UAC manifest, which should prevent Windows Vista from triggering any PCA dialogs for the EXE.

After consulting with the Windows application compatibility team, we finally found the root cause and were able to fix this system so it no longer triggers PCA dialogs for this EXE.  In the process, we learned some interesting details about how Windows Vista works behind the scenes when deciding whether or not to trigger a PCA dialog.  I haven't seen this information documented very thoroughly so far, so I wanted to post about this experience here in case it is helpful to anyone else.

Problem 1 - The embedded UAC manifest was not recognized

The EXE in question had an embedded UAC manifest, but it was embedded using resource ID value 0.  Once the EXE was rebuilt with the UAC manifest embedded using resource ID value 1, then Windows Vista was able to find the manifest, and it stopped triggering PCA dialogs on this system.

For reference, the instructions in this MSDN topic can be used to embed a UAC manifest in an EXE using Visual Studio or via command line using mt.exe from the Windows SDK.  The examples in this topic demonstrate how to embed the manifest using resource ID value 1, but the topic does not specifically state that you have to use resource ID value 1.  The experience we had debugging this issue suggests otherwise.

Problem 2 - Windows Vista logic for triggering PCA was affected by an uninstalled application

Due to the unrecognized UAC manifest described in problem 1, this EXE was treated by Windows Vista as a legacy application, which caused it to be eligible for PCA dialogs when the user exits the EXE.  However, PCA is designed to help resolve problems with failing legacy setup programs, and this EXE is not a setup.  That left us with another issue we needed to diagnose in order to fully understand the behavior on this system - why was this non-setup EXE being treated as a failing setup?

After discussing this scenario with the Windows application compatibility team, we discovered that Windows Vista monitors events it receives from the UAC virtualization system driver.  If Windows Vista finds from these events that an application attempted to create a folder under the Program Files directory and then attempted to create/copy files into a folder under the Program Files directory and those operations fail, then it will treat the application as a failed installer and trigger a PCA dialog if it does not contain an embedded UAC manifest.

Fortunately, the UAC virtualizaton system driver logs events in the event log that can be used to help narrow down the root cause of this type of issue.  In the case of this system and this EXE, several events were logged from the Microsoft-Windows-UAC-FileVirtualization source.  These events indicated attempts to perform operations on a path under Program Files for an application that has recently been uninstalled.

It turned out that the recently uninstalled application left behind some information in the PATH environment variable and in the registry that caused other programs to attempt to read and write from its installed location under Program Files.

In the case of this EXE, because of these orphaned settings as well as the unrecognized UAC manifest, Windows Vista treated the EXE as a failing legacy setup program and triggered a PCA dialog each time it exited on this specific system.  Other users did not see this behavior because they did not have this other application installed and uninstalled on their systems.

Summary of what we learned from this scenario

We were able to fully resolve this issue by doing the following:

  1. Updating the EXE to embed the UAC manifest using resource ID value 1
  2. Removing the left over information in the PATH environment variable and in the registry on this system

We also learned a few key things about UAC and PCA on Windows Vista:

  1. Embedded UAC manifests must be placed at resource ID value 1 in order to be correctly recognized by Windows Vista
  2. Windows Vista treats programs that attempt to create folders and files under Program Files as setup programs
  3. Windows Vista logs some events in the event log from the Microsoft-Windows-UAC-FileVirtualization source that can be used to diagnose why a PCA dialog is being triggered for a given EXE