How to modify an application behavior when you don't have the source
From time to time we need to help customers change the way an application interacts with the operating system or SDKs. The challenge is often the access to the code. Sometimes neither party may own the application in question and none of the parties have access to the source. Luckily, the Microsoft Research team came up with the Detours SDK to address this problem a number of years ago and the latest version makes it easy to implement a solution to a situation like this. In short, Detours allows you to create a DLL that hooks one or more operating system functions, so that when that function is called, the caller will actually invoke your custom Detours code instead.
The process is very simple:
· Download the detours SDK http://research.microsoft.com/sn/detours/ and build it.
· You can start with the SIMPLE Sample or our included sample that builds in the Visual Studio command-line environment.
· Create a function pointer prototype for the API you want to detour (TrueCreateFile in the example below). It should have the same parameters and return value as the function you will detour. As part of the declaration set the function pointer value to the real API Address. In the following sample we will detour the CreateFile API.
· You will also need to create your own version of the API you are detouring (ModifyCreateFile below). In this case we are creating our own Createfile, which will call the original CreateFile with the FILE_FLAG_WRITE_THROUGH flag OR’d into the dwFlagsAndAttributes parameter.
static HANDLE (WINAPI * TrueCreateFile)(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) = CreateFile;
HANDLE WINAPI ModifyCreateFile(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode,
LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile)
dwFlagsAndAttributes |= FILE_FLAG_WRITE_THROUGH;
return TrueCreateFile(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes,
dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);
· You will need to write your detour code in the DLLmain of your dll. This should be executed when your DLL loads and dwReason is == DLL_PROCESS_ATTACH. In our call to DetourAttach we pass our TrueCreateFile pointer (the real CreateFile address), and the address of ModifyCreateFile (our custom create file api). The detour API handles the intercept for us.
· When the DLL_PROCESS_DETACH happens you will need to clean up the detour and unhook the real API.
So how do you get the DLL loaded into the target process? There are a couple ways. I recommend using the setdll tool that comes as part of the Detour SDK. In the following case we are modifying NTBackup to automatically load our detoured DLL when NTbackup runs.
C:\test>setdll /d:nocache.dll ntbackup.exe
Adding nocache.dll to binary files.
MFC42u.dll -> MFC42u.dll
msvcrt.dll -> msvcrt.dll
ADVAPI32.dll -> ADVAPI32.dll
KERNEL32.dll -> KERNEL32.dll
GDI32.dll -> GDI32.dll
USER32.dll -> USER32.dll
ntdll.dll -> ntdll.dll
COMCTL32.dll -> COMCTL32.dll
SHELL32.dll -> SHELL32.dll
MPR.dll -> MPR.dll
comdlg32.dll -> comdlg32.dll
NETAPI32.dll -> NETAPI32.dll
RPCRT4.dll -> RPCRT4.dll
ole32.dll -> ole32.dll
SETUPAPI.dll -> SETUPAPI.dll
USERENV.dll -> USERENV.dll
NTMSAPI.dll -> NTMSAPI.dll
CLUSAPI.dll -> CLUSAPI.dll
query.dll -> query.dll
sfc_os.dll -> sfc_os.dll
SYSSETUP.dll -> SYSSETUP.dll
OLEAUT32.dll -> OLEAUT32.dll
VSSAPI.DLL -> VSSAPI.DLL
Note that if you modify a binary that is protected by Windows File Protection the modified binary will be replaced by the OS with the original binary. I recommend keeping your modified version in another directory so it does not get replaced.
Click here for the sample source code and Makefile, which will build from a Visual Studio command prompt.