Generating a memory dump of the process running your code

Thought this was a pretty cool way to gather data for a problem I’ve been trying to track down in a large farm environment where you have a large number of suspect processes, and need to identify which process and code is causing some behavior.  The main problem we are tackling is that Self-Service Site Creation announcements keep showing up in the middle of the night.  Not just one, but sometimes 4+.  Troubleshooting this is problematic as the only code path that creates the announcement is when you set the SPWebApplication.SelfServiceSiteCreationEnabled property to true, and there is no logging down the entire code path.  Without logging, you cannot use the ULS logs or Event Viewer to determine which process is creating the announcements.

We did some of the typical troubleshooting.  We checked the IIS logs for Central Admin to see if anyone was browsing the page that enables Self-Service Site Creation [ConfigSSC.aspx].  Confirmed there are no content deployment jobs.  Tried using Process Monitor to see if there was a process doing work at the time of the events [someone running stsadm.exe –o enablessc for example], but this proved difficult across multiple machines and a large time window.  Thought about trying to catch this with a breakpoint, but doing this across multiple machines with multiple processes didn’t seem like a great way to go.  Since I’m writing this, none of those approaches panned out :)

Then while talking with a co-worker [Thanks Alonzo!], I went with an Event Receiver.  I did some testing and confirmed that the Event Receiver runs in the process the list item is created from.  If you run stsadm.exe –o enablessc, the event receiver runs in stsadm.exe.  From there, I wrote some code that would run the command-line debugger, cdb.exe, to create a memory dump of the process the code is running in.  Got this working, and there is potential to use a similar approach to tackle other types of problems.

There are two parts here…the first part is the Event Receiver.  In this case, we cared about the ItemAdding event as we want to catch the item as it’s being created, and cancel the item after we get the data.  Here’s info on creating an Event Receiver from MSDN, so we won’t cover those details.  In the ItemAdding event, we use System.Diagnostics.Process to call cdb.exe and pass it a text file that we dynamically generate.  Here’s what the code looks like:


    1:  if (properties.AfterProperties["Title"].ToString().ToLower().Contains("self-service site creation"))
    2:  {
    3:      //Get the Process ID, machine name, and Process Name of the process this code is running in
    4:      string pid = System.Diagnostics.Process.GetCurrentProcess().Id.ToString();
    5:      string machineName = System.Diagnostics.Process.GetCurrentProcess().MachineName;
    6:      string process = System.Diagnostics.Process.GetCurrentProcess().ProcessName;
    8:      string dateTime = DateTime.Now.Ticks.ToString();
   10:      //Build a path to a file to contain the debug commands to run
   11:      string commandFilePath = "C:\\Program Files\\Debugging Tools for Windows (x64)\\CDBCommand_" + pid + "_" + dateTime + ".txt";
   13:      //Path to the command line debugger: cdb.exe
   14:      string cdbPath = "C:\\Program Files\\Debugging Tools for Windows (x64)\\cdb.exe";
   16:      //Command to pass into cdb.exe.  This command attaches non-invasively to the PID and passes
   17:      //the command file to tell the debugger which commands to run.
   18:      string command = " -pv -p " + pid + " -cf \"" + commandFilePath + "\"";
   20:      try
   21:      {
   22:          //Create the command file in the path
   23:          using (FileStream commandFile = File.Create(commandFilePath))
   24:          {
   26:              using (StreamWriter outfile = new StreamWriter(commandFile))
   27:              {
   28:                  //Add the debugger commands to create a memory dump and quit the debugger
   29:                  outfile.WriteLine(".dump /ma C:\\SiteCreationDump\\" + machineName + "_" + process + "_" + pid + "_" + dateTime + ".dmp");
   30:                  outfile.WriteLine("q");
   31:              }
   32:          }
   34:          //Start cdb.exe and pass in the command info.
   35:          //Loop until the cdb.exe process completes so the dump will contain this thread at this point.
   36:          Process cdbProcess = System.Diagnostics.Process.Start(cdbPath, command);
   37:          while (!cdbProcess.HasExited) { cdbProcess.Refresh(); }
   39:      }
   40:      finally
   41:      {
   42:          //Clean up the command file
   43:          if (File.Exists(commandFilePath))
   44:          {
   45:              File.Delete(commandFilePath);
   46:          }
   48:          //Cancel the item creation
   49:          properties.Cancel = true;
   50:          properties.Status = SPEventReceiverStatus.CancelNoError;
   51:      }
   52:  }


Second part is enabling/disabling the Event Receiver on the list you want.  Since we only wanted this hooked to a specific list, I wrote a console app to do this work.  We deployed the Event Receiver DLL via a Solution Package, then enabled/disabled the Event Receiver on the list using the Console App.  There’s a sample on how to hook up an Event Receiver to a specific list here.


Couple stumbling blocks I hit while putting this together:

  • Getting CDB.exe to run the commands you want.  I tried using the –c switch with cdb.exe to pass in the .dump and the q command to dump the process and quit the debugger.  Come to find out this only runs a single command, so the debugger would attach, the dump would be generated, then you’re stuck with a hung process as the debugger didn’t detach.  In order to run multiple commands I wrote out the commands to a text file, pass this in using the –cf switch, then clean up the text file when I’m done.

  • Getting the dump with the culprit code on the stack.  Since the call to System.Diagnostics.Process.Start is async, I had to add the while loop to ensure that the CDB.exe process has exited prior to continuing.  The cool thing is that this loop doesn’t run as much as you may think.  As soon as the debugger attaches, the code execution is halted.  Then when the debugger detaches, the loop stops.