I have spent the last couple years investigating the performance of various parts of the CLR and managed code, in that time I have learned that a profiler is an invaluable tool. There are quite a few profilers available including: CLR Profiler, Intel's VTune, AMD's CodeAnalyst, Microsoft's VSTS Profiler, Kernrate and various profilers which Microsoft has developed internally (including the predecessors to VSTS Profiler) as well as many others which I don't list here only because I don't have experience with them. While every profiler has strengths and weaknesses, lately I've been finding that I turn to the VSTS Profiler more often than not for my first analysis.
If you have VSTS and want to learn more about the profiler see the team's blog: http://blogs.msdn.com/profiler/
VSTS Profiler has many options and through those can get at a lot of data about your application, however I usually find that some of the most interesting can be obtained with simple vanilla sample based profiling. You can do this through the VS interface, or from the command line using the VSTS Profiler command line utilities, basically the steps are as follows:
1) Only if profiling managed code: vsPerfClrEnv.cmd /sampleOn
2) vsPerfCmd.exe /start:sample /output:<name>.VSP
3) vsPerfCmd.exe /launch:<application>.exe
<<wait for application to close>>
4) vsPerfCmd.exe /shutdown
By following those 4 steps you can profile an application, the result is a .VSP file named <name>.VSP. This contains all the profile data collected during the run. This file is readable directly by VSTS. There are however interesting analysis to do on the data that the VSTS 2005 interface doesn't have support for, to address this issue the VSTS Profiler team has included a utility called vsPerfReport.exe which can translate a .VSP file into a set of comma seperated value (CSV) files, one for each of the major views seen in VSTS. In order to do this you execute:
5) vsPerfReport.exe /summary:all <name>.VSP
at the end of all this you will have a set of files named <name>_<type>.CSV which are views into the data in the VSP, you can open these in Notepad to view the text or import them into Excel to see the columns more clearly called out.
Once the data is in this form we can do whatever we like with it, for instance sorting by a particular column in Excel or searching for a particular row using a string in Notepad. However, wouldn't it be interesting to be able to do more?