ASP.NET Precompilation: Speeding up Orchard cold start
Orchard is a "Content Management System" that you can use to easily host/customize your content (such as your blog for example). It is built by Microsoft and built on top of ASP.NET MVC. It is one of the Microsoft open source projects so it has a great community and many external modules you can easily add to power your application.
One of the teams I work with in Azure uses it for their Portal Web experience.
The problem:
The load time of the website was very slow the first time each page was opened (cold start latency). It would sometimes take upwards of 10 seconds to open a page for the first time. Normally cold start is a very low priority for a web application. The reason being that a given page might be seen thousands of times so if you have a slow experience only one time of 1000 you will be ok.
Even when you have performance tests for your scenarios you focus on percentiles for your latencies measurements. So 99th percentile will tell you how long it’s taking for 99 percent of your users. As you can imagine it’s very easy to dismiss the cold start scenario.
In our project this was not the case. Due to many reasons all of our customers would see this at least once for each page they visited (not just the first customer). They would also see it again whenever the server was rebooted which would be at least once a week but probably more than that (since it's hosted in Azure).
Also any new customer would see this whenever they tried the service for the first time (So a very bad first experience).
All of these made it a deal breaker that had to be fixed.
Analysis
So I had to figure out what the problem was and how to fix it. I deployed the service and connected to it. Then I started Perfview to take a profile while the first page was getting loaded. Here is the result:
ASP.NET Stats View
Statistics Per Request URL
Method |
Path |
Query String |
Num |
Total MSec |
Start of Max |
End of Max |
GET |
/ |
|
1 |
24,552 |
2,191.056 |
26,743.336 |
Processes view
Processes that did live for the entire trace.
Name |
CPU MSec |
Ave Procs Used |
Command Line |
w3wp |
1,426 |
0.039 |
d:\windows\system32\inetsrv\w3wp.exe -ap "DefaultAppPool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipmb30c9525-04d5-42ee-afca-a2530cf3bc4b -h "D:\inetpub\temp\apppools\DefaultAppPool\DefaultAppPool.config" -w "" -m 0 |
Processes that did not live for the entire trace.
Name |
CPU MSec |
Duration MSec |
Start MSec |
Command Line |
csc |
4,001 |
6,178.415 |
20,013.375 |
"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\b2554fb2\9f26c9ab\cwyho2o2.cmdline" |
csc |
3,891 |
6,480.819 |
12,201.761 |
"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\b2554fb2\9f26c9ab\svtkuesz.cmdline" |
csc |
3,429 |
7,121.114 |
4,497.857 |
"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\b2554fb2\9f26c9ab\0pdpgema.cmdline" |
csc |
562 |
7,891.250 |
28,363.120 |
"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\b2554fb2\9f26c9ab\tn0rpr3e.cmdline" |
Let me translate that for you:
1) Opening the root of our app took 24.5 seconds!
2) The w3wp process (which is the ASP.NET worker process) only consumed 1.4 seconds of CPU cycles. The rest of the time it was waiting on something
3) There is a bunch of csc processes running during this time
CSC is the C# compiler. But why am I seeing it here?
“By default, ASP.NET Web pages and code files are compiled dynamically when users first request a resource such as a page from the Web site. After pages and code files have been compiled the first time, the compiled resources are cached. Therefore, subsequent requests to the same page are very efficient.”
My first question was why would anyone want to do this? Why would you make your customer pay for this instead of doing it in your developer machine? The answer to that is that this improves the developer experience by a lot! Whenever you are iterating in a website you make many code changes and having to compile after each change is very time consuming. In this mode ASP.NET allows you to go to your website and modify the code (such as the content on your Razor views) and ASP.NET will realize that something changed and will recompile that page for you.
Now I’m not 100% sure this is very useful in actual production environments. But I’m not an expert on ASP.NET MVC. However the ASP.NET team released a tool that basically lets you precompile your site before you deploy it in case you don’t care about this dynamic changes.
Here is a guide on how to do it from MSDN.
Solution
So we have to use this aspnet_compiler tool to fix the problem. However after trying it on our Orchard site I hit some errors such as this:
Error 2 The type or namespace name 'Markdown' could not be found (are you missing a using directive or an assembly reference?) c:\Users\shacorn\AppData\Local\Temp\Temporary ASP.NET Files\orchardlocal\2b51729c\84e739fa\App_Web_ztbavtrn.0.cs 29 0 Orchard.Web
..\Portal\0.0.2063.0\Modules\Orchard.Alias\web.config(35): error ASPCONFIG: Could not load file or assembly 'System.Web.Mvc, Version=5.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Some Orchard forum posts mentioned that this is not supported. Such as this one.
---------------
I talked with one of the Orchard core developers (Sebastien Ros) and he helped me figure out how to make it work with Orchard.
Orchard will dynamically load modules that are in the Orchard\Modules directories. So to precompile it we need to do a little extra work:
- Run the precompilation tool against a “deploy” version of your site. Not from the plain source directory.
- When you do “deploy” from Visual Studio there are some things specific for Orchard that make it necessary to do some other things. In particular due to the way that Orchard loads modules you need to build Orchard modules and pack them in assemblies. This can be done with this command from the Orchard root folder:
build /t:precompiled - Note that you need the entire source code for Orchard. Not just the redistributable. You can get it from here: https://www.orchardproject.net/download. Make sure you select “download the source”
- Deploy your web application to a directory. For example: %outputDir%\Web
- Make sure there are no csproj files under modules or under themes directories. Otherwise they will confuse the tool. Actually there should be no csproj files or cs files. We still will have ASP.NET views (cshtml).
pushd .\Output\Release\Portals.Web\Modules
del *.csproj /s - Precompile with a command like this:
aspnet_compiler -v / -p %outputDir%\Web %outputDir%\Web.Precompiled
The parameters we are passing to the aspnet_compiler are the following:
- First we are telling the compiler the virtual path to be root or your site (-v /) and the physical path to be %outputDir%\Web. Note that this is the same path we gave as the output path for step (3) above.
- Then we pass the output directory for the precompiled version
The fact that we are passing root as the virtual path is important because this value will be inserted in the compiled files. So you want them to match the value in your deployed machines.
How to find the virtual path root for your app?
The way I found the correct value for this was a little roundabout. As I mentioned before whenever ASP.NET sees a view (cshtml file) it will compile it. The output of this is placed in the folder: “Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files”.
So I went to the machine where I took the profile and opened one of the generated .compiled files. That showed me something like this:
virtualPath="/Themes/Portals.Themes.Bootstrap/Views/Portals.Web.Authentication/Home/_ProviderInfo.cshtml"
Since Themes is a physical directory I know that the virtualPath should be “/”. So by passing root (“/”) I am able to match this.
Some other things to note about precompilation:
1) It will definitely increase your build time. It added 10 minutes in my local machine. So this is something that you probably want to do as part of your deployment script, not something you do every time you build.
2) It will find errors that you probably did not know you had. Some old views that you need to remove for example.
3) Precompilation has a known issue with VirtualPathProvider. In our case passing (HostingEnvironment.VirtualPathProvider) to the VirtualPathProvider made the issue go away.
For us the result was dramatic. We were able to get up to 70% performance improvement in latency (99th percentile). The portal feels much faster and the overall experience is much better. Totally recommended :)