Common Windows Store certification errors: 3.8 App must meet the basic performance criteria on a low-power computer
This week, I am writing a blog post series explaining the most common certification errors when submitting an app to the Windows Store. I began the series with some general guidance and overall tips & tricks. Then yesterday we discussed the commonly-failed requirement 1.2 on apps being fully functional.
Today, we will examine certification requirement 3.8: “Your app must meet the basic performance criteria on a low-power computer” . Here is the description of this requirement from the certification requirements page:
“Your app must meet the basic performance criteria on a low-power computer.
- The app must launch in 5 seconds or less
- The app must suspend in 2 seconds or less
Low-power computers are described in How to test your app with the Windows App Certification Kit.”
So let’s talk performance of Windows Store apps!
First of all, one of the tests in the Windows App Certification Kit (WACK) is a performance test. (You should read that link because there is a ton of useful information on performance testing of Windows Store apps!) Side note: If you’re not familiar with it, I describe the WACK in this post and in this post, and there is MSDN documentation for it here.
So hopefully you’re already running the WACK at least on your development machine, and it passes. But during Windows Store certification, the testers are running it on a low-power machine. To adequately test performance, you should run the WACK on a low-power computer as well. One recommendation in our documentation is to use an Intel Atom processor-based computer with a screen resolution of 1366x768 (or higher) and a rotational hard drive (as opposed to a solid-state hard drive).
If possible, it is also advisable to test your application on an ARM device if you are claiming “ARM” or “Any CPU” support. If you are unsure how to test on ARM since Visual Studio isn’t available for ARM devices, you can use remote debugging. Tim Heuer wrote a fabulous post on how to enable remote debugging on an ARM device.
Now, the certification testers will be testing your app on a clean installation of Windows 8 with no additional software running, so you can utilize that environment for your testing as well. Other software running on the machine will negatively affect the results of the WACK’s performance test, so I often start the WACK running right before I’m heading to bed and I won’t be using my machine (although it only takes about 5 minutes to run, so you could do it during a coffee break or something as well). Of course, running the WACK in a virtual machine or through remote desktop will also skew the performance results, so running on the metal is best.
The performance certification requirements (and tests in the WACK) cover launch time and suspend time of an app specifically. These times depend on many factors, like system load and configuration, which can change between machines and even between runs of the same test depending on what else is going on in your machine. Therefore, a launch time or suspend time may change enough to just stay under the allowed time limits in one occurrence and fail in slightly different conditions. If the app’s launch time or suspend time is close to the limit, review what the app is doing to handle those events and look for ways to reduce that activity.
How do you find out if your app’s times are close to the limit? Great question. See the WACK screenshot above? That is the screen that you will see at the end of the WACK run, if your app passes. Click on the “Click here to view full report”. It will take you to a nicely-formatted HTML page called ValidationResult.htm. In the same directory as that file, there will be another file called “ValidationResult.xml”. Open that file and search for tests named “Performance launch” and “Performance suspend”. Those tests will each have an attribute called “EXECUTIONTIME” that gives you the time that each took to run.
So how can we improve performance!?!??! Here are a few tips, and then some fabulous resources for even more information:
- Package content locally (or cache it) when possible, so you don’t have to pull resources from a network during launch.
- Load and do only what you need on launch. You can load other data and do other work asynchronously in the background or when you actually need it.
- Use a “dirty bit” when saving data on suspend, so you only save data that has changed. (Rather than re-serializing your app’s state if that data hasn’t changed, create a Boolean flag variable (or “dirty bit”) which signals that your data has been modified, and only re-serialize when the data has changed.)
Windows Store Performance Resources
Launch time reduction:
Minimize startup time – C#/VB/C++ and XAML
Suspend/Resume time reduction:
Minimize suspend/resume time – C#/VB/C++ and XAML
General performance tips & tricks:
Other blog posts in this “Common Windows Store Certification Failures” series: