MTBF Testing Requirements and Setup
This section describes the tasks that you must complete before you run Mean Time Between Failures (MTBF) tests by using the Hardware Lab Kit (HLK):
Test Requirements
Microsoft account sign-in information
Email server
Wi-Fi access point & connection information
SIM card with phone number that is provisioned on the SIM card
SIM card with unlimited calling, unlimited data, unlimited messaging
Note
The requirement for unlimited calling, unlimited data, unlimited messaging is very important; these tests send many text messages, make many phone calls, and browse many webpages. If these items are billed individually, the cost can be very high. It is not unexpected for a device running MTBF constantly to use up to and possibly over 18,000 minutes of call minutes, 12,000 combined SMS and MMS messages, and 5 GB of data.
Test Environment Setup
Wi-Fi Access Point
The MTBF test will connect to a named access point (AP) to validate Wi-Fi connectivity. The access point must meet the following requirements for testing:
Configure the AP’s SSID as a unique SSID in the range of the test execution. The SSID must be broadcasted and not a hidden AP
Configure the AP’s SSID as a unique SSID in the range of the test execution. The SSID must be broadcasted and not a hidden AP
The AP or its connected network must provide DHCP services
The AP cannot provide any kind of security or web redirects that could cause the browser to launch during connection setup
The AP cannot provide any kind of security or web redirects that could cause the browser to launch during connection setup
The AP does not need to provide Internet or other network connectivity
Cellular Environment
The provided MTBF test configuration has networking support and validation for:
GSM technologies – UMTS and LTE
CDMA technologies – EVDO, and LTE
The device, the associated cellular account, and the test environment location will need to provide support for each design network configuration that is being tested.
Dialup Server
The MTBF test will make and receive several phone calls to a dialup server, send commands across DTMF, and then respond accordingly. This server should be setup based on the Microsoft Dialup Server specification.
Phone numbers using the default dial-up server also need to be added to an approve list.
For dialup server documentation or support please contact dialsrvs@microsoft.com.
SMS/MMS Endpoint
The MTBF test will send multiple SMS and MMS messages to a defined endpoint. This end point will need to accept all incoming messages and must not respond with any messages or notifications.
It is recommended that this endpoint be another Windows Phone device, with the retail demo software installed. This ensures that the device is always on, accepts messages to ensure network queues do not get full, and resets itself at night to ensure the device does not fill up.
Device Setup
SIM Provisioning
The phone number provisioned on the SIM card is used by default to find the Microsoft Account configured for that device and for calling the device back during the Incoming Call Test. The phone number provisioned on the device can be queried and set with the tools installed with the MTBFSuite.
Retrieving the provisioned phone number:
PS > cmdd \data\test\bin\dmtools.exe getphoneids /outfile \PhoneID.xml /silent
PS > getd \PhoneID.xml
The XML file contents should be validated to see if the correct phone number is provisioned on the device.
Provisioning the phone number
PS> cmdd \data\test\bin\SetOwnPhoneNumber.exe –change ##########
Note
If this number includes the country code of 1, Microsoft Accounts will need to be created containing this leading 1.
Microsoft Account
A unique Microsoft Account will need to be created for each device in test. This can be done by going to https://www.live.com and clicking the sign-up link.
It is strongly recommended that these accounts are not re-used for any other testing purposes as any unexpected data or account transactions outside the device in test may cause test failures lowering a devices pass rate.
By default the MTBF Suite is configured to use an account name pattern to allow for easier configuration of multiple devices.
Note
Birth date: must be over 18 years old.
Location: It is recommended to use a US zip code, such as 98052.
Note
It is recommended that this Microsoft Account does not add any additional data in Microsoft Services that could interfere with the MTBF test. This includes:
https://outlook.com, do not add additional contacts, calendar appointments, or attached mail accounts
Windows Phone Store, do not purchase additional applications with this account
Windows Phone Backup, do not backup any Windows Phone’s with this account
The Microsoft Account is only signed in on a single device at a time.
MTBF Configuration Sets
The MTBF Test Suite has multiple configuration sets for testing in different environments. The desired configuration set to execute can be passed in through the ‘MTBFConfigurationSet’ parameter.
| Configuration Set | Description |
|---|---|
FullRunIterationSettings_UMTS_LTE |
This is the default MTBF configuration set. This configuration set runs the full sets of test cases and enables network switching between the UMTS and LTE networks. |
FullRunIterationSettings_UMTS |
This configuration set runs the full set of test cases on the UMTS network. |
FullRunIterationSettings_EVDO_LTE |
This configuration set runs the full sets of test cases and enables network switching between the EVDO and LTE networks. |
FullRunIterationSettings |
This configuration set runs the full sets of test cases, however does not enable any network switching. The device will run the MTBF test cases on the default network that the device is connected to. |
MinRunIterationSettings_UMTS_LTE |
This configuration set will run each test case a minimal number of times to exercise each test case switching between UMTS and LTE. |
MinRunIterationSettings |
This configuration set will run each test case a minimal number of times to exercise each test case. It does not enable any network switching and the device will run the MTBF test cases on the default network that the device is connected to. |
MTBF Configuration Parameters
The default MTBF configuration and test cases are defined in the MtbfMix.xml. While this can be modified, it is recommended that you do not modify this file to ease integrations of new Windows Phone builds as this file may change between each drop.
Your custom MTBF configurations should be placed in an override XML that can be maintained by you and passed into as a parameter each time to configure values that are unique to your test environment. Only values that need to be changed should be included; values that that are not included will use the default values in the Microsoft MtbfMix.xml
A basic Custom_Override.xml:
<?xml version="1.0" encoding="utf-8"?>
<MtbfConfiguration>
<ConfigurationSets>
<!-- Configuration Set specifying override engineering settings. -->
<ConfigurationSet Name="ExternalEnviromentSettings" BaseConfigurationSetName="InternalEnviromentSettings">
<Setting Name="EnvSettings_WiFi_SSID" Value="EXAMPLE_AP"/>
</ConfigurationSet>
</ConfigurationSets>
</MtbfConfiguration>
You can include as many of the settings as needed. The most commonly change settings are outlined below. This can be applied to the device by specifying your custom XML for the “MtbfMixFile” parameter.
<?xml version="1.0" encoding="utf-8"?>
<MtbfConfiguration>
<ConfigurationSets>
<!-- Configuration Set specifying override engineering settings. -->
<ConfigurationSet Name="ExternalEnviromentSettings" BaseConfigurationSetName="InternalEnviromentSettings">
<Setting Name="EnvSettings_WiFi_SSID" Value="EXAMPLE_AP"/>
</ConfigurationSet>
</ConfigurationSets>
</MtbfConfiguration>
An OEM can include as many of the settings as needed. The most commonly changed settings are outlined below.
Microsoft Account User Name and Password
<Setting Name="DeviceSettings_MSA_Username" Value="TestUsername@live.com"/>
<Setting Name="DeviceSettings_MSA_Password" Value="Passw0rd!"/>
These accounts must already be created or the MTBF test will fail.
Dialup Server Numbers
<Setting Name="EnvSettings_Phone_CallServerNumber" Selection="Random">
<Value>4257276836</Value>
<Value>4257276837</Value>
<Value>4257278701</Value>
</Setting>
A list of one or more dialup servers to be used during the phone test. Only one number is selected at the beginning of the test and used for the entire execution instance. This must be specified as all digits with no formatting.
SMS and MMS Endpoint
<Setting Name="EnvSettings_Messaging_SMSEndpoint" Value="4257859632"/>
The phone number that MMS & SMS messages are sent to. This must be specified as all digits with no formatting.
<!-- values in the parameters are /easrange (acctname) (emailformat) (maxnumber) (passwd) (usernameformat) (server) (domain) [(ssl)] -->
<Setting Name="Comms.Email.Account" Value="/easrange MOTestAccount mtbf{0}@mtbf.syncxp.net 500 OMLab01! mtbf{0} mtbf.syncxp.net mtbfdom.syncxp.net 1"/>
<Setting Name="Comms.Email.EmailRecipient" Value=""/>
Exchange Server email server is required:
Parameter reference
| Command line | Description |
|---|---|
/eas <acctname> <email> <passwd> <username> <server> <domain> [<ssl>] |
Creates a general Exchange partnership. <ssl> must be 1 or 0, and it is optional. |
/live <acctname> <email> <passwd> |
Creates a Windows Live partnership. |
/pop <acctname> <email> <passwd> <username> <popserver> <smtpserver> |
Creates a POP partnership. |
/imap <acctname> <email> <passwd> <username> <imapserver> <smtpserver> |
Creates an IMAP partnership. |
Note
An email account is required for the Email tests. Third parties may use an Exchange account, a Microsoft account, an IMAP account, or a POP account. A second email account is required to receive the email sent in the Email tests. This second account is NOT synced to the device, but it must exist in order to avoid bounced email. The second account will fill up with test messages, so it is recommended to manually clear the account out on a regular basis.
Video Streaming URL
<Setting Name="Media.Apps.MusicVideo.Video.URL" Value="https://www.sfone.net/test/MTBF/Tennis_VC1(SP)_176x144_12fps_96kbps_WMA9_128kbps_CBR_44.1KHz.wmv"/>
The URL of the video file to stream during the video streaming test.
Wi-Fi Access Point
<Setting Name="OSPlat.Wifi.SSID" Value="DEVHEALTH"/>
The Wi-Fi Access Point name to connect to during the Wi-Fi test section.
Note
The Wi-Fi Access Point names are case sensitive.
Starting MTBF from a Random Section
<Setting Name="StartFromRandomSection" Value="false"/>
MTBF client can start the MTBF testing from a random section in case additional coverage is needed on test cases.
OMA Client Provisioning Overrides
<Setting Name="MTBFCustomWapCfg" Value="\data\test\tools\mtbf\mtbf_wapcfg_default.xml"/>
When testing in various test environments, it may be needed to override certain device settings. This extra configuration can be specified using the OMA Client Provisioning\WAP Configuration XML file format. This file must be located on the device and specified as the full file path.
Disk Filler
<Setting Name="EnableDiskFiller" Value="true"/>
DiskFiller will fill the disk contents up to the percent value specified in ‘LeavePercentage’ or size in Megabytes specified in ‘LeaveMB’ depending which is larger.
Disk Monitor
<Setting Name="EnableDiskMonitor" Value="true"/>
DiskMonitor will monitor the total storage space to ensure enough space is left for crash dumps and will exit the run if the device free space drops below the specified disk space.
Cab Monitor
<Setting Name="EnableCabMonitor" Value="false"/>
CabMonitor will monitor the number of cabs collected during execution and stop the run once ‘MaxCabCount’ (default 500) is hit.
Preparing a Phone for Running MTBF
MTBF should be run on devices during multiple stages of the development cycle to ensure that issues that block MTBF execution or mobile operators shipping requirements can be addressed in an appropriate time frame. This also means that during the development cycle different types of images may be used.
For final device sign off a Windows Phone Retail image should be used for testing.
Supported images for MTBF test execution
- Windows Phone Health SKU
Installing the FFU Image on the Phone
Refer to the HLK documentation for flashing the FFU image on the phone.
Installing the HLK Windows Phone Client
Refer to the HLK documentation for installing the HLK Windows Phone Client.
Desktop and HLK Controller Setup
Refer to the HLK documentation for installing the HLK Controller on the desktop.