ScatterGather (SYSTEM)

  • Article

This test validates file system I/O reliability scenarios, primarily asynchronously, utilizing the ReadFile/WriteFile/ReadFileScatter/WriteFileGather APIs. HCK tests exercises predefined automated regression variations.

Test details

Associated requirements

System.Fundamentals.StorageAndBoot.BootPerformance

See the system hardware requirements.

Platforms

Windows RT (ARM-based) Windows 8 (x86) Windows RT 8.1 Windows 8.1 x64 Windows 8.1 x86

Expected run time

~240 minutes

Categories

Certification Reliability

Type

Automated

 

Running the test

Before you run the test, complete the test setup as described in the test requirements: WDTF System Fundamentals Testing Prerequisites.

Simultaneously performs I/O (both writes and read) in a multi-threaded fashion utilizing I/O completion ports to complete the I/O. It will start out by initially setting the EOF of the file and scheduling the I/O threads. While the file is being written, it performs online verification where it expects zero or offset computed value. Once the entire file has been written, it performs full verification of the file and will fail the test if zero or an unexpected value is encountered.

Troubleshooting

For troubleshooting information, see Troubleshooting System Fundamentals Testing.

This test returns Pass or Fail. To review test details, review the test log from Windows Hardware Certification Kit (Windows HCK) Studio.

More information

Supported test modes:

We expect most users to use predefined regression variations.

Default, Regression (predefined variations) and stress indicate certain types of failures, such as Virtual Alloc retries etc. For default mode, there is a flagsauto flag which manages most of the parameters if the user doesn't want to supply the parameters manually. In addition timer functionality is supported with default and stress modes. For regression mode, there is a pre-defined timer variation.

Supported read/write modes:

Read/Write mode:

  • RW( Read, Write)

  • RWG( Read, Write Gather)

  • RSW( Read Scatter, Write)

  • RSWG( Read Scatter, Write Gather)

  • RRW( Random Read, Write)

  • RRWG( Random Read, Write Gather)

  • RWR( Read, Random Write)

  • RSWR( Read Scatter, Random Write)

  • RRWR( Random Read, Random Write - Approx. 50% Scatter/Gather) -> Default

  • RRWR7( Random Read, Random Write - Approx. 30% Scatter/Gather)

  • RRWR3( Random Read, Random Write - Approx. 70% Scatter/Gather

Additionally, the test supports Reverse Read and Reverse Write.

I/O Throttle Mechanism:

Supports Upper & Lower threshold for controlled IO requests. Bypass mode is also possible which essentially blasts the system with I/O.

Data Pattern:

Simple Offset computed alphabetic value. For timer variation, there can be holes though.

Data Validation Modes:

Supports both online (simultaneous validation along with writes), fullcheck (which is after all write’s completed), and offline verification modes. All these modes can be turned off/on. If online verification is turned off, essentially only writes are happening during the initial phase after which fullcheck will be performed.

Difference between Online and Full check is that full check is only done after all the writes are completed by single thread while online verification is done simultaneously during the writes and during, which it either expects zero or offset computed value, unlike fullcheck where it only excepts offset computed value.

Command usage

Command Description

ScatterMultiThread.exe -default -rwmode:rw -iomode:sync

Perform Read Write IO in Sync mode with default file size.

ScatterMultiThread.exe -default -rwmode:RSWG -RR -RER:2 -TESTPATH:d:\Scatter -FILESIZE:500m

Perform full Scatter Gather, 500m file, in the testpath specified with random offset and reverse read.

ScatterMultiThread.exe -default -rwmode:RRWR -FILESIZE:2g -IODIFFUPPERTHRESHOLD:500 -IODIFFLOWERTHRESHOLD:100

Perform 50% Scatter Gather IO 2GB file with I/O upper and lower threshold

ScatterMultiThread.exe -regression -regression:6

Perform pre-defined regression variation 6.

ScatterMultiThread.exe -stress -RWMODE:rrwr -THREADS:2 -BS:100k

Perform scatter gather in stress mode with 2 reader and 2 writer threads with I/O block size of 100k/

ScatterMultiThread.exe -default -FILE:timer.dat -timer:19s -FILESIZE:5g -ONLINEVERIFY:FALSE

Timer mode ReadWrite Unbuffered I/O and no OnlineVerify mode.

ScatterMultiThread.exe -default -OFFLINEVERIFY -FILE:timer.dat -FULLCHECK:FALSE -ALLOWZERO4VERIFY

Offline verification of the above variation with fullcheck disabled and AllowZero4Verify mode.

 

Command syntax

Command option Description

ScatterMultiThread.exe

The command line options for the test are listed below.

-help or /?

Displays help.

-default

Runs the default variation with test parameters from the command-line.

-stress

Stress mode. Retries the memory allocation failures, etc.

-regression

Regression mode. Runs the predefined regression mode.

-testpath

Full path to the file.

Default value: CWD

-file

File in the testpath or full path to the file.

Default value: Scatter.dat

-filesize

File size in bytes. You can also append k, m, g, or t.

Default value: 1m

-threads

Number of exclusive reader and writer threads.

Default value: 2

-completionthreads

The number of completion threads.

Default value: Twice the number of processors

-rwmode

Read/write mode:

  • RW( Read, Write)

  • RWG( Read, Write Gather)

  • RSW( Read Scatter, Write)

  • RSWG( Read Scatter, Write Gather)

  • RRW( Random Read, Write)

  • RRWG( Random Read, Write Gather)

  • RWR( Read, Random Write)

  • RSWR( Read Scatter, Random Write)

  • RRWR( Random Read, Random Write – Approximately 50% Scatter/Gather) -> Default

  • RRWR7( Random Read, Random Write - Approximately 30% Scatter/Gather)

  • RRWR3( Random Read, Random Write - Approximately 70% Scatter/Gather

-iomode

sync or async

Default value: async

-flagsauto

TRUE or FALSE

Default value: TRUE

Note  

Certain file flags are automatically managed based on the situations

 

-ffnb

TRUE or FALSE

Default value: FALSE

Note  

FILE_FLAG_NO_BUFFERING will be automatically turned on in certain situations like Scatter/Gather.

 

-ffss

TRUE or FALSE

Default value: FALSE

Note  

FILE_FLAG_SEQUENTIAL_SCAN will be automatically turned on in certain situations like serialized I/O.

 

-ffra

TRUE or FALSE

Default value: FALSE

Note  

FILE_FLAG_RANDOM_ACCESS will be automatically turned on in certain situations like random I/O.

 

-ffwt

TRUE or FALSE

Default value: FALSE

Note  

FILE_FLAG_WRITE_THROUGH will not be turned on automatically, unless manually specified.

Disable FLAGSAUTO flag for independent management of file flags and attributes.

FLAGSAUTO behavior will change depending on other parameters such as shared file handle or reverse I/O, etc.

The FFSS and FFRA flags have no effect if the file system does not support cached I/O and FFNB is not supported by the file system or specified.

FFSS and FFRA flags are mutually exclusive and cannot be combined as it's self-defeating.

 

-bs

Block Size in bytes. You can append k, m, g, t.

Default: 64k

-bypass

Bypasses the I/O Threshold wait. This can be either TRUE or FALSE.

Default value: FALSE

-nsegments

The segment array size for Scatter/Gather.

Default value: 16

Note  

This option automatically gets corrected in certain situations for Scatter/Gather/

 

-totalsleep

The total sleep time for failed virtual allocation or I/O threshold scenarios.

Default value: 30 minutes

-sleepint

The loop sleep interval for failed virtual allocation scenarios

Default value: 2 minutes

-iodifferupperthreshold

The upper threshold for issuing I/O requests.

Default value: 1000

-iodifflowerthreshold

The lower threshold to resume I/O requested after I/O diff falls below this number.

Default value: 100

Note  

This option is mutually exclusive with bypass mode and is ignored if bypass mode is on.

 

-wfw

The writer(WFW) flag waits for WriterThreads to be done before any read happens. This option can be either true or false.

Default value: FALSE

-rr

The random offset read (RR) option makes ReaderThreads starts at a random offset. This value can be either true or false.

Default value: TRUE

-rw

The random offset write (RW) option makes WriterThreads start at random offset. This value can be either true of false.

Default value: FALSE

-rer

The reverse read from starting offset. This value can be either 0, 1, or 2.

Default value: 2

-rew

The reverse write from starting offset. This value can be either 0, 1, or 2.

Default value: 0

Note  

For RER and REW flags, 0 means no reversal, 1 means straight reversal, and 2 means random reversal (not applied always).

For RER and REW flags, full file I/O may not complete and the statistics and results reported might be inaccurate.

 

-timer

Timer mode (in seconds). You can append s, m, h.

Default value: 0

This performs I/O until the timer has expired.

-timerloop

The timer loop interval (in seconds). You can append s, m, h.

Default value: same as the timer option

Note  

The timer option is only for issuing requests and the completion will still go through (no timer is applicable for completion).

 

-onlineverify

Verifies the contents of the file after issuing write requests in online mode.

Default value: TRUE

-offlineverify

Verifies the contents of the file in offline mode by rerunning the test again.

Default value: FALSE

-allowzero4verify

Allows a zero to be part of the full verify (OFFLINE or ONLINE).

Default value: FALSE

-onlyzero4verify

Allows a zero to be part of the full verify(OFFLINE or ONLINE).

Default value: FALSE

-sharedfh

Allows the same file handle to be used for all I/O.

Default value: FALSE

-regression

The regression variation. This value can be either – 1 - 10, or 1001(refers to all variations)

Default value: 0 (no regression)

-displayprogress

Display I/O progress (in seconds), You can append s, m, h,

Default value: 2m

-fullcheck

Full verification of the data after the online or offline verify.

Default value: true

-dbgcheck

Breaks into a kernel debugger on failure. This value can be either true or false.

Default value: true

-toleratesurpriseremove

Do not break into a kernel debugger on a SurpriseRemove failure. This value can be either true or false.

Default value: false

-toleraterepair

Do not break into a kernel debugger on a repair failure. This value can be either true or false.

Default value: false

-toleratefailio

Do not break into a kernel debugger on a FailIO failure. This value can be either true or false.

Default value: false

-exitonerror

Exit on a failure. This value can be either true or false.

Default value: False

 

Note  

For command-line help for this test, type /?.

 

File list

File Location

ScatterMultiThread.exe

<[testbinroot]>\NTTest\BASETEST\core_file_services\NTFS\ScatterGather\

ntlog.dll

<[osbinroot]>\NTTest\CommonTest\NtLog\

ntlogger.ini

<[osbinroot]>\NTTest\CommonTest\NtLog\

stresslog.dll

<[osbinroot]>\basetest\core_file_services\shared_libs\

fbslog.dll

<[osbinroot]>\basetest\core_file_services\shared_libs\fbslog\

 

 

 

Send comments about this topic to Microsoft