Working with NVMe drives

Applies to:

  • Windows 10
  • Windows Server 2016

Learn how to work with high-speed NVMe devices from your Windows application. Device access is enabled via StorNVMe.sys, the in-box driver first introduced in Windows Server 2012 R2 and Windows 8.1. It's also available to Windows 7 devices through a KB hot fix. In Windows 10, several new features were introduced, including a pass-through mechanism for vendor-specific NVMe commands and updates to existing IOCTLs.

This topic provides an overview of general-use APIs that you can use to access NVMe drives in Windows 10. It also describes:

APIs for working with NVMe drives

You can use the following general-use APIs to access NVMe drives in Windows 10. These APIs can be found in winioctl.h for user mode applications, and ntddstor.h for kernel mode drivers. For more information about header files, see Header files.

  • IOCTL_STORAGE_PROTOCOL_COMMAND : Use this IOCTL with the STORAGE_PROTOCOL_COMMAND structure to issue NVMe commands. This IOCTL enables NVMe pass-through and supports the Command Effects log in NVMe. You can use it with vendor-specific commands. For more info, see Pass-through mechanism.

  • STORAGE_PROTOCOL_COMMAND : This input-buffer structure includes a ReturnStatus field that can be used report the following status values.

    • STORAGE_PROTOCOL_STATUS_PENDING
    • STORAGE_PROTOCOL_STATUS_SUCCESS
    • STORAGE_PROTOCOL_STATUS_ERROR
    • STORAGE_PROTOCOL_STATUS_INVALID_REQUEST
    • STORAGE_PROTOCOL_STATUS_NO_DEVICE
    • STORAGE_PROTOCOL_STATUS_BUSY
    • STORAGE_PROTOCOL_STATUS_DATA_OVERRUN
    • STORAGE_PROTOCOL_STATUS_INSUFFICIENT_RESOURCES
    • STORAGE_PROTOCOL_STATUS_NOT_SUPPORTED
  • IOCTL_STORAGE_QUERY_PROPERTY : Use this IOCTL with the STORAGE_PROPERTY_QUERY structure to retrieve device information. For more info, see Protocol-specific queries and Temperature queries.

  • STORAGE_PROPERTY_QUERY : This structure includes the PropertyId and AdditionalParameters fields to specify the data to be queried. In the PropertyId filed, use the STORAGE_PROPERTY_ID enumeration to specify the type of data. Use the AdditionalParameters field to specify more details, depending on the type of data. For protocol-specific data, use the STORAGE_PROTOCOL_SPECIFIC_DATA structure in the AdditionalParameters field. For temperature data, use the STORAGE_TEMPERATURE_INFO structure in the AdditionalParameters field.

  • STORAGE_PROPERTY_ID : This enumeration includes new values that allow IOCTL_STORAGE_QUERY_PROPERTY to retrieve protocol-specific and temperature information.

    • StorageAdapterProtocolSpecificProperty
    • StorageDeviceProtocolSpecificProperty

    Use one of these protocol-specific property IDs in combination with STORAGE_PROTOCOL_SPECIFIC_DATA to retrieve protocol-specific data in the STORAGE_PROTOCOL_DATA_DESCRIPTOR structure.

    • StorageAdapterTemperatureProperty
    • StorageDeviceTemperatureProperty

    Use one of these temperature property IDs to retrieve temperature data in the STORAGE_TEMPERATURE_DATA_DESCRIPTOR structure.

  • STORAGE_PROTOCOL_SPECIFIC_DATA : Retrieve NVMe-specific data when this structure is used for the AdditionalParameters field of STORAGE_PROPERTY_QUERY and a STORAGE_PROTOCOL_NVME_DATA_TYPE enum value is specified. Use one of the following STORAGE_PROTOCOL_NVME_DATA_TYPE values in the DataType field of the STORAGE_PROTOCOL_SPECIFIC_DATA structure:

    • Use NVMeDataTypeIdentify to get Identify Controller data or Identify Namespace data.
    • Use NVMeDataTypeLogPage to get log pages (including SMART/health data).
    • Use NVMeDataTypeFeature to get features of the NVMe drive.
  • STORAGE_TEMPERATURE_INFO : This structure is used to hold specific temperature data. It's used in the STORAGE_TEMERATURE_DATA_DESCRIPTOR to return the results of a temperature query.

  • IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD : Use this IOCTL with the STORAGE_TEMPERATURE_THRESHOLD structure to set temperature thresholds. For more info, see Behavior changing commands.

  • STORAGE_TEMPERATURE_THRESHOLD : This structure is used as an input buffer to specify the temperature threshold. The OverThreshold field (boolean) specifies if the Threshold field is the over threshold value or not (otherwise, it's the under threshold value).

Pass-through mechanism

Commands which are not defined in the NVMe specification are the most difficult for the host OS to handle – the host has no insight into the effects that the commands may have on the target device, the exposed infrastructure (namespaces/block sizes), and its behavior.

To better carry such device specific commands through the Windows storage stack, a new pass-through mechanism allows vendor-specific commands to be piped through. This pass-through pipe will also aid in development of management and testing tools. However, this pass-through mechanism requires use of the Command Effects Log. Moreover, StoreNVMe.sys requires all commands, not just pass-through commands, to be described in the Command Effects Log.

[!Important]
StorNVMe.sys and Storport.sys will block any command to a device if it is not described in the Command Effects Log.

 

Supporting the Command Effects Log

The Command Effects Log (as described in Commands Supported and Effects, section 5.10.1.5 of NVMe Specification 1.2) allows the description of the effects of vendor-specific commands together with specification-defined commands. This facilitates both command support validation as well as command behavior optimization, and therefore should be implemented for the entire set of commands that the device supports. The following conditions describe the result on how the command is sent based on its Command Effects Log entry.

For any specific command described in the Command Effects Log...

While:

  • Command Supported (CSUPP) is set to ‘1’ signifying that the command is supported by the controller (Bit 01)

    Note

    When CSUPP is set to ‘0’ (signifying that the command is not supported) the command will be blocked

     

And if any of the following is set:

  • Controller Capability Change (CCC) is set to ‘1’ signifying that the command may change controller capabilities (Bit 04)

  • Namespace Inventory Change (NIC) is set to ‘1’ signifying that the command may change the number, or capabilities for multiple namespaces (Bit 03)

  • Namespace Capability Change (NCC) is set to ‘1’ signifying that the command may change the capabilities of a single namespace (Bit 02)

  • Command Submission and Execution (CSE) is set to 001b or 010b, signifying that the command may be submitted when there is no other outstanding command to the same or any namespace, and that another command should not be submitted to the same or any namespace until this command is complete (Bits 18:16)

Then the command will be sent as the only command outstanding to the adapter.

Else if:

  • Command Submission and Execution (CSE) is set to 001b, signifying that the command may be submitted when there is no other outstanding command to the same namespace, and that another command should not be submitted to the same namespace until this command is complete (Bits 18:16)

Then the command will be sent as the only command outstanding to the Logical Unit Number object (LUN).

Otherwise, the command is sent with other commands outstanding without inhibition. For example, if a vendor-specific command is sent to the device to retrieve statistical information that is not spec-defined, there should be no risk to changing the device’s behavior or capability to execute I/O commands. Such requests could be serviced in parallel to I/O and no pause-resume would be necessary.

Using IOCTL_STORAGE_PROTOCOL_COMMAND to send commands

Pass-through can be conducted using the IOCTL_STORAGE_PROTOCOL_COMMAND, introduced in Windows 10. This IOCTL was designed to have a similar behavior as the existing SCSI and ATA pass-through IOCTLs, to send an embedded command to the target device. Via this IOCTL, pass-through can be sent to a storage device, including an NVMe drive.

For example, in NVMe, the IOCTL will allow the sending down of the following command codes.

  • Vendor Specific Admin Commands (C0h – FFh)
  • Vendor Specific NVMe Commands (80h – FFh)

As with all other IOCTLs, Use DeviceIoControl to send the pass-through IOCTL down. The IOCTL is populated using the STORAGE_PROTOCOL_COMMAND input-buffer structure found in ntddstor.h. Populate the Command field with the vendor-specific command.

typedef struct _STORAGE_PROTOCOL_COMMAND {

    ULONG   Version;                        // STORAGE_PROTOCOL_STRUCTURE_VERSION
    ULONG   Length;                         // sizeof(STORAGE_PROTOCOL_COMMAND)

    STORAGE_PROTOCOL_TYPE  ProtocolType;
    ULONG   Flags;                          // Flags for the request

    ULONG   ReturnStatus;                   // return value
    ULONG   ErrorCode;                      // return value, optional

    ULONG   CommandLength;                  // non-zero value should be set by caller
    ULONG   ErrorInfoLength;                // optional, can be zero
    ULONG   DataToDeviceTransferLength;     // optional, can be zero. Used by WRITE type of request.
    ULONG   DataFromDeviceTransferLength;   // optional, can be zero. Used by READ type of request.

    ULONG   TimeOutValue;                   // in unit of seconds

    ULONG   ErrorInfoOffset;                // offsets need to be pointer aligned
    ULONG   DataToDeviceBufferOffset;       // offsets need to be pointer aligned
    ULONG   DataFromDeviceBufferOffset;     // offsets need to be pointer aligned

    ULONG   CommandSpecific;                // optional information passed along with Command.
    ULONG   Reserved0;

    ULONG   FixedProtocolReturnData;        // return data, optional. Some protocol, such as NVMe, may return a small amount data (DWORD0 from completion queue entry) without the need of separate device data transfer.
    ULONG   Reserved1[3];

    _Field_size_bytes_full_(CommandLength) UCHAR Command[ANYSIZE_ARRAY];

} STORAGE_PROTOCOL_COMMAND, *PSTORAGE_PROTOCOL_COMMAND;

The vendor specific command desired to be sent should be populated in the highlighted field above. Note again that the Command Effects Log must be implemented for pass-through commands. In particular, these commands need to be reported as supported in the Command Effects Log (see previous section for more information). Also note that PRP fields are driver specific thus applications sending commands can leave them as 0.

Finally, this pass-through IOCTL is intended for sending vendor-specific commands. To send other admin or non-vendor specific NVMe commands such as Identify, this pass-through IOCTL should not be used. For example, IOCTL_STORAGE_QUERY_PROPERTY should be used for Identify or Get Log Pages. For more info, see the next section, Protocol-specific queries.

Don't update firmware through the pass-through mechanism

Firmware download and activation commands should not be sent using pass-through. IOCTL_STORAGE_PROTOCOL_COMMAND should only be used for vendor-specific commands.

Instead, use the following general storage IOCTLs (introduced in Windows 10) to avoid applications directly using the SCSI_miniport version of the Firmware IOCTL. Storage drivers will translate the IOCTL to either a SCSI command or the SCSI_miniport version of the IOCTL to the miniport.

These IOCTLs are recommended for developing firmware upgrade tools in Windows 10 and Windows Server 2016:

For getting storage information and updating firmware, Windows also supports PowerShell cmdlets for doing this quickly:

  • Get-StorageFirmwareInfo
  • Update-StorageFirmware

Note

To update firmware on NVMe in Windows 8.1, use IOCTL_SCSI_MINIPORT_FIRMWARE. This IOCTL was not backported to Windows 7. For more information, see Upgrading Firmware for an NVMe Device in Windows 8.1.

 

Returning errors through the pass-through mechanism

Similar to SCSI and ATA pass-through IOCTLs, when a command/request is sent to the miniport or device, the IOCTL returns if it was successful or not. In the STORAGE_PROTOCOL_COMMAND structure, the IOCTL returns the status through the ReturnStatus field.

Example: sending a vendor-specific command

In this example, an arbitrary vendor-specific command (0xFF) is sent via pass-through to an NVMe drive. The following code allocates a buffer, initializes a query, and then sends the command down to the device via DeviceIoControl.

    ZeroMemory(buffer, bufferLength);  
    protocolCommand = (PSTORAGE_PROTOCOL_COMMAND)buffer;  

    protocolCommand->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;  
    protocolCommand->Length = sizeof(STORAGE_PROTOCOL_COMMAND);  
    protocolCommand->ProtocolType = ProtocolTypeNvme;  
    protocolCommand->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;  
    protocolCommand->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;  
    protocolCommand->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);  
    protocolCommand->DataFromDeviceTransferLength = 4096;  
    protocolCommand->TimeOutValue = 10;  
    protocolCommand->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;  
    protocolCommand->DataFromDeviceBufferOffset = protocolCommand->ErrorInfoOffset + protocolCommand->ErrorInfoLength;  
    protocolCommand->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_ADMIN_COMMAND;  

    command = (PNVME_COMMAND)protocolCommand->Command;  

    command->CDW0.OPC = 0xFF;  
    command->u.GENERAL.CDW10 = 0xto_fill_in;  
    command->u.GENERAL.CDW12 = 0xto_fill_in;  
    command->u.GENERAL.CDW13 = 0xto_fill_in;  

    //  
    // Send request down.  
    //  

    result = DeviceIoControl(DeviceList[DeviceIndex].Handle,  
                             IOCTL_STORAGE_PROTOCOL_COMMAND,  
                             buffer,  
                             bufferLength,  
                             buffer,  
                             bufferLength,  
                             &returnedLength,  
                             NULL 
                             );  

In this example, we expect protocolCommand->ReturnStatus == STORAGE_PROTOCOL_STATUS_SUCCESS if the command succeeded to the device.

Protocol-specific queries

Windows 8.1 introduced IOCTL_STORAGE_QUERY_PROPERTY for data retrieval. In Windows 10, the IOCTL was enhanced to support commonly requested NVMe features such as Get Log Pages, Get Features, and Identify. This allows for the retrieval of NVMe specific information for monitoring and inventory purposes.

The input buffer for the IOCTL, STORAGE_PROPERTY_QUERY (from Windows 10) is shown here.

typedef struct _STORAGE_PROPERTY_QUERY {
    STORAGE_PROPERTY_ID PropertyId;
    STORAGE_QUERY_TYPE QueryType;
    UCHAR  AdditionalParameters[1];
} STORAGE_PROPERTY_QUERY, *PSTORAGE_PROPERTY_QUERY;

When using IOCTL_STORAGE_QUERY_PROPERTY to retrieve NVMe protocol-specific information in the STORAGE_PROTOCOL_DATA_DESCRIPTOR, configure the STORAGE_PROPERTY_QUERY structure as follows:

The STORAGE_PROTOCOL_SPECIFIC_DATA structure (from Windows 10) is shown here.

typedef struct _STORAGE_PROTOCOL_SPECIFIC_DATA {

    STORAGE_PROTOCOL_TYPE ProtocolType;
    ULONG   DataType;                 

    ULONG   ProtocolDataRequestValue;
    ULONG   ProtocolDataRequestSubValue;

    ULONG   ProtocolDataOffset;         
    ULONG   ProtocolDataLength;

    ULONG   FixedProtocolReturnData;   
    ULONG   Reserved[3];

} STORAGE_PROTOCOL_SPECIFIC_DATA, *PSTORAGE_PROTOCOL_SPECIFIC_DATA;

To specify a type of NVMe protocol-specific information, configure the STORAGE_PROTOCOL_SPECIFIC_DATA structure as follows:

  • Set the ProtocolType field to ProtocolTypeNVMe.

  • Set the DataType field to an enumeration value defined by STORAGE_PROTOCOL_NVME_DATA_TYPE:

    • Use NVMeDataTypeIdentify to get Identify Controller data or Identify Namespace data.
    • Use NVMeDataTypeLogPage to get log pages (including SMART/health data).
    • Use NVMeDataTypeFeature to get features of the NVMe drive.

When ProtocolTypeNVMe is used as the ProtocolType, queries for protocol-specific information can be retrieved in parallel with other I/O on the NVMe drive.

The following examples demonstrate NVMe protocol-specific queries.

Example: NVMe Identify query

In this example, the Identify request is sent to an NVMe drive. The following code initializes the query data structure and then sends the command down to the device via DeviceIoControl.

    BOOL    result;
    PVOID   buffer = NULL;
    ULONG   bufferLength = 0;
    ULONG   returnedLength = 0;

    PSTORAGE_PROPERTY_QUERY query = NULL;
    PSTORAGE_PROTOCOL_SPECIFIC_DATA protocolData = NULL;
    PSTORAGE_PROTOCOL_DATA_DESCRIPTOR protocolDataDescr = NULL;

    //
    // Allocate buffer for use.
    //
    bufferLength = FIELD_OFFSET(STORAGE_PROPERTY_QUERY, AdditionalParameters) + sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA) + NVME_MAX_LOG_SIZE;
    buffer = malloc(bufferLength);

    if (buffer == NULL) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: allocate buffer failed, exit.\n"));
        goto exit;
    }

    //
    // Initialize query data structure to get Identify Controller Data.
    //
    ZeroMemory(buffer, bufferLength);

    query = (PSTORAGE_PROPERTY_QUERY)buffer;
    protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;
    protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;

    query->PropertyId = StorageAdapterProtocolSpecificProperty;
    query->QueryType = PropertyStandardQuery;

    protocolData->ProtocolType = ProtocolTypeNvme;
    protocolData->DataType = NVMeDataTypeIdentify;
    protocolData->ProtocolDataRequestValue = NVME_IDENTIFY_CNS_CONTROLLER;
    protocolData->ProtocolDataRequestSubValue = 0;
    protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);
    protocolData->ProtocolDataLength = NVME_MAX_LOG_SIZE;

    //
    // Send request down.
    //
    result = DeviceIoControl(DeviceList[Index].Handle,
                             IOCTL_STORAGE_QUERY_PROPERTY,
                             buffer,
                             bufferLength,
                             buffer,
                             bufferLength,
                             &returnedLength,
                             NULL
                             );

    ZeroMemory(buffer, bufferLength);
    query = (PSTORAGE_PROPERTY_QUERY)buffer;  
    protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;  
    protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;  

    query->PropertyId = StorageDeviceProtocolSpecificProperty;  
    query->QueryType = PropertyStandardQuery;  

    protocolData->ProtocolType = ProtocolTypeNvme;  
    protocolData->DataType = NVMeDataTypeLogPage;  
    protocolData->ProtocolDataRequestValue = NVME_LOG_PAGE_HEALTH_INFO;  
    protocolData->ProtocolDataRequestSubValue = 0;  
    protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);  
    protocolData->ProtocolDataLength = sizeof(NVME_HEALTH_INFO_LOG);  

    //  
    // Send request down.  
    //  
    result = DeviceIoControl(DeviceList[Index].Handle,  
                             IOCTL_STORAGE_QUERY_PROPERTY,  
                             buffer,  
                             bufferLength,  
                             buffer, 
                             bufferLength,  
                             &returnedLength,  
                             NULL  
                             );  

    //
    // Validate the returned data.
    //
    if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||
        (protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Identify Controller Data - data descriptor header not valid.\n"));
        return;
    }

    protocolData = &protocolDataDescr->ProtocolSpecificData;

    if ((protocolData->ProtocolDataOffset < sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA)) ||
        (protocolData->ProtocolDataLength < NVME_MAX_LOG_SIZE)) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Identify Controller Data - ProtocolData Offset/Length not valid.\n"));
        goto exit;
    }

    //
    // Identify Controller Data 
    //
    {
        PNVME_IDENTIFY_CONTROLLER_DATA identifyControllerData = (PNVME_IDENTIFY_CONTROLLER_DATA)((PCHAR)protocolData + protocolData->ProtocolDataOffset);

        if ((identifyControllerData->VID == 0) ||
            (identifyControllerData->NN == 0)) {
            _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Identify Controller Data not valid.\n"));
            goto exit;
        } else {
            _tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***Identify Controller Data succeeded***.\n"));
        }
    }

  

Note that the caller needs to allocate a single buffer containing STORAGE_PROPERTY_QUERY and the size of STORAGE_PROTOCOL_SPECIFIC_DATA. In this example, it’s using the same buffer for input and output from the property query. That’s why the buffer that was allocated has a size of “FIELD_OFFSET(STORAGE_PROPERTY_QUERY, AdditionalParameters) + sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA) + NVME_MAX_LOG_SIZE”. Although separate buffers could be allocated for both input and output, we recommend using a single buffer to query NVMe related-information.

Example: NVMe Get Log Pages query

In this example, based off of the previous one, the Get Log Pages request is sent to an NVMe drive. The following code prepares the query data structure and then sends the command down to the device via DeviceIoControl.

    ZeroMemory(buffer, bufferLength);  

    query = (PSTORAGE_PROPERTY_QUERY)buffer;  
    protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;  
    protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;  

    query->PropertyId = StorageDeviceProtocolSpecificProperty;  
    query->QueryType = PropertyStandardQuery;  

    protocolData->ProtocolType = ProtocolTypeNvme;  
    protocolData->DataType = NVMeDataTypeLogPage;  
    protocolData->ProtocolDataRequestValue = NVME_LOG_PAGE_HEALTH_INFO;  
    protocolData->ProtocolDataRequestSubValue = 0;  
    protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);  
    protocolData->ProtocolDataLength = sizeof(NVME_HEALTH_INFO_LOG);  

    //  
    // Send request down.  
    //  
    result = DeviceIoControl(DeviceList[Index].Handle,  
                             IOCTL_STORAGE_QUERY_PROPERTY,  
                             buffer,  
                             bufferLength,  
                             buffer, 
                             bufferLength,  
                             &returnedLength,  
                             NULL  
                             );  

    if (!result || (returnedLength == 0)) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log failed. Error Code %d.\n"), GetLastError());
        goto exit;
    }

    //
    // Validate the returned data.
    //
    if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||
        (protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log - data descriptor header not valid.\n"));
        return;
    }

    protocolData = &protocolDataDescr->ProtocolSpecificData;

    if ((protocolData->ProtocolDataOffset < sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA)) ||
        (protocolData->ProtocolDataLength < sizeof(NVME_HEALTH_INFO_LOG))) {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log - ProtocolData Offset/Length not valid.\n"));
        goto exit;
    }

    //
    // SMART/Health Information Log Data 
    //
    {
        PNVME_HEALTH_INFO_LOG smartInfo = (PNVME_HEALTH_INFO_LOG)((PCHAR)protocolData + protocolData->ProtocolDataOffset);

        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log Data - Temperature %d.\n"), ((ULONG)smartInfo->Temperature[1] << 8 | smartInfo->Temperature[0]) - 273);

        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***SMART/Health Information Log succeeded***.\n"));
    }

Example: NVMe Get Features query

In this example, based off of the previous one, the Get Features request is sent to an NVMe drive. The following code prepares the query data structure and then sends the command down to the device via DeviceIoControl.

    //  
    // Initialize query data structure to Volatile Cache feature.  
    //  

    ZeroMemory(buffer, bufferLength);  


    query = (PSTORAGE_PROPERTY_QUERY)buffer;  
    protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;  
    protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;  

    query->PropertyId = StorageDeviceProtocolSpecificProperty;  
    query->QueryType = PropertyStandardQuery;  

    protocolData->ProtocolType = ProtocolTypeNvme;  
    protocolData->DataType = NVMeDataTypeFeature;  
    protocolData->ProtocolDataRequestValue = NVME_FEATURE_VOLATILE_WRITE_CACHE;  
    protocolData->ProtocolDataRequestSubValue = 0;  
    protocolData->ProtocolDataOffset = 0;  
    protocolData->ProtocolDataLength = 0;  

    //  
    // Send request down.  
    //  

    result = DeviceIoControl(DeviceList[Index].Handle,  
                             IOCTL_STORAGE_QUERY_PROPERTY,  
                             buffer,  
                             bufferLength,  
                             buffer,  
                             bufferLength,  
                             &returnedLength,  
                             NULL  
                             );  

    if (!result || (returnedLength == 0)) {  
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache failed. Error Code %d.\n"), GetLastError());  
        goto exit;  
    }  

    //  
    // Validate the returned data.  
    //  

    if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||  
        (protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {  
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache  - data descriptor header not valid.\n"));  
        return;                                           
    }  

    //
    // Volatile Cache 
    //
    {
        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache - %x.\n"), protocolDataDescr->ProtocolSpecificData.FixedProtocolReturnData);

        _tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***Get Feature - Volatile Cache succeeded***.\n"));
    }

Temperature queries

In Windows 10, IOCTL_STORAGE_QUERY_PROPERTY can also be used to query temperature data from NVMe devices.

To retrieve temperature information from an NVMe drive in the STORAGE_TEMPERATURE_DATA_DESCRIPTOR, configure the STORAGE_PROPERTY_QUERY structure as follows:

  • Allocate a buffer that can contains a STORAGE_PROPERTY_QUERY structure.

  • Set the PropertyID field to StorageAdapterTemperatureProperty or StorageDeviceTemperatureProperty for a controller or device/namespace request, respectively.

  • Set the QueryType field to PropertyStandardQuery.

The STORAGE_TEMPERATURE_INFO structure (from Windows 10) is shown here.

typedef struct _STORAGE_TEMPERATURE_INFO {

    USHORT  Index;                      // Starts from 0. Index 0 may indicate a composite value.
    SHORT   Temperature;                // Signed value; in Celsius.
    SHORT   OverThreshold;              // Signed value; in Celsius.
    SHORT   UnderThreshold;             // Signed value; in Celsius.

    BOOLEAN OverThresholdChangable;     // Can the threshold value being changed by using IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD.
    BOOLEAN UnderThresholdChangable;    // Can the threshold value being changed by using IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD.
    BOOLEAN EventGenerated;             // Indicates that notification will be generated when temperature cross threshold.
    UCHAR   Reserved0;
    ULONG   Reserved1;

} STORAGE_TEMPERATURE_INFO, *PSTORAGE_TEMPERATURE_INFO;

Behavior changing commands

Commands that manipulate device attributes or potentially impact device behavior are more difficult for the operating system to deal with. If device attributes change at run-time while I/O is being processed, synchronization or data integrity issues can arise if not properly handled.

The NVMe Set-Features command is a good example of a behavior changing command. It allows for the changing of the arbitration mechanism and the setting of temperature thresholds. To ensure that in-flight data is not at risk when behavior-affecting set commands are sent down, Windows will pause all I/O to the NVMe device, drain queues, and flush buffers. Once the set command has executed successfully, I/O is resumed (if possible). If I/O cannot be resumed, a device reset may be required.

Setting temperature thresholds

Windows 10 introduced IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD, an IOCTL for getting and setting temperature thresholds. You can also use it to get the current temperature of the device. The input/output buffer for this IOCTL is the STORAGE_TEMPERATURE_INFO structure, from the previous code section.

Example: Setting over-threshold temperature

In this example, an NVMe drive's over-threshold temperature is set. The following code prepares the command and then sends it down to the device via DeviceIoControl.

    BOOL    result;  
    ULONG   returnedLength = 0;  
    
    STORAGE_TEMPERATURE_THRESHOLD setThreshold = {0};  

    setThreshold.Version = sizeof(STORAGE_TEMPERATURE_THRESHOLD); 
    setThreshold.Size = sizeof(STORAGE_TEMPERATURE_THRESHOLD);  
    setThreshold.Flags = STORAGE_TEMPERATURE_THRESHOLD_FLAG_ADAPTER_REQUEST;  
    setThreshold.Index = SensorIndex;  
    setThreshold.Threshold = Threshold;  
    setThreshold.OverThreshold = UpdateOverThreshold; 

    //  
    // Send request down.  
    //  

    result = DeviceIoControl(DeviceList[DeviceIndex].Handle,  
                             IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD,  
                             &setThreshold,  
                             sizeof(STORAGE_TEMPERATURE_THRESHOLD),  
                             NULL,  
                             0,  
                             &returnedLength,  
                             NULL  
                             ); 

Setting vendor-specific features

Without the Command Effects Log, the driver has no knowledge of the ramifications of the command. This is why the Command Effects Log is required. It helps the operating system determine if a command is high impact and if it can be sent in parallel with other commands to the drive.

The Command Effects Log is not yet granular enough to encompass vendor-specific Set-Features commands. For this reason, it is not yet possible to send vendor-specific Set-Features commands. However, it is possible to use the pass-through mechanism, discussed earlier, to send vendor-specific commands. For more info, see Pass-through mechanism.

Header files

The following files are relevant to NVMe development. These files are included with the Microsoft Windows Software Development Kit (SDK).

Header file Description
ntddstor.h Defines constants and types for accessing the storage class drivers from kernel mode.
nvme.h For other NVMe-related data-structures.
winioctl.h For overall Win32 IOCTL definitions, including storage APIs for user mode applications.