Extended camera controls

Extended controls use the KSPROPERTY mechanism to expose camera controls to the application.

The following list of standardized extended controls (defined by Media Foundation) enable additional Windows camera features:

Some of the controls are exposed to applications as asynchronous controls and others are exposed as synchronous controls.

  • Synchronous controls – For these controls, the capture pipeline issues a KSPROPERTY camera control structure and expects the driver to synchronously return the request.

  • Asynchronous controls – For these controls, the capture pipeline issues a KSPROPERTY, enables a KSEVENT associated with that property, and waits on the event to get signaled. The driver must complete the KSPROPERTY synchronously and use that as a trigger to start the asynchronous operation. Upon the completion of the asynchronous operation, the driver must signal the associated KSEVENT on which the capture pipeline is waiting on. The capture pipeline notifies the application about the completion of the operation when it receives the signal.

    If an asynchronous control can be cancelled, it must specify the flag KSCAMERA_EXTENDEDPROP_FLAG_CANCELOPERATION in the control. If the control cannot be cancelled, the control's operation must not exceed 5 ms.

These extended controls are part of the following KS property set defined in ksmedia.h:

#define STATIC_KSPROPERTYSETID_ExtendedCameraControl \
     0x1CB79112, 0xC0D2, 0x4213, 0x9C, 0xA6, 0xCD, 0x4F, 0xDB, 0x92, 0x79, 0x72
DEFINE_GUIDSTRUCT("1CB79112-C0D2-4213-9CA6-CD4FDB927972", KSPROPERTYSETID_ExtendedCameraControl);
#define KSPROPERTYSETID_ExtendedCameraControl DEFINE_GUIDNAMED(KSPROPERTYSETID_ExtendedCameraControl);

Metadata

To retrieve metadata, the user mode component (DevProxy) must query the driver for the metadata buffer requirement. Once the user mode component has this information, it allocates the appropriate metadata buffer for the driver to fill and return back to the user mode component.

The KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA property ID that is defined in the KSPROPERTY_CAMERACONTROL_EXTENDED_PROPERTY enumeration is used by the client to query for the metadata buffer requirements, such as required metadata size, memory alignment requirements, and desired memory allocation type, for metadata buffer allocation.

Once user mode component has obtained the metadata buffer requirements from the driver, it allocates the appropriately sized metadata buffer with the desired memory alignment from the desired memory pool. This metadata buffer, along with the actual frame buffer, will be sent to the driver to fulfill and then returned back to the user mode component when filled. For multishot scenarios, a corresponding metadata buffer is allocated and delivered to the camera driver for each frame buffer allocated.

The KSSTREAM_METADATA_INFO structure, along with the following flag, is used to send the metadata buffer to the driver.

#define KSSTREAM_HEADER_OPTIONSF_METADATA           0x00001000

Once the buffer (metadata + frame) is queued to the driver, DevProxy sends a standard KSSTREAM_HEADER structure, followed by a KS_FRAME_INFO structure, and followed by a KSSTREAM_METADATA_INFO structure. DevProxy will further mask KSSTREAM_HEADER.OptionFlags with KSSTREAM_HEADER_OPTIONSF_METADATA before it passes the buffer down to the driver.

If the driver does not support metadata, or if KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA is not implemented, the KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA property control will fail. In this case, DevProxy will not allocate a metadata buffer and the payload that is passed down to the driver from DevProxy will not contain the KSSTREAM_METADATA_INFO structure.

If the driver supports metadata and the client does not want any metadata, DevProxy will neither allocate metadata buffer nor pass down KSSTREAM_METADATA_INFO when sending the buffer to the driver. This behavior reduces the unnecessary metadata memory allocation if an app does not want metadata on a given pin.

The following structures describe the layout of the metadata items to be filled by the camera driver in the metadata buffer.

The list below contains the layout of a metadata item. This must be 8-byte aligned.

The photo confirmation metadata is identified by MetadataId_PhotoConfirmation. When present, it indicates that the preview frame associated is a photo confirmation frame. Photo confirmation metadata is parsed by the DevProxy.

The custom metadata is identified by a MetadataId that starts from MetadataId_Custom_Start. The custom metadata item can contain a blob of metadata that can be a focus state and/or faces detected for the preview pin, EXIF and/or the OEM metadata for an image pin. The exact format of the custom blob is determined by the OEM who implements the driver and MFT0. The MFT0 is responsible for parsing the custom blob and attaching each metadata item as an attribute grouped under the MFSampleExtension_CaptureMetadata attribute bag in a format that is readable by the MF capture pipeline and/or WinRT.

The following IMFAttributes are defined in mfapi.h. These are required by the MF capture pipeline and/or WinRT. Note that MFT0 does not set any IMFAttributes for photo confirmation since the photo confirmation frame will not flow beyond DevProxy.

Attribute (GUID) Data type
MF_CAPTURE_METADATA_FOCUSSTATE UINT32
MF_CAPTURE_METADATA_FACEROIS Blob
MF_CAPTURE_METADATA_FRAME_RAWSTREAM IUnknown

The MF_CAPTURE_METADATA_FRAME_RAWSTREAM IMFAttributes are created and attached to MFSampleExtension_CaptureMetadata by the DevProxy, which contains a pointer to the IMFMediaBuffer interface associated with the raw metadata buffer (KSSTREAM_METADATA_INFO.Data). When the MFT0 receives an IMFSample, it gets the raw metadata buffer from the MF_CAPTURE_METADATA_FRAME_RAWSTREAM and parses any additional custom metadata items such as focus state and converts them into corresponding IMFAttributes defined above and attaches them to the MFSampleExtension_CaptureMetadata attribute bag.

To access the raw metadata buffer, the MFT0 does the following:

  1. Calls GetUnknown on MFSampleExtension_CaptureMetadata from the IMFSample interface to get the IMFAttributes interface for the attribute bag.

  2. Calls GetUnknown on MF_CAPTURE_METADATA_FRAME_RAWSTREAM from the IMFAttributes interface obtained from the previous step to get the IMFMediaBuffer interface.

  3. Calls Lock to get the raw metadata buffer associated with IMFMediaBuffer.

To add the required IMFAttributes to the MFSampleExtension_CaptureMetadata attribute bag, the MFT0 does the following:

  1. Calls GetUnknown on MFSampleExtension_CaptureMetadata from the IMFSample interface to get the IMFAttributes interface for the attribute bag.

  2. Calls SetUINT32, SetBlob, or SetUnknown on MF_CAPTURE_METADATA_XXX from the IMFAttributes interface obtained from the previous step based on the GUID and data type specified in the table above.

Focus priority

The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSPRIORITY property ID is the only control that is associated with the focus priority DDI.

Focus state

The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSSTATE property ID is the only control that is associated with the focus state DDI.

Extended region of interest ROI

The following property IDs are the controls that are associated with the ROI DDI:

Photo confirmation

The KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOCONFIRMATION property ID is the only control that is associated with the photo confirmation DDI.

Photo sequence submode

The KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOMODE property ID is the only control associated with the photo sequence DDI.

Photo capture feedback applied device settings

The MFT0 parses the metadata buffer provided by the driver and attaches the required applied device settings IMFAttributes to the MFSampleExtension_CaptureMetadata attribute bag associated with each IMFSample. The following IMFAttributes must be carried over by the MF pipeline and any third-party supplied MFTs:

Name Type
MFSampleExtension_CaptureMetadata IUnknown (IMFAttributes)
MFSampleExtension_EOS UINT32 (Boolean)
MFSampleExtension_PhotoThumbnail IUnknown (IMFMediaBuffer)
MFSampleExtension_PhotoThumbnailMediaType IUnknown (IMFMediaType)

Mandatory metadata attributes

MFSampleExtension_CaptureMetaData is a metadata attribute bag that can have the following attributes:

Name Type Description
MF_CAPTURE_METADATA_REQUESTED_FRAME_SETTING_ID UINT32 This attribute contains the frame ID for the corresponding frame in the variable photo sequence. This attribute is only set for a variable photo sequence capture.
MF_CAPTURE_METADATA_EXPOSURE_TIME UINT64 This attribute contains the exposure time applied in 100 nanoseconds.
MF_CAPTURE_METADATA_EXPOSURE_COMPENSATION Blob This attribute contains an EV compensation step flag and an EV compensation value in units of the step that was applied to the driver when the photo was captured. The CapturedMetadataExposureCompensation data structure describes the blob format for this attribute only. The metadata item structure format for EV compensation (KSCAMERA_METADATA_ITEMHEADER + EV compensation metadata payload) is provided by the driver and must be 8 byte aligned.
MF_CAPTURE_METADATA_ISO_SPEED UINT32 This attribute contains the ISO speed value applied as an integer.
MF_CAPTURE_METADATA_LENS_POSITION UINT32 This attribute contains the logical lens position when focus was applied to the photo captured. This value does not have a specific unit.
MF_CAPTURE_METADATA_SCENE_MODE UINT64 This attribute contains the scene mode applied as a UINT64KSCAMERA_EXTENDEDPROP_SCENEMODE_XXX flag.
MF_CAPTURE_METADATA_FLASH

UINT32

(Boolean)

This attribute contains a Boolean value that contains the flash state. A value of 1 specifies that the flash is on and a value of 0 specifies that the flash is off for the photo captured.
MF_CAPTURE_METADATA_FLASH_POWER

UINT32

This attribute contains the flash power applied as a percentage value between 0 and 100.
MF_CAPTURE_METADATA_WHITEBALANCE

UINT32

(Kelvin)

This attribute contains the white balance applied as a value in kelvin.
MF_CAPTURE_METADATA_ZOOMFACTOR

UINT32

(Q16)

This attribute contains the zoom value applied and is the same value that can be queried from KSPROPERTY_CAMERACONTROL_EXTENDED_ZOOM in a GET call. The value must be in Q16.
MF_CAPTURE_METADATA_FRAME_ILLUMINATION UINT64

The MF_CAPTURE_METADATA_FRAME_ILLUMINATION attribute for IR cameras specifies whether the frames are using active IR illumination and should be used in conjunction with FACEAUTH_MODE_ALTERNATIVE_FRAME_ILLUMINATION. It is only used for IR samples and should not be present on RGB frames if the camera supports both IR and color samples.

The value should be set to 0xXXXXXXXXXXXXXXX1 if the frame was captured when active illumination was on and set to 0xXXXXXXXXXXXXXXX0 if no illumination was present when capturing the frame.

Any Custom GUID Any variant type This attribute contains the custom data associated with the custom GUID.

EXIF and HW JPEG encoder

The pipeline is not required to process or warp any EXIF data for the HW JPEG encoder; therefore, the EXIF data format is provided by the driver, MFT0, and OEM HW JPEG encoder. OEMs partners can define any custom attribute GUID and variant type for the EXIF attribute and attach it to the MFSampleExtension_CaptureMetaData attribute bag for it to be consumed by the OEM components. If a HW JPEG encoder is available, the pipeline photo sink component will load the HW JPEG encoder and set the EXIF data held in the MFSampleExtension_CaptureMetaData attribute bag onto the HW JPEG encoder as an EXIF encoder option using the IPropertyBag2::Write method.

The encoder option property bag contains an array of PROPBAG2 structures that specify the available encoding option properties. The EXIF encoder option set onto HW JPEG encoder is identified by the following property in the encoder option property bag:

Property name VARTYPE Value Applicable codecs
SampleMetaData VT_UNKNOWN Pointer to an IMFAttributes interface for MFSampleExtension_CaptureMetaData attribute bag that contains an OEM sub attribute containing the EXIF data. JPEG

The MFSampleExtension_CaptureMetaData attribute bag can only contain any OEM defined EXIF sub attribute that the MFT0 and HW JPEG encoder can read to hold the EXIF data. To pass EXIF data from the driver to the HW JPEG encoder, the driver and MFT0 must do the following:

  1. The driver provides custom EXIF metadata in the metadata buffer supplied by the pipeline. This is attached to MFSampleExtension_CaptureMetadata as an MF_CAPTURE_METADATA_FRAME_RAWSTREAM IMFAttribute by DevProxy when the sample is returned back to DevProxy.

  2. When the MFT0 receives an IMFSample, it gets the raw metadata buffer from MF_CAPTURE_METADATA_FRAME_RAWSTREAM and parses the custom EXIF metadata item and converts it to an OEM defined IMFAttribute and attaches it to the MFSampleExtension_CaptureMetadata attribute bag.

To pass EXIF data from the MFT0 to the HW JPEG encoder, the pipeline photo sink does the following:

  1. Calls GetUnknown on MFSampleExtension_CaptureMetadata from IMFSample to get the IMFAttributes interface for the attribute bag when IMFSample is received.

  2. Calls IPropertyBag2::Write to set the encoder option property, identified by SampleMetadata, to the HW JPEG encoder. The encoder option property contains the IMFAttributes interface obtained from the previous step. This interface contains all custom sub attributes including the OEM EXIF sub attribute, and the standardized sub attributes in the Metadata section discussed earlier in this topic.

To retrieve the EXIF data for further processing, the HW JPEG encoder does the following:

  1. Calls IPropertyBag2::Read to retrieve the property value for the property identified by the SampleMetadata property name and VT_UNKNOWN type. When returned, the VARIANT.punkVal receives the IMFAttributes interface for MFSampleExtension_CaptureMetadata.

  2. Calls GetBlob or GetUnknown on the OEM EXIF sub attribute from the interface obtained from the previous step to get the EXIF data blob based on the GUID and data type of the OEM EXIF sub attribute.

Thumbnail

MFT0 is not required to produce any thumbnail for the camera driver. The camera app is expected to generate its own thumbnail. The thumbnail could be produced from the photo confirmation image, HW JPEG encoder, or resizing a full size image. This is up to the app developers. To maintain API and app compatibility with Windows 8.1 apps, the camera driver must not implement the KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOTHUMBNAIL control.

Integer ISO

The KSPROPERTY_CAMERACONTROL_EXTENDED_ISO_ADVANCED property ID is the only control associated with the integer ISO DDI.

Advanced focus

The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSMODE property ID is the only control associated with the integer ISO DDI.

Flash

The KSPROPERTY_CAMERACONTROL_EXTENDED_FLASHMODE property ID is the only control that is associated with the flash DDI.

Zoom

The KSPROPERTY_CAMERACONTROL_EXTENDED_ZOOM property ID is the only control that is associated with the zoom DDI.

Scene mode

The KSPROPERTY_CAMERACONTROL_EXTENDED_SCENEMODE property ID is the only control associated with the scene mode DDI.