使用 MediaFrameSourceGroup 从多个来源捕获Capture from multiple sources using MediaFrameSourceGroup

本文介绍如何使用多个嵌入式视频轨同时从多种来源将视频捕获到一个文件。This article shows you how to capture video from multiple sources simultaneously to a single file with multiple embedded video tracks. 从 RS3 开始,可以为单个 MediaEncodingProfile 指定多个 VideoStreamDescriptor 对象。Starting with RS3, you can specify multiple VideoStreamDescriptor objects for a single MediaEncodingProfile. 这样可以同时将多个流编码到单个文件中。This enables you to encode multiple streams simultaneously to a single file. 在此操作中编码的视频流必须包含在当前设备上指定了一组可同时使用的摄像机的单个 MediaFrameSourceGroupThe video streams that are encoded in this operation must be included in a single MediaFrameSourceGroup which specifies a set of cameras on the current device that can be used at the same time.

有关如何结合使用 MediaFrameSourceGroupMediaFrameReader 类来启用多个摄像机的实时计算机视觉方案的信息,请参阅使用 MediaFrameReader 处理媒体帧For information on using MediaFrameSourceGroup with the MediaFrameReader class to enable real-time computer vision scenarios that use multiple cameras, see Process media frames with MediaFrameReader.

本文的其余部分将介绍使用多个视频轨将来自两台彩色摄像机的视频录制到单个文件的步骤。The rest of this article will walk you through the steps of recording video from two color cameras to a single file with multiple video tracks.

查找可用的传感器组Find available sensor groups

一个 MediaFrameSourceGroup 表示一系列可同时访问的帧源(通常是摄像机)。A MediaFrameSourceGroup represents a collection of frame sources, typically cameras, that can be accessed simulataneously. 每个设备的可用帧源组各不相同,因此本示例中的第一步是获取可用帧源组的列表,并查找包含应用场景所需摄像机的帧源组,在本例中需要两个彩色摄像机。The set of available frame source groups is different for each device, so the first step in this example is to get the list of available frame source groups and finding one that contains the necessary cameras for the scenario, which in this case requires two color cameras.

MediaFrameSourceGroup.FindAllAsync 方法返回当前设备上的所有可用源组。The MediaFrameSourceGroup.FindAllAsync method returns all source groups available on the current device. 返回的每个 MediaFrameSourceGroup 都具有一系列 MediaFrameSourceInfo 对象,这些对象描述了组中的每一个帧源。Each returned MediaFrameSourceGroup has a list of MediaFrameSourceInfo objects that describes each frame source in the group. Linq 查询用于查找包含两个彩色摄像机的源组,一个在前面板,一个在背面。A Linq query is used to find a source group that contains two color cameras, one on the front panel and one on the back. 对于每个彩色摄像机都将返回一个包含所选 MediaFrameSourceGroupMediaFrameSourceInfo 的匿名对象。An anonymous object is returned that contains the selected MediaFrameSourceGroup and the MediaFrameSourceInfo for each color camera. 不使用 Linq 语法,也可以循环查找每个组,然后循环查找每个 MediaFrameSourceInfo,找到满足要求的组。Instead of using Linq syntax, you could instead loop through each group, and then each MediaFrameSourceInfo to look for a group that meets your requirements.

请注意,不是每个设备都包含具有两个彩色相机的源组,因此在尝试捕获视频之前,应检查以确保找到这样的源组。Note that not every device will contain a source group that contains two color cameras, so you should check to make sure that a source group was found before trying to capture video.

var sensorGroups = await MediaFrameSourceGroup.FindAllAsync();

var foundGroup = sensorGroups.Select(g => new
{
    group = g,
    color1 = g.SourceInfos.Where(info => info.SourceKind == MediaFrameSourceKind.Color && info.DeviceInformation.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Front).FirstOrDefault(),
    color2 = g.SourceInfos.Where(info => info.SourceKind == MediaFrameSourceKind.Color && info.DeviceInformation.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Back).FirstOrDefault()
}).Where(g => g.color1 != null && g.color2 != null).FirstOrDefault();

if (foundGroup == null)
{
    Debug.WriteLine("No groups found.");
    return;
}

初始化 MediaCapture 对象Initialize the MediaCapture object

MediaCapture 类是在 UWP 应用中用于大多数音频、视频和照片捕获操作的主类。The MediaCapture class is the primary class that is used for most audio, video, and photo capture operations in UWP apps. 通过调用 InitializeAsync、传入包含初始化参数对象的 MediaCaptureInitializationSettings 对象来初始化对象。Initialize the object by calling InitializeAsync, passing in a MediaCaptureInitializationSettings object that contains initialization parameters. 在本示例中,唯一指定的设置是 SourceGroup 属性,它设置为在上面的示例代码中检索到的 MediaFrameSourceGroupIn this example, the only specified setting is the SourceGroup property, which is set to the MediaFrameSourceGroup that was retrieved in the previous code example.

有关 MediaCapture 和其他用于捕获媒体的 UWP 应用功能可以执行的其他操作的信息,请参阅摄像机For information on other operations you can perform with MediaCapture and other UWP app features for capturing media, see Camera.

var settings = new MediaCaptureInitializationSettings()
{
    SourceGroup = foundGroup.group
};

mediaCapture = new MediaCapture();
await mediaCapture.InitializeAsync(settings);

创建 MediaEncodingProfileCreate a MediaEncodingProfile

MediaEncodingProfile 类告诉媒体捕获管道在将捕获到的音频和视频写入文件时应如何对其进行编码。The MediaEncodingProfile class tells the media capture pipeline how captured audio and video should be encoded as they are written to a file. 对于典型的捕获和转换代码应用场景,此类提供一系列静态方法来创建常用配置文件,如 CreateAviCreateMp3For typical capture and transcoding scenarios, this class provides a set of static methods for creating common profiles, like CreateAvi and CreateMp3. 对于此例,编码配置文件可使用 Mpeg4 容器和 H264 视频编码手动创建。For this example, an encoding profile is manually created using an Mpeg4 container and H264 video encoding. 视频编码设置使用 VideoEncodingProperties 对象指定。Video encoding settings are specified using a VideoEncodingProperties object. 对于此应用场景中使用的每个彩色摄像机,都会配置一个 VideoStreamDescriptor 对象。For each color camera used in this scenario, a VideoStreamDescriptor object is configured. 该描述符使用指定编码的 VideoEncodingProperties 对象来构造。The descriptor is constructed with the VideoEncodingProperties object specifying the encoding. VideoStreamDescriptorLabel 属性必须设置为将捕获到流中的媒体帧源的 ID。The Label property of the VideoStreamDescriptor must be set to the ID of the media frame source that will be captured to the stream. 这样,捕获管道就能知道对于每个摄像机应该使用的流描述符和编码属性。This is how the capture pipeline knows which stream descriptor and encoding properties should be used for each camera. 帧源的 ID 由在上一节中已选择 MediaFrameSourceGroup 的情况下找到的 MediaFrameSourceInfo 对象公布。The ID of the frame source is exposed by the MediaFrameSourceInfo objects that were found in the previous section, when a MediaFrameSourceGroup was selected.

自 Windows 10 版本 1709 起,可通过调用 SetVideoTracksMediaEncodingProfile 设置多个编码属性。Starting with Windows 10, version 1709, you can set multiple encoding properties on a MediaEncodingProfile by calling SetVideoTracks. 可以通过调用 GetVideoTracks 获取视频流描述符列表。You can retrieve the list of video stream descriptors by calling GetVideoTracks. 请注意,如果设置存储单个流描述符的 Video 属性,则通过调用 SetVideoTracks 设置的描述符列表将替换为包含你指定的单个描述符的列表。Note that if you set the Video property, which stores a single stream descriptor, the descriptor list you set by calling SetVideoTracks will be replaced with a list containing the single descriptor you specified.

var profile = new MediaEncodingProfile();
profile.Container = new ContainerEncodingProperties();
profile.Container.Subtype = MediaEncodingSubtypes.Mpeg4;

List<VideoStreamDescriptor> streams = new List<VideoStreamDescriptor>();

var encodeProps = VideoEncodingProperties.CreateH264();
encodeProps.Subtype = MediaEncodingSubtypes.H264;
var stream1Desc = new VideoStreamDescriptor(encodeProps);
stream1Desc.Label = foundGroup.color1.Id;
streams.Add(stream1Desc);

var encodeProps2 = VideoEncodingProperties.CreateH264();
encodeProps2.Subtype = MediaEncodingSubtypes.H264;
var stream2Desc = new VideoStreamDescriptor(encodeProps2);
stream2Desc.Label = foundGroup.color2.Id;
streams.Add(stream2Desc);

profile.SetVideoTracks(streams);
profile.Audio = null;

在媒体文件中对计时元数据进行编码Encode timed metadata in media files

自 Windows 10 版本 1803 起,除了音频和视频,还可将计时元数据编码为支持该数据格式的媒体文件。Starting with Windows 10, version 1803, in addition to audio and video you can encode timed metadata into a media file for which the data format is supported. 例如,GoPro 元数据 (gpmd) 可存储在 MP4 文件中,传递与视频流相关的地理位置。For example, GoPro metadata (gpmd) can be stored in MP4 files to convey the geographic location correlated with a video stream.

进行元数据编码时使用的是与音频或视频编码相似的模式。Encoding metadata uses a pattern that is parallel to encoding audio or video. TimedMetadataEncodingProperties 类可描述元数据的类型、子类型和编码属性,类似于视频中 VideoEncodingProperties 的功能。The TimedMetadataEncodingProperties class describes the type, subtype and encoding properties of the metadata, like VideoEncodingProperties does for video. TimedMetadataStreamDescriptor 可标识元数据流,类似于视频流中 VideoStreamDescriptor 的功能。The TimedMetadataStreamDescriptor identifies a metadata stream, just as the VideoStreamDescriptor does for video streams.

以下示例演示了如何初始化 TimedMetadataStreamDescriptor 对象。The following example shows how to intialize a TimedMetadataStreamDescriptor object. 首先,创建 TimedMetadataEncodingProperties 对象,并将 Subtype 设置为用于识别将包含在流中的元数据类型的 GUID。First, a TimedMetadataEncodingProperties object is created and the Subtype is set to a GUID that identifies the type of metadata that will be included in the stream. 本示例使用 GoPro 元数据 (gpmd) 的 GUID。This example uses the GUID for GoPro metadata (gpmd). 通过调用 SetFormatUserData 方法设置格式特定的数据。The SetFormatUserData method is called to set format-specific data. 对于 MP4 文件,格式特定的数据存储在 SampleDescription 框 (stsd) 中。For MP4 files, the format-specific data is stored in the SampleDescription box (stsd). 然后,通过编码属性创建新的 TimedMetadataStreamDescriptorNext, a new TimedMetadataStreamDescriptor is created from the encoding properties. LabelName 属性设置为识别要编码的流。The Label and Name properties are set to identify the stream to be encoded.

           TimedMetadataEncodingProperties encodingProperties = new TimedMetadataEncodingProperties
           {
               Subtype = "{67706D64-BF10-48B4-BC18-593DC1DB950F}"
           };

           byte[] streamDescriptionData = GetStreamDescriptionDataForGpmdEncodingSubtype();
           encodingProperties.SetFormatUserData(streamDescriptionData);

           TimedMetadataStreamDescriptor descriptor = new TimedMetadataStreamDescriptor(encodingProperties)
           {
               Name = "GPS Info",
               Label = "GPS Info"
           };

调用 MediaEncodingProfile SetTimedMetadataTracks 将元数据流描述符添加到编码配置文件。Call MediaEncodingProfile.SetTimedMetadataTracks to add the metadata stream descriptor to the encoding profile. 以下示例介绍的帮助程序方法采用了两个视频流描述符、一个音频流描述符和一个时标元数据流描述符,同时返回可用于对流进行编码的 MediaEncodingProfileThe following example shows a helper method that takes two video stream descriptors, one audio stream descriptor, and one timed metadata stream descriptor and returns a MediaEncodingProfile that can be used to encode the streams.

public MediaEncodingProfile CreateProfileForTranscoder(VideoStreamDescriptor videoStream1, VideoStreamDescriptor videoStream2, AudioStreamDescriptor audioStream, TimedMetadataStreamDescriptor timedMetadataStream)
{
    ContainerEncodingProperties container = new ContainerEncodingProperties()
    {
        Subtype = MediaEncodingSubtypes.Mpeg4
    };

    MediaEncodingProfile profile = new MediaEncodingProfile()
    {
        Container = container
    };


    VideoStreamDescriptor encodingVideoStream1 = videoStream1.Copy();
    encodingVideoStream1.EncodingProperties.Subtype = MediaEncodingSubtypes.H264;
    encodingVideoStream1.Label = videoStream1.Name;

    VideoStreamDescriptor encodingVideoStream2 = videoStream2.Copy();
    encodingVideoStream2.EncodingProperties.Subtype = MediaEncodingSubtypes.H264;
    encodingVideoStream2.Label = videoStream2.Name;

    AudioStreamDescriptor encodingAudioStream = audioStream.Copy();
    encodingAudioStream.EncodingProperties.Subtype = MediaEncodingSubtypes.Ac3;
    encodingAudioStream.Label = audioStream.Name;

    TimedMetadataStreamDescriptor encodingTimedMetadataStream = timedMetadataStream.Copy();

    profile.SetTimedMetadataTracks(new TimedMetadataStreamDescriptor[] { encodingTimedMetadataStream });
    profile.SetVideoTracks(new VideoStreamDescriptor[] { encodingVideoStream1, encodingVideoStream2 });
    profile.SetAudioTracks(new AudioStreamDescriptor[] { encodingAudioStream });
    return profile;
}

使用多流 MediaEncodingProfile 进行录制Record using the multi-stream MediaEncodingProfile

本示例的最后一步是通过调用 StartRecordToStorageFileAsync 并传入要将所捕获媒体写入其中的 StorageFile 以及在前面的示例代码中创建的 MediaEncodingProfile,来启动视频捕获。The final step in this example is to initiate video capture by calling StartRecordToStorageFileAsync, passing in the StorageFile to which the captured media is written, and the MediaEncodingProfile created in the previous code example. 等待几秒钟后,录制将通过调用 StopRecordAsync 停止。After waiting a few seconds, the recording is stopped with a call to StopRecordAsync.

var recordFile = await Windows.Storage.KnownFolders.CameraRoll.CreateFileAsync("record.mp4", Windows.Storage.CreationCollisionOption.GenerateUniqueName);
await mediaCapture.StartRecordToStorageFileAsync(profile, recordFile);
await Task.Delay(8000);
await mediaCapture.StopRecordAsync();

操作完成时,应该已创建了一个视频文件,其中包含从每个摄像机捕获的视频(编码为文件中单独的流)。When the operation is complete, a video file will have been created that contains the video captured from each camera encoded as a separate stream within the file. 有关播放包含多个视频轨的媒体文件的信息,请参阅媒体项、播放列表和轨For information on playing media files containing multiple video tracks, see Media items, playlists, and tracks.