编写自定义全息远程处理播放器应用Writing a custom Holographic Remoting player app

重要

本文档介绍了如何为 HoloLens 2 创建自定义播放器应用程序。This document describes the creation of a custom player application for HoloLens 2. 为 HoloLens 2 编写的自定义播放器与为 HoloLens 1 编写的远程应用程序不兼容。Custom players written for HoloLens 2 are not compatible with remote applications written for HoloLens 1. 这意味着两个应用程序必须使用 NuGet 包 版本2.x。This implies that both applications must use NuGet package version 2.x.x.

通过创建自定义的全息远程处理播放器应用程序,你可以创建一个自定义应用程序,该应用程序能够在你的 HoloLens 2 上的远程计算机上显示 沉浸式视图By creating a custom Holographic Remoting player app, you can create a custom application capable of displaying immersive views from on a remote machine on your HoloLens 2. 此页面上的所有代码和工作项目都可以在 " 全息远程处理示例 github 存储库" 中找到。All code on this page and working projects can be found in the Holographic Remoting samples github repository.

使用全息远程处理播放机,你的应用可以显示桌面 PC 或 UWP 设备上 呈现 的全息内容,如 Xbox one,可以访问更多系统资源。A Holographic Remoting player lets your app display holographic content rendered on a desktop PC or UWP device like the Xbox One with access to more system resources. 全息远程处理播放机应用将输入数据流式传输到全息远程处理远程应用程序,并将沉浸式视图作为视频和音频流接收回来。A Holographic Remoting player app streams input data to a Holographic Remoting remote application and receives back an immersive view as video and audio stream. 使用标准 Wi-fi 建立连接。The connection is made using standard Wi-Fi. 若要创建播放机应用,请使用 NuGet 包将全息远程处理添加到 UWP 应用。To create a player app, use a NuGet package to add Holographic Remoting to your UWP app. 然后编写代码来处理连接并显示沉浸式视图。Then write code to handle the connection and to display an immersive view.

先决条件Prerequisites

很好的起点是基于 DirectX 的基于 DirectX 的 UWP 应用,已经以 Windows Mixed Reality API 为目标。A good starting point is a working DirectX based UWP app that already targets the Windows Mixed Reality API. 有关详细信息,请参阅 DirectX 开发概述For details see DirectX development overview. 如果你没有现成的应用程序,并且想要从头开始, c + + 全息项目模板 是一个不错的开端。If you don't have an existing app and want to start from scratch the C++ holographic project template is a good starting point.

重要

使用全息远程处理的任何应用都应该编写为使用 多线程单元Any app using Holographic Remoting should be authored to use a multi-threaded apartment. 支持使用 单线程单元 ,但会导致在播放过程中出现欠最佳的性能,并且可能会断断续续。The use of a single-threaded apartment is supported but will lead to sub-optimal performance and possibly stuttering during playback. 使用 c + +/WinRT WinRT:: init_apartment 多线程单元是默认值。When using C++/WinRT winrt::init_apartment a multi-threaded apartment is the default.

获取全息远程处理 NuGet 包Get the Holographic Remoting NuGet package

将 NuGet 包添加到 Visual Studio 中的项目时需要执行以下步骤。The following steps are required to add the NuGet package to a project in Visual Studio.

  1. 在 Visual Studio 中打开项目。Open the project in Visual Studio.
  2. 右键单击项目节点,然后选择 "管理 NuGet 包 ... "Right-click the project node and select Manage NuGet Packages...
  3. 在出现的面板中,选择 " 浏览 ",然后搜索 "全息远程处理"。In the panel that appears, select Browse and then search for "Holographic Remoting".
  4. 选择 " Microsoft",并选择 "安装",并选择 "安装 "。Select Microsoft.Holographic.Remoting, ensure to pick the latest 2.x.x version and select Install.
  5. 如果显示 " 预览 " 对话框,请选择 "确定"If the Preview dialog appears, select OK.
  6. 当 "许可协议" 对话框出现时,选择 " 我接受 "。Select I Accept when the license agreement dialog appears.

重要

build\native\include\HolographicAppRemoting\Microsoft.Holographic.AppRemoting.idlNuGet 包内部包含由全息远程处理公开的 API 的详细文档。The build\native\include\HolographicAppRemoting\Microsoft.Holographic.AppRemoting.idl inside the NuGet package contains detailed documentation for the API exposed by Holographic Remoting.

修改应用程序的 appxmanifest.xmlModify the Package.appxmanifest of the application

若要使应用程序了解 NuGet 包添加的 Microsoft.Holographic.AppRemoting.dll,需要对项目执行以下步骤:To make the application aware of the Microsoft.Holographic.AppRemoting.dll added by the NuGet package, the following steps need to be taken on the project:

  1. 在解决方案资源管理器右键单击 appxmanifest.xml 文件并选择 "打开方式 ... "In the Solution Explorer right-click on the Package.appxmanifest file and select Open With...
  2. 选择 " XML (文本) 编辑器 ",然后选择 "确定"Select XML (Text) Editor and select OK
  3. 将以下行添加到文件中并保存Add the following lines to the file and save
  </Capabilities>

  <!--Add lines below -->
  <Extensions>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>Microsoft.Holographic.AppRemoting.dll</Path>
        <ActivatableClass ActivatableClassId="Microsoft.Holographic.AppRemoting.PlayerContext" ThreadingModel="both" />
      </InProcessServer>
    </Extension>
  </Extensions>
  <!--Add lines above -->

</Package>

创建播放机上下文Create the player context

作为第一步,应用程序应创建播放机上下文。As a first step the application should create a player context.

// class declaration:

#include <winrt/Microsoft.Holographic.AppRemoting.h>

...

private:
// PlayerContext used to connect with a Holographic Remoting remote app and display remotely rendered frames
winrt::Microsoft::Holographic::AppRemoting::PlayerContext m_playerContext = nullptr;
// class implementation:

// Create the player context
// IMPORTANT: This must be done before creating the HolographicSpace (or any other call to the Holographic API).
m_playerContext = winrt::Microsoft::Holographic::AppRemoting::PlayerContext::Create();

警告

全息远程处理的工作原理是通过使用远程处理特定运行时替换作为 Windows 一部分的 Windows Mixed Reality 运行时。Holographic Remoting works by replacing the Windows Mixed Reality runtime which is part of Windows with a remoting specific runtime. 这是在创建播放机上下文的过程中完成的。This is done during the creation of the player context. 因此,在创建播放机上下文之前对任何 Windows Mixed Reality API 进行的任何调用都可能导致意外的行为。For that reason any call on any Windows Mixed Reality API before creating the player context can result in unexpected behavior. 推荐的方法是尽早创建播放机上下文,然后再与任何混合现实 API 交互。The recommended approach is to create the player context as early as possible before interaction with any Mixed Reality API. 永远不要混合通过任何 Windows Mixed Reality API 创建或检索的对象 PlayerContext::CreateNever mix objects created or retrieved through any Windows Mixed Reality API before the call to PlayerContext::Create with objects created or retrieved afterwards.

接下来,可以通过调用 CreateForCoreWindow创建 HolographicSpace。Next the HolographicSpace can be created, by calling HolographicSpace.CreateForCoreWindow.

m_holographicSpace = winrt::Windows::Graphics::Holographic::HolographicSpace::CreateForCoreWindow(window);

连接到远程应用Connect to the remote app

播放机应用准备好呈现内容后,可以建立到远程应用的连接。Once the player app is ready for rendering content a connection to the remote app can be established.

可以通过以下方式之一建立连接:The connection can be established in one of the following ways:

  1. 在 HoloLens 2 上运行的播放机应用会连接到远程应用。The player app running on HoloLens 2 connects to the remote app.
  2. 远程应用连接到在 HoloLens 2 上运行的播放机应用。The remote app connects to the player app running on HoloLens 2.

若要从播放机应用连接到远程应用,请 Connect 在指定主机名和端口的播放机上下文上调用方法。To connect from the player app to the remote app call the Connect method on the player context specifying the hostname and port. 默认端口为 8265The default port is 8265.

try
{
    m_playerContext.Connect(m_hostname, m_port);
}
catch(winrt::hresult_error& e)
{
    // Failed to connect. Get an error details via e.code() and e.message()
}

重要

与任何 c + +/WinRT API 一样, Connect 都可能引发需要处理的 WinRT:: hresult_error。As with any C++/WinRT API Connect might throw an winrt::hresult_error which needs to be handled.

可以通过调用方法来侦听播放机应用上的传入连接 ListenListening for incoming connections on the player app can be done by calling the Listen method. 在此调用过程中,可以同时指定握手端口和传输端口。Both the handshake port and transport port can be specified during this call. 握手端口用于初始握手。The handshake port is used for the initial handshake. 然后通过传输端口发送数据。The data is then sent over the transport port. 默认情况下,使用端口号 82658266By default port number 8265 and 8266 are used.

try
{
    m_playerContext.Listen(L"0.0.0.0", m_port, m_port + 1);
}
catch(winrt::hresult_error& e)
{
    // Failed to listen. Get an error details via e.code() and e.message()
}

PlayerContext公开了三个事件来监视连接的状态The PlayerContext exposes three events to monitor the state of the connection

  1. OnConnected:已成功建立到远程应用程序的连接时触发。OnConnected: Triggered when a connection to the remote app has been successfully established.
m_onConnectedEventToken = m_playerContext.OnConnected([]() 
{
    // Handle connection successfully established
});
  1. OnDisconnected:如果已建立的连接终止或无法建立连接,则触发。OnDisconnected: Triggered if an established connection is terminated or a connection couldn't be established.
m_onDisconnectedEventToken = m_playerContext.OnDisconnected([](ConnectionFailureReason failureReason)
{
    switch (failureReason)
    {
        // Handle connection failed or terminated.
        // See ConnectionFailureReason for possible reasons.
    }
}

备注

可能 ConnectionFailureReason 的值记录在 Microsoft.Holographic.AppRemoting.idl 文件中。Possible ConnectionFailureReason values are documented in the Microsoft.Holographic.AppRemoting.idl file.

  1. OnListening:侦听传入连接时启动。OnListening: When listening for incoming connections starts.
m_onListeningEventToken = m_playerContext.OnListening([]()
{
    // Handle start listening for incoming connections
});

此外,还可以使用 ConnectionState 播放机上下文上的属性查询连接状态。Additionally the connection state can be queried using the ConnectionState property on the player context.

winrt::Microsoft::Holographic::AppRemoting::ConnectionState state = m_playerContext.ConnectionState();

显示远程呈现的帧Display the remotely rendered frame

若要显示远程呈现的内容,请 PlayerContext::BlitRemoteFrame 在呈现 HolographicFrame时调用。To display the remotely rendered content, call PlayerContext::BlitRemoteFrame while rendering a HolographicFrame.

BlitRemoteFrame 要求当前 HolographicFrame 的后台缓冲区绑定为呈现器目标。BlitRemoteFrame requires that the back buffer for the current HolographicFrame is bound as render target. 可以通过Direct3D11BackBuffer属性从HolographicCameraRenderingParameters接收后台缓冲区。The back buffer can be received from the HolographicCameraRenderingParameters via the Direct3D11BackBuffer property.

调用时,会 BlitRemoteFrame 将远程应用程序中接收的最新帧复制到 HolographicFrame 的后台缓冲区中。When called, BlitRemoteFrame copies the latest received frame from the remote application into the BackBuffer of the HolographicFrame. 此外,如果远程应用程序在呈现远程帧期间指定了焦点,则会设置焦点集。Additionally the focus point set is set, if the remote application has specified a focus point during the rendering of the remote frame.

// Blit the remote frame into the backbuffer for the HolographicFrame.
winrt::Microsoft::Holographic::AppRemoting::BlitResult result = m_playerContext.BlitRemoteFrame();

备注

PlayerContext::BlitRemoteFrame 可能覆盖当前帧的焦点。PlayerContext::BlitRemoteFrame potentially overwrites the focus point for the current frame.

如果成功,则 BlitRemoteFrame 返回 BlitResult::Success_ColorOn success, BlitRemoteFrame returns BlitResult::Success_Color. 否则,它将返回失败原因:Otherwise it returns the failure reason:

  • BlitResult::Failed_NoRemoteFrameAvailable:失败,因为没有可用的远程帧。BlitResult::Failed_NoRemoteFrameAvailable: Failed because no remote frame is available.
  • BlitResult::Failed_NoCamera:由于不存在任何照相机而失败。BlitResult::Failed_NoCamera: Failed because no camera present.
  • BlitResult::Failed_RemoteFrameTooOld:由于远程帧太旧,导致失败 (请参阅 PlayerContext:: BlitRemoteFrameTimeout 属性) 。BlitResult::Failed_RemoteFrameTooOld: Failed because remote frame is too old (see PlayerContext::BlitRemoteFrameTimeout property).

重要

从版本 2.1.0 开始,自定义播放器可以通过全息远程处理使用深度 reprojection。Starting with version 2.1.0 it is possible with a custom player to use depth reprojection via Holographic Remoting.

BlitResult``````BlitResult::Success_Color_Depth在下列条件下,也可以返回:BlitResult can also return BlitResult::Success_Color_Depth under the following conditions:

如果满足这些条件,则 BlitRemoteFrame 会将远程深度 array.blit 当前绑定的本地深度缓冲区。If these conditions are met BlitRemoteFrame will blit the remote depth into the currently bound local depth buffer. 然后,您可以呈现其他本地内容,这将与远程呈现的内容具有深度交集。You can then render additional local content, which will have depth intersection with the remote rendered content. 此外,还可以通过自定义播放机中的 HolographicCameraRenderingParameters 来提交本地深度缓冲区,以获得远程和本地呈现内容的深度 reprojection。Additionally you can commit the local depth buffer via HolographicCameraRenderingParameters.CommitDirect3D11DepthBuffer in your custom player to have depth reprojection for remote and local rendered content. 有关详细信息,请参阅 深度 ReprojectionSee Depth Reprojection for details.

投影转换模式Projection Transform Mode

一个问题是,在通过全息远程处理使用深度 reprojection 的情况下,可以使用与自定义播放器应用直接呈现的本地内容不同的投影转换来呈现远程内容。One problem, which surfaces when using depth reprojection via Holographic Remoting is that the remote content can be rendered with a different projection transform than local content directly rendered by your custom player app. 常见的用例是在播放机端和远程端 (通过 HolographicCamera:: SetNearPlaneDistanceHolographicCamera:: SetFarPlaneDistance) 为近和远平面指定不同的值。A common use-case is to specify different values for near and far plane (via HolographicCamera::SetNearPlaneDistance and HolographicCamera::SetFarPlaneDistance) on the player side and the remote side. 在这种情况下,如果播放机端上的投影转换应反映远程的接近/远平面距离或局部,则不清楚。In this case, it's not clear if the projection transform on the player side should reflect the remote near/far plane distances or the local ones.

从版本 2.1.0 开始,可以通过来控制投影转换模式 PlayerContext::ProjectionTransformConfigStarting with version 2.1.0 you can control the projection transform mode via PlayerContext::ProjectionTransformConfig. 支持的值是:Supported values are:

  • Local - HolographicCameraPose::P rojectiontransform 返回投影转换,该转换反映了 HolographicCamera 上的自定义播放器应用设置的近/远平面距离。Local - HolographicCameraPose::ProjectionTransform returns a projection transform, which reflects the near/far plane distances set by your custom player app on the HolographicCamera.
  • Remote -投影转换反映远程应用指定的近/远平面距离。Remote - Projection transform reflects the near/far plane distances specified by the remote app.
  • Merged -与你的远程应用和自定义播放器应用的接近/远平面距离合并。Merged - Near/Far plane distances from your remote app and your custom player app are merged. 默认情况下,这是通过使用接近平面距离和最大平面距离的最小值来完成的。By default this is done by taking the minimum of the near plane distances and the maximum of the far plane distances. 如果远程或本地侧反转,说远 < 接近,则会翻转远程近/远平面距离。In case either the remote or local side are inverted, say far < near, the remote near/far plane distances are flipped.

可选:设置 BlitRemoteFrameTimeout Optional: Set BlitRemoteFrameTimeout

重要

PlayerContext::BlitRemoteFrameTimeout 从版本 2.0.9开始支持。PlayerContext::BlitRemoteFrameTimeout is supported starting with version 2.0.9.

PlayerContext::BlitRemoteFrameTimeout如果未接收到新的远程帧,则属性指定重复使用远程帧的时间长度。The PlayerContext::BlitRemoteFrameTimeout property specifies the amount of time a remote frame is reused if no new remote frame is received.

常见的用例是,如果在一段时间内没有收到新帧,BlitRemoteFrame 超时将显示空白屏幕。A common use-case is to enable the BlitRemoteFrame timeout to display a blank screen if no new frames are received for a certain amount of time. 启用时,方法的返回类型 BlitRemoteFrame 还可用于切换到本地呈现的回退内容。When enabled the return type of the BlitRemoteFrame method can also be used to switch to a locally rendered fallback content.

若要启用超时,请将属性值设置为等于或大于 100 ms 的持续时间。To enable the timeout, set the property value to a duration equal or greater than 100 ms. 若要禁用超时,请将属性设置为零。To disable the timeout, set the property to zero duration. 如果在设置的持续时间内启用了超时并且未收到任何远程帧,则 BlitRemoteFrame 将失败并返回, Failed_RemoteFrameTooOld 直到接收到新的远程帧。If the timeout is enabled and no remote frame is received for the set duration, BlitRemoteFrame will fail and return Failed_RemoteFrameTooOld until a new remote frame is received.

using namespace std::chrono_literals;

// Set the BlitRemoteFrame timeout to 0.5s
m_playerContext.BlitRemoteFrameTimeout(500ms);

可选:获取有关最后一个远程帧的统计信息Optional: Get statistics about the last remote frame

若要诊断性能或网络问题,可以通过属性检索有关最后一个远程帧的统计信息 PlayerContext::LastFrameStatisticsTo diagnose performance or network issues, statistics about the last remote frame can be retrieved via the PlayerContext::LastFrameStatistics property. 在调用 HolographicFrame 期间更新统计信息 ::P resentusingcurrentpredictionStatistics are updated during the call to HolographicFrame::PresentUsingCurrentPrediction.

// Get statistics for the last presented frame.
winrt::Microsoft::Holographic::AppRemoting::PlayerFrameStatistics statistics = m_playerContext.LastFrameStatistics();

有关详细信息,请参阅 PlayerFrameStatistics 文件中的文档 Microsoft.Holographic.AppRemoting.idl For more information, see the PlayerFrameStatistics documentation in the Microsoft.Holographic.AppRemoting.idl file.

可选:自定义数据通道Optional: Custom data channels

自定义数据信道可用于通过已建立的远程处理连接发送用户数据。Custom data channels can be used to send user data over the already established remoting connection. 有关详细信息,请参阅 自定义数据通道See custom data channels for more information.

另请参阅See Also