Writing a Holographic Remoting host app

Important

This document describes the creation of a host application for HoloLens 2. Host application for HoloLens (1st gen) must use NuGet package version 1.x.x. This implies that host applications written for HoloLens 2 are not compatible with HoloLens 1 and vice versa. The documentation for HoloLens 1 can be found here.

By creating a Holographic Remoting host app remote content that is rendered on a remote machine can be streamed to HoloLens 2. This article describes how this can be achieved. All code on this page and working projects can be found in the Holographic Remoting samples github repository.

Holographic remoting allows an app to target HoloLens 2 with holographic content hosted on a desktop PC or on a UWP device such as the Xbox One, allowing access to more system resources and making it possible to integrate remote immersive views into existing desktop PC software. A remoting host app receives an input data stream from HoloLens 2, renders content in a virtual immersive view, and streams content frames back to HoloLens 2. The connection is made using standard Wi-Fi. Holographic Remoting is added to a desktop or UWP app via a NuGet packet. Additional code is required which handles the connection and renders in an immersive view.

A typical remoting connection will have as low as 50 ms of latency. The player app can report the latency in real-time.

Prerequisites

A good starting point is a working DirectX based Desktop or UWP app which targets the Windows Mixed Reality API. For details see DirectX development overview. The C++ holographic project template is a good starting point.

Important

Any app using Holographic Remoting should be authored to use a multi-threaded apartment. The use of a single-threaded apartment is supported but will lead to sub-optimal performance and possibly stuttering during playback. When using C++/WinRT winrt::init_apartment a multi-threaded apartment is the default.

Get the Holographic Remoting NuGet package

The following steps are required to add the NuGet package to a project in Visual Studio.

  1. Open the project in Visual Studio.
  2. Right-click the project node and select Manage NuGet Packages...
  3. In the panel that appears, click Browse and then search for "Holographic Remoting".
  4. Select Microsoft.Holographic.Remoting, ensure to pick the latest 2.x.x version and click Install.
  5. If the Preview dialog appears, click OK.
  6. The next dialog that appears is the license agreement. Click on I Accept to accept the license agreement.

Note

Version 1.x.x of the NuGet package is still available for developers who want to target HoloLens 1. For details see Add Holographic Remoting (HoloLens (1st gen)).

Create the remote context

As a first step the application should create a remote context.

// class declaration
#include <winrt/Microsoft.Holographic.AppRemoting.h>

...

private:
    // RemoteContext used to connect with a Holographic Remoting player and display rendered frames
    winrt::Microsoft::Holographic::AppRemoting::RemoteContext m_remoteContext = nullptr;
// class implementation
#include <HolographicAppRemoting\Streamer.h>

...

CreateRemoteContext(m_remoteContext, 20000, false, PreferredVideoCodec::Default);

Warning

Holographic Remoting works by replacing the Windows Mixed Reality runtime which is part of Windows with a remoting specific runtime. This is done during the creation of the remote context. For that reason any call on any Windows Mixed Reality API before creating the remote context can result in unexpected behavior. The recommended approach is to create the remote context as early as possible before interaction with any Mixed Reality API. Never mix objects created or retrieved through any Windows Mixed Reality API before the call to CreateRemoteContext with objects created or retrieved afterwards.

Next the holographic space needs to be created. Specifying a CoreWindow is not required. Desktop apps that do not have a CoreWindow can just pass a nullptr.

m_holographicSpace = winrt::Windows::Graphics::Holographic::HolographicSpace::CreateForCoreWindow(nullptr);

Connect to the device

Once the host app is ready for rendering content a connection to the device can be established.

Connection can be done in one of two ways.

  1. The host app connects to the player running on the device.
  2. The player running on the device connects to the host app.

To establish a connection from the host app to HoloLens 2 call the Connect method on the remote context specifying the hostname and port. The port used by the Holographic Remoting Player is 8265.

try
{
    m_remoteContext.Connect(m_hostname, m_port);
}
catch(winrt::hresult_error& e)
{
    DebugLog(L"Connect failed with hr = 0x%08X", e.code());
}

Important

As with any C++/WinRT API Connect might throw an winrt::hresult_error which needs to be handled.

Tip

To avoid using C++/WinRT language projection the file build\native\include\<windows sdk version>\abi\Microsoft.Holographic.AppRemoting.h located inside the Holographic Remoting NuGet package can be included. It contains declarations of the underlying COM interfaces. The use of C++/WinRT is recommended though.

Listening for incoming connections on the host app can be done by calling the Listen method. Both the handshake port and transport port can be specified during this call. The handshake port is used for the initial handshake. The data is then send over the transport port. By default 8265 and 8266 are used.

try
{
    m_remoteContext.Listen(L"0.0.0.0", m_port, m_port + 1);
}
catch(winrt::hresult_error& e)
{
    DebugLog(L"Listen failed with hr = 0x%08X", e.code());
}

Important

The build\native\include\HolographicAppRemoting\Microsoft.Holographic.AppRemoting.idl inside the NuGet package contains detailed documentation for the API exposed by Holographic Remoting.

Handling Remoting specific events

The remote context exposes three events which are important to monitor the state of a connection.

  1. OnConnected: Triggered when a connection to the device has been successfully established.
winrt::weak_ref<winrt::Microsoft::Holographic::AppRemoting::IRemoteContext> remoteContextWeakRef = m_remoteContext;

m_onConnectedEventRevoker = m_remoteContext.OnConnected(winrt::auto_revoke, [this, remoteContextWeakRef]() {
    if (auto remoteContext = remoteContextWeakRef.get())
    {
        // Update UI state
    }
});
  1. OnDisconnected: Triggered if an established connection is closed or a connection could not be established.
m_onDisconnectedEventRevoker =
    m_remoteContext.OnDisconnected(winrt::auto_revoke, [this, remoteContextWeakRef](ConnectionFailureReason failureReason) {
        if (auto remoteContext = remoteContextWeakRef.get())
        {
            DebugLog(L"Disconnected with reason %d", failureReason);
            // Update UI

            // Reconnect if this is a transient failure.
            if (failureReason == ConnectionFailureReason::HandshakeUnreachable ||
                failureReason == ConnectionFailureReason::TransportUnreachable ||
                failureReason == ConnectionFailureReason::ConnectionLost)
            {
                DebugLog(L"Reconnecting...");

                ConnectOrListen();
            }
            // Failure reason None indicates a normal disconnect.
            else if (failureReason != ConnectionFailureReason::None)
            {
                DebugLog(L"Disconnected with unrecoverable error, not attempting to reconnect.");
            }
        }
    });
  1. OnListening: When listening for incoming connections starts.
m_onListeningEventRevoker = m_remoteContext.OnListening(winrt::auto_revoke, [this, remoteContextWeakRef]() {
    if (auto remoteContext = remoteContextWeakRef.get())
    {
        // Update UI state
    }
});

Additionally the connection state can be queried using the ConnectionState property on the remote context.

auto connectionState = m_remoteContext.ConnectionState();

Handling speech events

Using the remote speech interface it is possible to register speech triggers with HoloLens 2 and have them remoted to the host application.

This additional member is required to track the state of the remote speech.

winrt::Microsoft::Holographic::AppRemoting::IRemoteSpeech::OnRecognizedSpeech_revoker m_onRecognizedSpeechRevoker;

First the remote speech interface needs to be retrieved.

if (auto remoteSpeech = m_remoteContext.GetRemoteSpeech())
{
    InitializeSpeechAsync(remoteSpeech, m_onRecognizedSpeechRevoker, weak_from_this());
}

Using an asynchronous helper method you can then initialize the remote speech. This should be done asynchronously as initializing might take a considerable amount of time. Concurrency and asynchronous operations with C++/WinRT explains how to author asynchronous functions with C++/WinRT.

winrt::Windows::Foundation::IAsyncOperation<winrt::Windows::Storage::StorageFile> LoadGrammarFileAsync()
{
    const wchar_t* speechGrammarFile = L"SpeechGrammar.xml";
    auto rootFolder = winrt::Windows::ApplicationModel::Package::Current().InstalledLocation();
    return rootFolder.GetFileAsync(speechGrammarFile);
}

winrt::fire_and_forget InitializeSpeechAsync(
    winrt::Microsoft::Holographic::AppRemoting::IRemoteSpeech remoteSpeech,
    winrt::Microsoft::Holographic::AppRemoting::IRemoteSpeech::OnRecognizedSpeech_revoker& onRecognizedSpeechRevoker,
    std::weak_ptr<SampleHostMain> sampleHostMainWeak)
{
    onRecognizedSpeechRevoker = remoteSpeech.OnRecognizedSpeech(
        winrt::auto_revoke, [sampleHostMainWeak](const winrt::Microsoft::Holographic::AppRemoting::RecognizedSpeech& recognizedSpeech) {
            if (auto sampleHostMain = sampleHostMainWeak.lock())
            {
                sampleHostMain->OnRecognizedSpeech(recognizedSpeech.RecognizedText);
            }
        });

    auto grammarFile = co_await LoadGrammarFileAsync();

    std::vector<winrt::hstring> dictionary;
    dictionary.push_back(L"Red");
    dictionary.push_back(L"Blue");
    dictionary.push_back(L"Green");
    dictionary.push_back(L"Default");
    dictionary.push_back(L"Aquamarine");

    remoteSpeech.ApplyParameters(L"en-US", grammarFile, dictionary);
}

There are two ways of specifying phrases to be recognized.

  1. Specification inside a speech grammar xml file. See How to create a basic XML Grammar for details.
  2. Specify by passing them inside the dictionary vector to ApplyParameters.

Inside the OnRecognizedSpeech callback the speech events can then be processed:

void SampleHostMain::OnRecognizedSpeech(const winrt::hstring& recognizedText)
{
    bool changedColor = false;
    DirectX::XMFLOAT4 color = {1, 1, 1, 1};

    if (recognizedText == L"Red")
    {
        color = {1, 0, 0, 1};
        changedColor = true;
    }
    else if (recognizedText == L"Blue")
    {
        color = {0, 0, 1, 1};
        changedColor = true;
    }
    else if (recognizedText == L"Green")
    {
        ...
    }

    ...
}

Preview streamed content locally

To display the same content in the host app that is send to the device the OnSendFrame event of the remote context can be used. The OnSendFrame event is triggered every time the Holographic Remoting library sends the current frame to the remote device. This is the ideal time to take the content and also blit it into the desktop or UWP window.

#include <windows.graphics.directx.direct3d11.interop.h>

...

m_onSendFrameEventRevoker = m_remoteContext.OnSendFrame(
    winrt::auto_revoke, [this](const winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface& texture) {
        winrt::com_ptr<ID3D11Texture2D> texturePtr;
        {
            winrt::com_ptr<ID3D11Resource> resource;
            winrt::com_ptr<::IInspectable> inspectable = texture.as<::IInspectable>();
            winrt::com_ptr<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
            winrt::check_hresult(inspectable->QueryInterface(__uuidof(dxgiInterfaceAccess), dxgiInterfaceAccess.put_void()));
            winrt::check_hresult(dxgiInterfaceAccess->GetInterface(__uuidof(resource), resource.put_void()));
            resource.as(texturePtr);
        }

        // Copy / blit texturePtr into the back buffer here.
    });

Optional: Custom data channels

Custom data channels can be used to send user data over the already established remoting connection. See custom data channels for more information.

See Also