How to optimize MediaElement for live streaming?

Gabriel Remus Linck Duarte 0 Reputation points
2024-02-14T19:07:24.1066667+00:00

Greetings,

Currently I’m attempting to stream a H264 encoded video from another device in a local network using MediaElement XAML for the UI and MediaStreamSource as its source. This works fine but offers a noticeable delay (150-300ms). All options that should optimize decoding and rendering are set, such as RealTimePlayback for MediaElement and IsLive and a small BufferTime for MediaStreamSource.

Upon using an external open media player such as ffplay to play the same video stream this delay is not present. I’d like to use native solutions to keep this Windows App friendly with UWP for portability. I've already tried using MediaPlayerElement instead, but it seems to ignore the BufferTime set for MediaStreamSource and uses 3 seconds by default.

Are there any further optimizations or new optimized methods dedicated for streaming video within the XAML platform for UWP Apps SDK?

Thanks

Edit: added code snippet for reference

<MediaElement 
    x:Name="mediaElement"
    HorizontalAlignment="Stretch"
    VerticalAlignment="Stretch"
    RealTimePlayback="True"
/>
private readonly ConcurrentQueue<MediaStreamSample> _videoQueue = new();
private MediaStreamSource _videoSource;
private MediaStreamSourceSampleRequest _videoRequest;
private MediaStreamSourceSampleRequestDeferral _videoDeferral;

void InitializeMediaElement()
{
    _ = CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
    {
        var properties = VideoEncodingProperties.CreateH264();
        _videoSource = new MediaStreamSource(new VideoStreamDescriptor(properties))
        {
            BufferTime = TimeSpan.FromMilliseconds(150),
            IsLive = true,
            CanSeek = false,
        };
        _videoSource.SampleRequested += VideoSampleRequested;
        mediaElement.SetMediaStreamSource(_videoSource);
        mediaElement.Play();
    });
}

// This is called from another part of the application whenever a video packet is received
void VideoReceived(byte[] payload)
{
    _ = CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
    {
        MediaStreamSample mediaSample = GetVideoSample(payload);
        if (_videoDeferral != null)
        {
            _videoRequest.Sample = mediaSample;
            _videoRequest.ReportSampleProgress(100);
            var deferral = _videoDeferral;
            _videoDeferral = null;
            deferral.Complete();
        }
        else
        {
            _videoQueue.Enqueue(mediaSample);
        }
    });
}

private MediaStreamSample GetVideoSample(byte[] payload)
{
    IBuffer buffer = WindowsRuntimeBufferExtensions.AsBuffer(payload);
    TimeSpan timestamp = TimeSpan.Zero;
    return MediaStreamSample.CreateFromBuffer(buffer, timestamp);
}

private void VideoSampleRequested(MediaStreamSource sender, MediaStreamSourceSampleRequestedEventArgs args)
{
    if (_videoQueue.TryDequeue(out MediaStreamSample mediaSample))
    {
        args.Request.Sample = mediaSample;
    }
    else if (_videoDeferral == null)
    {
        args.Request.ReportSampleProgress(0);
        _videoDeferral = args.Request.GetDeferral();
        _videoRequest = args.Request;
    }
}
Universal Windows Platform (UWP)
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,299 questions
{count} votes