I have been trying to determine how to process and support thumbnail images for encoded videos and the answers are very hard to find for Azure Media Services V3, many links to docs on forums are also throwing 404s. Part of this issue is a new Roku store policy that requires them to be included in HLS or DASH streams for new release and they would be nice for other platforms as well. After all of this I am left with a number of questions: Does Dyanmic Packaging have thumbnail support? If so were can we learn how to work with it? Can I process and provide thumbnails for existing videos in our catalog without re-encoding the entire video? Is their any documentation for the thumbnails or the schema for transforms for V3? It looks like the JpgImage in the transform schema will at least generate the image files, is this correct?

Hi Justin, A few quick answers. Dynamic packaging does not generate thumbnails on the fly at all. The default presets for Content Aware, Adaptive Bitrate, etc, do generate a "best" thumbnail, but if you want to control the thumbnail settings you do need to add that to a custom encoding job. I have some Javascript/TS samples here that show how to do that: https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/b0ad91bb7f0b466fb6ec1237301ff7c6e130c49b/VideoEncoding/Encoding_H264/index.ts#L115 You can send any existing Asset in AMS back through a "thumbnail" specific transform. See example above on how to create a thumbnail in a preset. Just create a preset that does nothing other than generate a single JPG, or a Sprite with VTT file as in this sample. Just remove the video and audio tracks from that Transform - https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/main/VideoEncoding/Encoding_Sprite_Thumbnail/index.ts All the schema for this is actually in the REST API. You should look at the Transform entity details in the REST API for all the encoding and thumbnail settings. Search for PNGFormat, JPGFormat, JPGImage, PNGImage, - https://learn.microsoft.com/en-us/rest/api/media/transforms/create-or-update Correct the JPG or PNGImage objects in the REST API have all the settings you would need. Hope this helps!

Azure Media Services V3 and Thumbnails

Accepted answer

John Deutscher (MSFT) 2,126 Reputation points

2022-05-20T16:56:55.983+00:00
Hi Justin,

A few quick answers.

Dynamic packaging does not generate thumbnails on the fly at all. The default presets for Content Aware, Adaptive Bitrate, etc, do generate a "best" thumbnail, but if you want to control the thumbnail settings you do need to add that to a custom encoding job. I have some Javascript/TS samples here that show how to do that:
https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/b0ad91bb7f0b466fb6ec1237301ff7c6e130c49b/VideoEncoding/Encoding_H264/index.ts#L115

You can send any existing Asset in AMS back through a "thumbnail" specific transform. See example above on how to create a thumbnail in a preset. Just create a preset that does nothing other than generate a single JPG, or a Sprite with VTT file as in this sample. Just remove the video and audio tracks from that Transform - https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/main/VideoEncoding/Encoding_Sprite_Thumbnail/index.ts

All the schema for this is actually in the REST API. You should look at the Transform entity details in the REST API for all the encoding and thumbnail settings. Search for PNGFormat, JPGFormat, JPGImage, PNGImage, - https://learn.microsoft.com/en-us/rest/api/media/transforms/create-or-update

Correct the JPG or PNGImage objects in the REST API have all the settings you would need.

Hope this helps!
Please sign in to rate this answer.

1 person found this answer helpful.
John Deutscher (MSFT) 2,126 Reputation points

2022-05-20T16:57:29.737+00:00

Ohh, and if you can find the links that are 404 - let me know! I can try to figure out why.

Justin Griep 41 Reputation points

2022-05-20T18:17:11.293+00:00

If you search for Azure Media Services V3, they are all over the various forum posts, StackOverlow, the old MS forum, etc. So I don't know what you can do about that.

John Deutscher (MSFT) 2,126 Reputation points

2022-05-20T18:28:31.657+00:00

Yea, it is tough to fix the older posts on the Stack and previous MSDN forum. Bummer, it is just an issue with moving forward with docs and samples.
If you find any big issues let us know.

As to the Roku store issue - I think that they had a proprietary image format track that was not actually in the HLS or DASH specification in the past. If you have the link to that I can take a look again, but recently there was an update to the official HLS spec to support image tracks - that should be what ROKU adopts rather than pushing their proprietary thing.

I was tracking this issue in the CTA WAVE working group - https://github.com/cta-wave/dash-hls/issues/37 which also mentioned the same

"There is the Image media playlists HLS extension proposal from Roku/Disney/Warner here https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md but its relevance/adoption is limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage iframe-only tracks, it would be super useful for the rest of the ecosystem to get this officially part of the HLS RFC."

On 2/23, Apple announced upcoming iOS support for M-JPEG (ISO/IEC 23008-12 "High efficiency coding and media delivery in heterogeneous environments —Part 12: Image File Format") encapsulated in fMP4 (CMAF?) containers.

Justin Griep 41 Reputation points

2022-05-20T18:30:23.84+00:00

Hi John,

Thank you so much for the answers, this gets me most of the way there.

We are using .NET, so JS/TS isn't preferred, but we are using the examples I found here which should cover your examples:
https://learn.microsoft.com/en-us/azure/media-services/latest/samples-overview?tabs=node

On my first question, I am not assuming that Dynamic Packaging would generate the thumbnails, but to get it to work does require the inclusion of the thumbnails in the manifest file based on the specifications for HLS and DASH.

Here is the DASH spec:
https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf

And the HLS spec:
https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-04

So my question is really, does Dynamic Packaging support generating these thumbnail portions of the manifest? If so, how do we configure that? If not, how can we work around this with Media Services?

Thank you again,

Justin

John Deutscher (MSFT) 2,126 Reputation points

2022-05-20T18:33:34.187+00:00

For live we only generate IFrame tracks with IDR frames for trick-play mode. But that requires a live stream going through out encoding live events.

For VOD, we do not yet have any packager support for the HLS or DASH style image tracks. As pointed out above, there are some updates to the HLS spec recently that we are looking into and might add that to our roadmap in the future. I'll inform the team of your interest here and point to the latest updates from HLS and DASH orgs.

Justin Griep 41 Reputation points

2022-05-20T18:52:02.717+00:00

Roku is moving away from the proprietary and toward everything you have mentioned:
https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md

But this is a hairy standards mess for now.

It looks like we can avoid the store requirement by just not enabling trick play/mode and look to this again once the standards settle out. I am going to see about us dropping this capability requirement.

I can't tell you how much I appreciate your time and insights! I can tell you really know streaming deeply. Going to cancel my Azure ticket on this now.

Justin Griep 41 Reputation points

2022-05-20T19:02:15.397+00:00

Last note, I see who you are, and thank you again for taking the time to educate me.

I also see why from a product management perspective you may want to hold off until the standards solidify, so we won't be pushing on this feature from Media Service until then.
Sign in to comment

Azure Media Services V3 and Thumbnails

0 additional answers