Subject: Re: [Hls-interest] Image-based subtitles and trickplay tracks

From: "Weil, Nicolas" <[email-masked]>
Date: Thu, 28 May 2020 15:44:45 +0000

Hi Boy,

The user experience is really strange when you move your cursor on the timeline after loading the manifest, with the thumbnails display lagging behind the cursor position. It gets better after some time. Is it a player implementation problem or a side effect of the CMAF encapsulation?

Thanks,
Nicolas

From: Hls-interest <[email-masked]> On Behalf Of Boy van Dijk
Sent: Thursday, May 28, 2020 1:53 AM
To: Roger Pantos <rpantos=[email-masked]>
Cc: Law, Will <wilaw=[email-masked]>; [email-masked]
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks

Hi all,

Thank you for your valuable insights, Roger. Also on behalf of Unified Streaming.

We have a demo available that showcases the use of a tiled thumbnails trickplay track packaged as JPEG in CMAF. Dash.js already supports this:

https://reference.dashif.org/dash.js/v3.1.1/samples/dash-if-reference-player/index.html?url=https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd<https://reference..dashif.org/dash.js/v3.1.1/samples/dash-if-reference-player/index.html?url=https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd>

Hope this is helpful.

Regards,
Boy

On Wed, May 27, 2020 at 8:19 PM Roger Pantos <rpantos=[email-masked]<mailto:[email-masked]>> wrote:

On May 26, 2020, at 2:52 PM, Law, Will <wilaw=[email-masked]<mailto:wilaw=[email-masked]>> wrote:

Some additional data for this thread from a recent study just conducted for Apple device playback against Akamai CDN by the Client Optimization Team last month. You can see that for this ATV playback session, over the time period monitored, there were 22K requests in total, of which 98% were small range requests for thumbnails/trickplay.

Every request has a fulfilment cost at some point. Tiled images would serve to lower the request rate against the edge, in this case, by an order of magnitude.   From a CDN perspective, we would lend our support towards the development of an optional tiled-image-based thumbnail solution for HLS.

Be careful about that data. A lot of those i-frame requests are being done to support scanning playback where i-frames loads are sparse. For example at 96x only every ~16th frame is requested.

Having said that, you can aggregate multiple JPEGs into a single fmp4 file pretty easily, which would set up the client for the same kind of optimizations to lower the request rate if it wanted to load every i-frame.

Roger.

Cheers
Will

From: "Weil, Nicolas" <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, May 26, 2020 at 12:22 PM
To: "May, Bill" <[email-masked]<mailto:[email-masked]>>, Roger Pantos <rpantos=[email-masked]<mailto:rpantos=[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Hls-interest] Image-based subtitles and trickplay tracks

Comments inline.

From: May, Bill <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, May 26, 2020 9:31 AM
To: Roger Pantos <rpantos=[email-masked]<mailto:rpantos=[email-masked]>>
Cc: Weil, Nicolas <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks

On May 22, 2020, at 11:17 AM, Roger Pantos <rpantos=[email-masked]<mailto:rpantos=[email-masked]>> wrote:

Hello Nicholas. Thanks for bringing these up. I have some questions:

On May 20, 2020, at 3:12 PM, Weil, Nicolas <[email-masked]<mailto:[email-masked]>> wrote:

Hello,

We are often seeing two image-related topics causing interoperability problems as they are not currently covered by the HLS spec.. Normalizing the implementations around an official specification for these two points would be great:

Image-based subtitles tracks
For workflow reasons and charset reasons, some content owners don't include text-based subtitles in the live channels sources that they provide to distributors, but rather image-based subtitles (like DVB-Sub). While it's possible to transform these subtitles as IMSC1 Image Profile as per DASH-IF IOP section 6.4.4, there is no equivalent IMSC1 Image Profile support in the HLS RFC, which means that companies will continue to rely on proprietary forks of the HLS RFC to support these use cases. Even if it wasn't supported by Apple players, it would be tremendously helpful for interoperability in the rest of the HLS ecosystem.

I'd like to understand how widely validated the Image Profile of IMSC1 has been. Can anyone volunteer some examples where it’s been commercially deployed successfully? (Specifically IMSC1, vs. some other fork of TTML.)
[NW] IMSC1 Image profile is now supported by ATSC3, DASH-IF IOP (with support in dash.js) and IMF (with support in IMFTool<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IMFTool_IMFTool&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=dBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHcgJLKmlBotj0&s=fBrrNLMD_eSxO9MYInkAMOhyaX6Zicdvk3HTD6p5mho&e=> which development has been sponsored by Netflix and other studios initially).

Another question: there are requirements in the US at least that require the ability to change font sizes, colors, etc.   And, TBH, these are changes that help people world-wide.

How would you meet those requirements with bit mapped subtitles?  Wouldn’t it be better to work to eliminate bitmapped subtitles completely?
[NW] I believe these font size/color change requirements can be satisfied with IMSC1 Text Profile which is supported in rfc8216bis since 2017.
As much as I’d like to get rid of bitmap subtitles, sometimes the content owners cannot provide anything else than bitmaps in the source feed. And it’s very challenging to apply a reliable OCR pass on it, for all target languages (Latin/Cyrillic/Asian/… charsets). IMSC1 Image Profile has got a decent industry support, and the Text Profile is already supported in HLS, so I would expect it to be a natural extension for HLS to support also the Image Profile.

Image-based trickplay tracks
For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image Media playlists HLS extension proposal from Roku/Disney/WarnerMedia here https://github.com/image-media-playlist/spec<https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fimage-2Dmedia-2Dplaylist-252Fspec-26data-3D02-257C01-257Cbill.may-2540disneystreaming.com-257C49edde2508884eb51cca08d7fe7c807e-257C65f03ca86d0a493e9e4ac85ac9526a03-257C1-257C0-257C637257683002059478-26sdata-3D9lV-252BiNvxiK5feFCV1hYgo5I1lW0TEpMlWl-252BpwKwVKZU-253D-26reserved-3D0&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=dBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHcgJLKmlBotj0&s=yKAoFpJAUPAHylJJXOXaXVSeWHIwBnO5F9MOWl5tgWc&e=> but its relevance/adoption is currently limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage I-frame tracks, it would be super useful for the rest of the HLS ecosystem to get this officially part of the RFC.

I'd like to better understand what’s driving this. Is the limitation essentially one of not being able to support an AVC decoder for i-frame display?

If that’s the case then it seems that putting JPEG images into fMP4 containers and using EXT-X-I-FRAME-STREAM-INF would be a smaller extension to HLS, both in terms of departure from the existing approach and less new spec to invent.

One of the things I don’t love about the image-media-playlist spec is that it doesn’t follow the regular HLS timing model, where the media presentation time is defined in the media data itself. Instead it relies on precise synchronization of the EXTINF values, which seems like a recipe for long-term accumulation of floating point error, as well as difficult to achieve with multiple geographically-dispersed packagers for live.

The limitation is exactly that.  A second decoder (AVC or HEVC) is not available on many devices.  This also makes mid-fragment switching difficult as well and makes switching between codecs impossible as well.

The image-media-playlist spec does rely somewhat on floating point; no more so that a seek to date or seek to time does in a regular HLS playlist, however.  I’m not sure that anyone is asking for precise millisecond switching from these images to regular AV.

I see 2 solutions to this problem: give a PTS/timescale in the HLS playlist (something like we did for transport stream to webVTT timing, but in the playlist), or wrap the jpeg in some sort of wrapper with timing (fmp4?).  It would be good, if that is the route, to have guidance from Apple on what specification to use.

The thing about JPEGs is that they are easy; almost any software decodes them; wrapping them in FMP4 doesn’t make it easier or better.

[NW] Thumbnails in the DASH-IF IOP are simple jpeg images and it makes it easy to produce and to manipulate on the service side (like aggregating several live thumbnails into tiles of thumbnails when a program is transitioned from Live to VOD). Using the same simple image container would allow direct interoperability with DASH, without requiring an additional CMAF+DASH specification cycle. As regards HLS, I was hoping that the use of EXT-PROGRAM-DATE-TIME would become mandatory, as per the preliminary LL-HLS specification. That would give use the millisecond-accurate time reference that we need to avoid drifts if we keep images in a simple image container.

--
Hls-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hls-interest

--
Hls-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hls-interest

---

From: "Law, Will" <[email-masked]>
Date: Thu, 28 May 2020 16:01:22 +0000

That’s the player late binding to the tile. You could resolve that issue by downloading all tiles up front and not just-in-time. (or just-late, per your observation).

From: "Weil, Nicolas" <[email-masked]>
Date: Thursday, May 28, 2020 at 8:46 AM
To: Boy van Dijk <[email-masked]>, Roger Pantos <[email-masked]>
Cc: "Law, Will" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks

Hi Boy,

Thanks,
Nicolas

Hi all,

https://reference.dashif.org/dash.js/v3.1.1/samples/dash-if-reference-player/index.html?url=https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd<https://urldefense.proofpoint.com/v2/url?u=https-3A__reference..dashif.org_dash.js_v3.1.1_samples_dash-2Dif-2Dreference-2Dplayer_index.html-3Furl-3Dhttps-3A__demo.unified-2Dstreaming.com_video_tears-2Dof-2Dsteel_tears-2Dof-2Dsteel-2Dtiled-2Dthumbnails-2Dstatic.mpd&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=w4ovBI-qwOGEPdaK9xFw621gsgC7hv8nZ6UbCFTbmDQ&s=Osw__LhHCWRfyyupk-NQL4HAYqrB_4AVOmH395QuPpc&e=>

Hope this is helpful.

Regards,
Boy

Roger.

Cheers
Will

Comments inline.

Hello,

--
Hls-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hls-interest<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_hls-2Dinterest&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=w4ovBI-qwOGEPdaK9xFw621gsgC7hv8nZ6UbCFTbmDQ&s=HCByMvpnmrjKJ5EdTqd7iDTEgI_VZWdp2c0CqL0Ioc8&e=>

--
Hls-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hls-interest<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_hls-2Dinterest&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=w4ovBI-qwOGEPdaK9xFw621gsgC7hv8nZ6UbCFTbmDQ&s=HCByMvpnmrjKJ5EdTqd7iDTEgI_VZWdp2c0CqL0Ioc8&e=>

---

From: <[email-masked]>
Date: Thu, 28 May 2020 18:31:26 +0200

Dear Nicolas, I could not reproduce any of the behaviour you describe (windows+chrome/edge), CMAF uses the timing in MP4, same as media,

So it is quite unlikely to cause any sync related issues, best, Rufael 

From: Hls-interest <[email-masked]> On Behalf Of Weil, Nicolas
Sent: donderdag 28 mei 2020 17:45
To: Boy van Dijk <[email-masked]>; Roger Pantos <rpantos=[email-masked]>
Cc: Law, Will <wilaw=[email-masked]>; [email-masked]
Subject: Re: [Hls-interest] Image-based subtitles and trickplay tracks

Hi Boy,

Thanks,

Nicolas

From: Hls-interest <[email-masked] <mailto:[email-masked]> > On Behalf Of Boy van Dijk
Sent: Thursday, May 28, 2020 1:53 AM
To: Roger Pantos <rpantos=[email-masked] <mailto:rpantos=[email-masked]> >
Cc: Law, Will <wilaw=[email-masked] <mailto:wilaw=[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks

Hi all,

https://reference.dashif.org/dash.js/v3.1.1/samples/dash-if-reference-player/index.html?url=https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd <https://reference..dashif.org/dash.js/v3.1.1/samples/dash-if-reference-player/index.html?url=https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd> 

Hope this is helpful.

Regards,

Boy

On Wed, May 27, 2020 at 8:19 PM Roger Pantos <rpantos=[email-masked] <mailto:[email-masked]> > wrote:

On May 26, 2020, at 2:52 PM, Law, Will <wilaw=[email-masked] <mailto:wilaw=[email-masked]> > wrote:

 

Roger.

Cheers

Will

 

From: "Weil, Nicolas" <[email-masked] <mailto:[email-masked]> >
Date: Tuesday, May 26, 2020 at 12:22 PM
To: "May, Bill" <[email-masked] <mailto:[email-masked]> >, Roger Pantos <rpantos=[email-masked] <mailto:rpantos=[email-masked]> >
Cc: "[email-masked] <mailto:[email-masked]> " <[email-masked] <mailto:[email-masked]> >
Subject: Re: [Hls-interest] Image-based subtitles and trickplay tracks

Comments inline. 

From: May, Bill <[email-masked] <mailto:[email-masked]> > 
Sent: Tuesday, May 26, 2020 9:31 AM
To: Roger Pantos <rpantos=[email-masked] <mailto:rpantos=[email-masked]> >
Cc: Weil, Nicolas <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks

On May 22, 2020, at 11:17 AM, Roger Pantos <rpantos=[email-masked] <mailto:rpantos=[email-masked]> > wrote:

On May 20, 2020, at 3:12 PM, Weil, Nicolas <[email-masked] <mailto:[email-masked]> > wrote:

Hello,

[NW] IMSC1 Image profile is now supported by ATSC3, DASH-IF IOP (with support in dash.js) and IMF (with support in  <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IMFTool_IMFTool&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=dBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHcgJLKmlBotj0&s=fBrrNLMD_eSxO9MYInkAMOhyaX6Zicdvk3HTD6p5mho&e=> IMFTool which development has been sponsored by Netflix and other studios initially).

Image-based trickplay tracks
For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image Media playlists HLS extension proposal from Roku/Disney/WarnerMedia here https://github.com/image-media-playlist/spec but its relevance/adoption is currently limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage I-frame tracks, it would be super useful for the rest of the HLS ecosystem to get this officially part of the RFC.

 

-- 
Hls-interest mailing list
 <mailto:[email-masked]> [email-masked]
 <https://www.ietf.org/mailman/listinfo/hls-interest> https://www.ietf.org/mailman/listinfo/hls-interest

-- 
Hls-interest mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/hls-interest

-- 
Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware.
https://www.avast.com/antivirus