Subject: Re: [Hls-interest] Survey: signaling HLS client capability

From: "Weil, Nicolas" <[email-masked]>
Date: Fri, 28 May 2021 19:12:16 +0000

Side note on Will’s filters examples: while _HLS_admode filtering query arg would make sense as it’s purely related to HLS, I don’t think we need standardized parameters like _HLS_codec, _HLS_bitrate or _HLS_lang.

These parameters are indeed handled by origins/packagers across multiple formats already, not only for HLS. Playback URLs just need to be built with the query args supported by the origin.

Examples:
https://docs.unified-streaming.com/documentation/vod/playout-control.html#using-dynamic-track-selection
https://docs.microsoft.com/en-us/azure/media-services/latest/filters-dynamic-manifest-concept
https://docs.aws.amazon.com/mediapackage/latest/ug/manifest-filtering.html

Thanks,
Nicolas

----------------
Nicolas Weil | Senior Product Manager – Media Services
AWS Elemental

From: Hls-interest <[email-masked]> On Behalf Of Roger Pantos
Sent: Thursday, May 27, 2021 11:53 AM
To: [email-masked]
Subject: RE: [Hls-interest] Survey: signaling HLS client capability

Great discussion, thank you to everyone for contributing insights. This is a single reply with responses to several of the major points; please let me know if it gets too hard to follow:

On May 21, 2021, at 1:48 PM, Law, Will <wilaw=[email-masked]<mailto:wilaw=[email-masked]>> wrote:
Architecturally, I see concerns in having servers making decisions for clients. The reasons for this are two-fold:
1. There is a clear and strong movement towards client privacy. This includes the deprecation and freezing of User-Agent and other client identifiers, which is well under way. If not already, it will become increasingly harder for servers to make any sort of reliable inference about a client's capabilities based purely upon the request.

Agreed that privacy is a concern. I expect that this kind of a signal would typically follow in lockstep with either the OS release (Apple native clients) or app generation (HLS.js et al), so the anonymity set (per https://w3c.github.io/fingerprinting-guidance/) would remain fairly large.

To give everyone a sense of where we're leaning at the moment, I'm thinking of a single _HLS_capabilities query parameter of the form _HLS_capabilities="CS-1,INT,LL,V-9" (alpha-ordered).
  CS-1 : client supports Content Steering protocol version 1
  INT : support Interstitials
  LL : supports low-latency
  V-9 : understands HLS protocol version 9

Haven't decided on the exact set yet; I'm interested in hearing about requirements.

Two schools of thought then as to how clients can handle diversity of content & features:

  1.  Clients should ask explicitly for what they want in a way that is consistent and the response cacheable . You have already established a convention of reserved _HLS_ query args  on which the newer features of HLS are quite dependent. It would make sense to extend this to allow players to add additional filters as they request content. Examples of filters:

     *   _HLS_codec=hevc, avc
     *   _HLS_bitrate=500000-4000000
     *   _HLS_lang=fr
     *   _HLS_admode=interstitial

  1.  Another approach is that the main playlist is a menu of all content available, and that the client is capable of extracting the pieces from it that it wants. Real-time feature-based detection, rather than brittle device categorization, is the sustainable way forward. We already require smart clients to play back HLS. Those same clients are aware of which features (low latency, steering, interstitial support) they support as well as are capable of discovering the capabilities (codec, HDCP, HDR etc) of the host device. This approach gives the same main playlist to all clients, simplifying distribution.

Right. [2] is what we have today, and in general I still prefer having the server vend everything and having the client be smart enough to choose what to use.

We're considering [1] for cases in which providing everything in a single master playlist places too much of a burden on the server. Gaetan gave an example where offering both (new) interstitials and (backward-compatible) segment stitching ad mode would be a big lift.

One problem with approach [1] is that you would want to use it when the client requests the main playlist, however it doesn’t know if the origin supports such filtering. It could therefore request AVC as the only codec, but receive a static playlist back from an origin that did not support filtering. There could be a playlist attribute added to indicate that the origin acknowledges the requests and has implemented the filters. If the client does not see such an acknowledgement echoed back, then it would know that it can’t trust the origin and would have to implement some decisions of its own. One could then argue that if it can make those decisions on its own, then why entertain the complexity of the filters? If a client is modern enough to add these filters to its requests, then couldn’t it simply apply them itself?

My advice to clients would be not to rely on any kind of playlist filtering in response to the capabilities offered, and to be prepared to deal with anything the server throws its way.

Related, _HLS_capabilities is clearly optional, on both sides.

On May 21, 2021, at 2:24 PM, Alex Giladi <[email-masked]<mailto:[email-masked]>> wrote:

Regardng [1] in Will's e-mail, I'm concerned about nuclear
proliferation of query parameters and special snowflake server-side
logic.

Agreed. I think that if we restrict it to a single query parameter, it becomes manageable for CDNs to include it or exclude it from their cache keys as appropriate.

Regarding a proliferation of special snowflake server behavior, I agree that this feature is somewhat prone to what I'd consider misuse (or at least overuse). I don't have an immediate solution for that.

On May 25, 2021, at 12:30 PM, Gaëtan Hervouet <gaetanhervouet=[email-masked]<mailto:gaetanhervouet=[email-masked]>> wrote:
I do agree with Will’s response. Relying on user-agent [1] is indeed becoming harder and we have to move away from that. Cacheability and debuggability are of course valid points [2] which is why the query parameters do seem like a good solution (as long as there is no explosion of combinations). I do see the problem with having the query parameter sent on master manifest request impact the content of the variant playlists; it would break cacheability. However, if that parameter were to be set for all requests of the playback session (including variants) then the problem goes away and the response is now deterministic and cacheable.

I'm not sure if I agree with this last bit. If a client indicates that it does (or doesn't) support interstitials, the server can hand it a master playlist with either interstitial-laden URLs or stitched URLs. There is no reason that I can see for them to be the same URLs, so they ought to be independently cacheable.

We may also want to consider an approach where the filters are not "features" but rather an HLS spec/draft version that the player is compatible with.

My thinking is to provide both. Note that solely providing a spec/draft version is insufficient because a client might implement aspect of the spec (interstitials) but not another (low-latency).

Roger.

---

From: Ofer Shem Tov <[email-masked]>
Date: Sun, 30 May 2021 11:23:39 +0300

On the Yahoo side of Verizon Media we have been using an internal Video
Policy API (since 2015) for filtering streams based on user agent, our
player capabilities (per platform), content type (sports, finance, etc.)
and the stream variants information.

The reasons to do that were:

- Compatibility with older devices (for example b-frames on some devices
used to be an issue)
- Devices not supporting 60FPS
- Filtering out low bitrate streams for TV devices like Apple TV
- Filtering out high bit rates for phones (data cap considerations, etc.)
- Selecting the starting bitrate by putting it first in the master playlist
- Removing higher bitrates in case of network issues (for events with
millions of concurrent)
- Testing non HTTP transports like UDP and QUIC for compatible players
- CDN selection (for compatibility issues with some players)

One important note is that in addition to bitrate filtering we added
resolution filtering when VBR was added to HLS. It is an easier filter to
manage because of different types of encodings and content (sports has much
higher bitrate than finance for the same resolution).

Ofer

---------- Forwarded message ----------
From: Roger Pantos <[email-masked]>
To: [email-masked]
Cc:
Bcc:
Date: Thu, 20 May 2021 09:00:30 -0700
Subject: [Hls-interest] Survey: signaling HLS client capability
With the increasing number of optional features in HLS (such as low
latency, content steering, interstitial support, different codecs, etc.),
there is a desire in some quarters to present different streams to
different devices, beyond the regular variant stream selection performed on
the client.

We're considering some kind of signal of client capability, such as a query
parameter on the Master Playlist URL, to support this kind of server
decision. But before we go too far I'd like to understand if it would
actually be of practical benefit, and what universe of use cases it makes
sense to support. So:

- Do you think you would use this kind of thing? For what?

- If you are already doing something along these lines, perhaps driven by
the User-Agent string or something else, what requirement is driving that?

(Alternately, if you think that this would only make things worse, let us
know why!)

Feel free to respond here, or to send your response privately to
[email-masked].

thanks,

Roger Pantos
Apple Inc.

-- 
<http://www.verizonmedia.com>

Ofer Shem Tov

Distinguished Software Dev Engineer
Media Platform

M +972548842664
2A Jabotinsky St.
Amot Atrium 36th Floor
Ramat Gan

<http://www.facebook.com/verizonmedia>   <http://twitter.com/verizonmedia>
<https://www.linkedin.com/company/verizon-media/>
<http://www.instagram.com/verizonmedia>