Subject: [Hls-interest] Survey: signaling HLS client capability

From: Roger Pantos <[email-masked]>
Date: Thu, 20 May 2021 09:00:30 -0700

With the increasing number of optional features in HLS (such as low latency, content steering, interstitial support, different codecs, etc.), there is a desire in some quarters to present different streams to different devices, beyond the regular variant stream selection performed on the client.

We're considering some kind of signal of client capability, such as a query parameter on the Master Playlist URL, to support this kind of server decision. But before we go too far I'd like to understand if it would actually be of practical benefit, and what universe of use cases it makes sense to support. So:

- Do you think you would use this kind of thing? For what? 

- If you are already doing something along these lines, perhaps driven by the User-Agent string or something else, what requirement is driving that? 

(Alternately, if you think that this would only make things worse, let us know why!)

Feel free to respond here, or to send your response privately to [email-masked].

thanks,

Roger Pantos
Apple Inc.

---

From: Trevor Hunsaker <[email-masked]>
Date: Thu, 20 May 2021 14:28:59 -0600

As a "people already doing something along these lines... what requirement
is driving that":

For us At Verizon Media, the requirement is often to omit some feature that
is causing an issue in a player over which our customers have little or no
control. For example, we semi-recently discovered that some
AirPlay2-enabled smart TVs are incapable of handling full-segment-encrypted
.aac alternate audio tracks, and crash when presented with them (submitted
bug FB8948360 about this). Our encoding pipeline takes longer to adapt than
our manifest generation system, so we added support for a URL parameter as
a means of instructing our manifest server to omit such tracks from the
manifest. We have a small handful or two of such URL params floating in our
system, as workarounds for all kinds of players in the ecosystem.

We've also used the User-Agent to help us decide which HLS version a device
is capable of supporting, and try to be compliant about which features we
include for a given HLS version.

On Thu, May 20, 2021 at 10:00 AM Roger Pantos <rpantos=
[email-masked]> wrote:

> With the increasing number of optional features in HLS (such as low
> latency, content steering, interstitial support, different codecs, etc.),
> there is a desire in some quarters to present different streams to
> different devices, beyond the regular variant stream selection performed on
> the client.
>
> We're considering some kind of signal of client capability, such as a
> query parameter on the Master Playlist URL, to support this kind of server
> decision. But before we go too far I'd like to understand if it would
> actually be of practical benefit, and what universe of use cases it makes
> sense to support. So:
>
> - If you are already doing something along these lines, perhaps driven by
> the User-Agent string or something else, what requirement is driving that?
>
> Feel free to respond here, or to send your response privately to
> [email-masked].
>
>
> thanks,
>
> Roger Pantos
> Apple Inc.
> --
> Hls-interest mailing list
> [email-masked]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_hls-2Dinterest&d=DwICAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XOy1sMJB-v4-pT8o8aTMSGXStN19eHAo6D_AaKvUJso&m=gG3_laqdYt1I2B9tqBUeHVl36ZLo-643YDPapqmGjm4&s=6cLzncMjPm1DCxoql2i_D9waQJFNTn4k3tF3J-qw-Eo&e=
>

---

From: "Law, Will" <[email-masked]>
Date: Fri, 21 May 2021 20:48:03 +0000

Hi Roger

Architecturally, I see concerns in having servers making decisions for clients. The reasons for this are two-fold:

1. There is a clear and strong movement towards client privacy. This includes the deprecation and freezing of User-Agent and other client identifiers, which is well under way. If not already, it will become increasingly harder for servers to make any sort of reliable inference about a client's capabilities based purely upon the request.

2. Servers should respond unambiguously to client requests. A brittle system is one in which 3 clients ask for the same asset (a main playlist for example), but each is returned a different version of that asset based upon which codecs that server thinks the client supports. We now have three bit-different assets with the same file name, which causes cache consistency problems. QA becomes difficult, because you get a different answer depending upon who is asking. A deterministic system, in which each request has a single possible response, is far easier to debug and eventually far more robust when it comes to scaling up complex systems.

Two schools of thought then as to how clients can handle diversity of content & features:

  1.  Clients should ask explicitly for what they want in a way that is consistent and the response cacheable . You have already established a convention of reserved _HLS_ query args  on which the newer features of HLS are quite dependent. It would make sense to extend this to allow players to add additional filters as they request content. Examples of filters:
     *   _HLS_codec=hevc, avc
     *   _HLS_bitrate=500000-4000000
     *   _HLS_lang=fr
     *   _HLS_admode=interstitial
  2.  Another approach is that the main playlist is a menu of all content available, and that the client is capable of extracting the pieces from it that it wants. Real-time feature-based detection, rather than brittle device categorization, is the sustainable way forward. We already require smart clients to play back HLS. Those same clients are aware of which features (low latency, steering, interstitial support) they support as well as are capable of discovering the capabilities (codec, HDCP, HDR etc) of the host device. This approach gives the same main playlist to all clients, simplifying distribution.

One problem with approach [1] is that you would want to use it when the client requests the main playlist, however it doesn’t know if the origin supports such filtering. It could therefore request AVC as the only codec, but receive a static playlist back from an origin that did not support filtering. There could be a playlist attribute added to indicate that the origin acknowledges the requests and has implemented the filters. If the client does not see such an acknowledgement echoed back, then it would know that it can’t trust the origin and would have to implement some decisions of its own. One could then argue that if it can make those decisions on its own, then why entertain the complexity of the filters? If a client is modern enough to add these filters to its requests, then couldn’t it simply apply them itself?

The problem with approach [2] is legacy clients. If every feature implemented is backwards compatible, then perhaps you are OK and the client gets a degraded but still functional experience. Eventually Apple will want to introduce a breaking change for efficiency (next-gen interstitial ads for example) and at that point playlists would need to be hard-coded to specific client sets, which is exactly the situation we are in today if you examine most large content distribution solutions.

Neither approach is without issues. Overall, I think an implementation of [1] is a practical means of introducing new features to modern clients, as only modern clients will know how to ask for them. In parallel, I’d also like to see Apple moving the industry in the direction of making the players smarter. Player behavior specifications, updateable player logic (decoupled from device updates) , player conformance tests, and test suites would make for more manageable content & feature diversity in the decades ahead.

Cheers

Will

On 5/20/21, 9:00 AM, "Roger Pantos" <rpantos=[email-masked]> wrote:

    thanks,

    Roger Pantos

    Apple Inc.

    --

    [email-masked]

    https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/hls-interest__;!!GjvTz_vk!H1Hsba7y664WmwG3l-OQEw9A4O9jRoyGL0i2Y2-YPW42-Pz-zXuCdZq_qwwY$

---

From: "Ali C. Begen" <[email-masked]>
Date: Fri, 21 May 2021 23:55:27 +0300

Not that I don’t agree with what Will said in the rest of his email, I would like to express my 100x support for the statement below.

> On May 21, 2021, at 11:48 PM, Law, Will <wilaw=[email-masked]> wrote:
> 
> In parallel, I’d also like to see Apple moving the industry in the direction of making the players smarter. Player behavior specifications, updateable player logic (decoupled from device updates) , player conformance tests, and test suites would make for more manageable content & feature diversity in the decades ahead.

---

From: John Luther <[email-masked]>
Date: Fri, 21 May 2021 17:06:59 -0400

On Fri, May 21, 2021 at 4:55 PM Ali C. Begen <[email-masked]>
wrote:

> Not that I don’t agree with what Will said in the rest of his email, I
> would like to express my 100x support for the statement below.
>

+100 from JW Player

> > In parallel, I’d also like to see Apple moving the industry in the
> direction of making the players smarter. Player behavior specifications,
> updateable player logic (decoupled from device updates) , player
> conformance tests, and test suites would make for more manageable content &
> feature diversity in the decades ahead.
>
> --
> https://www.ietf.org/mailman/listinfo/hls-interest
>

---

From: Alex Giladi <[email-masked]>
Date: Fri, 21 May 2021 15:24:54 -0600

Regarding [1] in Will's e-mail, I'm concerned about nuclear
proliferation of query parameters and special snowflake server-side
logic.
While everything is implementable, a deployment with smarter and more
independent client selecting from a large (and growing) menu of
options (i.e., [2] scales better

On Fri, May 21, 2021 at 3:07 PM John Luther
<jluther=[email-masked]> wrote:
>
>
>
> On Fri, May 21, 2021 at 4:55 PM Ali C. Begen <[email-masked]> wrote:
>>

---

From: Gaëtan Hervouet <[email-masked]>
Date: Tue, 25 May 2021 15:30:23 -0400

Hi all,

I do agree with Will’s response. Relying on user-agent [1] is indeed
becoming harder and we have to move away from that. Cacheability and
debuggability are of course valid points [2] which is why the query
parameters do seem like a good solution (as long as there is no explosion
of combinations). I do see the problem with having the query parameter sent
on master manifest request impact the content of the variant playlists; it
would break cacheability. However, if that parameter were to be set for all
requests of the playback session (including variants) then the problem goes
away and the response is now deterministic and cacheable. We may also want
to consider an approach where the filters are not "features" but rather an
HLS spec/draft version that the player is compatible with.

We would also be totally fine with Will’s 2nd proposition however the
danger there is that the master playlist might explode in size if it needs
one EXT-X-STREAM-INF url for every combination of features. Sticking to the
client deciding what it can plays though, we could envision following an
LL-HLS approach where the EXT-X-SERVER-CONTROL tag appears in the *master*
manifest:

GET http://content.io./some-coment/master.m3u8

#EXTM3U

#EXT-X-VERSION:10

#EXT-X-SERVER-CONTROL:admode=interstitials

#EXT-X-STREAM-INF:BANDWIDTH=300000

http://content.io./some-content/variant1.m3u8
<http://content.io./some-coment/variant1.m3u8>

#EXT-X-STREAM-INF:BANDWIDTH=1000000

http://content.io./some-content/variant2.m3u8

Then for a device that supports interstitials, it requests the variant with:

GET http://content.io./some-coment/variant1.m3u8?_HLS_admode=interstitials
<http://content.io./some-coment/variant1.m3u8?_HLS_AdMode=interstitials>

Or a device that does not support interstitials:

GET http://content.io./some-coment/variant1.m3u8
<http://content.io./some-coment/variant1.m3u8?_HLS_AdMode=interstitials>

The query parameters would still allow the master and variant requests to
be cached deterministically and the client is the one making the decision
on what it plays while the server just sticks to advertising what it can
offer. Older players will ignore that EXT-X-SERVER-CONTROL tag and the
master playlist can stay relatively small and common to all users.

This example describes interstitials however there could be more
EXT-X-SERVER-CONTROL master attributes for important features, list would
have to be kept to a minimum to avoid an explosion of combination and keep
cache effective.

We think it’s important for that client compatibility feature to be
introduced at the same time as interstitials so that it can be used right
when it’s released instead of waiting years for a big enough number of
clients to support it. Supporting both interstitials and segment stitching
ad mode in one session does not really make sense even if it’s backward
compatible.

Cheers,
Gaetan

---

From: Daniel Weinberger <[email-masked]>
Date: Wed, 26 May 2021 01:33:05 +0200

I agree with most of what Will said. However, from the client perspective I
want to comment on this part specifically:

One problem with approach [1] is that you would want to use it when the
> client requests the main playlist, however it doesn’t know if the origin
> supports such filtering. It could therefore request AVC as the only codec,
> but receive a static playlist back from an origin that did not support
> filtering. There could be a playlist attribute added to indicate that the
> origin acknowledges the requests and has implemented the filters. If the
> client does not see such an acknowledgement echoed back, then it would know
> that it can’t trust the origin and would have to implement some decisions
> of its own. One could then argue that if it can make those decisions on its
> own, then why entertain the complexity of the filters? If a client is
> modern enough to add these filters to its requests, then couldn’t it simply
> apply them itself?

It's certainly a challenge to handle the case where the server/CDN doesn't
understand the filtering, but
a) it would lead to simply what we have today - a static playlist
b) one could argue that the application should in many use cases know
whether the backend can handle it or not.

I can't agree with the last sentence though. In fact, approach [1] provides
more power to the application that is using the player as it can simply
pass a playlist URL to the player with the correct parameters included.
This makes it completely transparent for the HLS client and any HLS client
can benefit from this new feature, it doesn't need to know about it at all
- as long as the application does.
This can be very powerful when thinking of (also legacy) devices and
platforms with a built-in HLS client, which often don't get updated by a
vendor at all or sporadically.

I agree with the concerns around approach [2] and in my view approach [1]
has fewer down-sides than [2], at least from a client perspective.

Regards,
Daniel
________
Bitmovin

---

From: Roger Pantos <[email-masked]>
Date: Thu, 27 May 2021 11:52:38 -0700

Great discussion, thank you to everyone for contributing insights. This is a single reply with responses to several of the major points; please let me know if it gets too hard to follow:

> On May 21, 2021, at 1:48 PM, Law, Will <wilaw=[email-masked] <mailto:wilaw=[email-masked]>> wrote:

Agreed that privacy is a concern. I expect that this kind of a signal would typically follow in lockstep with either the OS release (Apple native clients) or app generation (HLS.js et al), so the anonymity set (per https://w3c.github.io/fingerprinting-guidance/ <https://w3c.github.io/fingerprinting-guidance/>) would remain fairly large.

To give everyone a sense of where we're leaning at the moment, I'm thinking of a single _HLS_capabilities query parameter of the form _HLS_capabilities="CS-1,INT,LL,V-9" (alpha-ordered).
  CS-1 : client supports Content Steering protocol version 1
  INT : support Interstitials
  LL : supports low-latency
  V-9 : understands HLS protocol version 9

Haven't decided on the exact set yet; I'm interested in hearing about requirements.

> Clients should ask explicitly for what they want in a way that is consistent and the response cacheable . You have already established a convention of reserved _HLS_ query args  on which the newer features of HLS are quite dependent. It would make sense to extend this to allow players to add additional filters as they request content. Examples of filters:
> _HLS_codec=hevc, avc
> _HLS_bitrate=500000-4000000
> _HLS_lang=fr
> _HLS_admode=interstitial
> Another approach is that the main playlist is a menu of all content available, and that the client is capable of extracting the pieces from it that it wants. Real-time feature-based detection, rather than brittle device categorization, is the sustainable way forward. We already require smart clients to play back HLS. Those same clients are aware of which features (low latency, steering, interstitial support) they support as well as are capable of discovering the capabilities (codec, HDCP, HDR etc) of the host device. This approach gives the same main playlist to all clients, simplifying distribution.

Right. [2] is what we have today, and in general I still prefer having the server vend everything and having the client be smart enough to choose what to use. 

We're considering [1] for cases in which providing everything in a single master playlist places too much of a burden on the server. Gaetan gave an example where offering both (new) interstitials and (backward-compatible) segment stitching ad mode would be a big lift.

> One problem with approach [1] is that you would want to use it when the client requests the main playlist, however it doesn’t know if the origin supports such filtering. It could therefore request…

My advice to clients would be not to rely on any kind of playlist filtering in response to the capabilities offered, and to be prepared to deal with anything the server throws its way. 

Related, _HLS_capabilities is clearly optional, on both sides.

> On May 21, 2021, at 2:24 PM, Alex Giladi <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Regardng [1] in Will's e-mail, I'm concerned about nuclear

Agreed. I think that if we restrict it to a single query parameter, it becomes manageable for CDNs to include it or exclude it from their cache keys as appropriate.

Regarding a proliferation of special snowflake server behavior, I agree that this feature is somewhat prone to what I'd consider misuse (or at least overuse). I don't have an immediate solution for that.

> On May 25, 2021, at 12:30 PM, Gaëtan Hervouet <gaetanhervouet=[email-masked] <mailto:gaetanhervouet=[email-masked]>> wrote:
> I do agree with Will’s response. Relying on user-agent [1] is indeed becoming harder and we have to move away from that. Cacheability and debuggability are of course valid points [2] which is why the query parameters do seem like a good solution (as long as there is no explosion of combinations). I do see the problem with having the query parameter sent on master manifest request impact the content of the variant playlists; it would break cacheability. However, if that parameter were to be set for all requests of the playback session (including variants) then the problem goes away and the response is now deterministic and cacheable.

I'm not sure if I agree with this last bit. If a client indicates that it does (or doesn't) support interstitials, the server can hand it a master playlist with either interstitial-laden URLs or stitched URLs. There is no reason that I can see for them to be the same URLs, so they ought to be independently cacheable. 

> We may also want to consider an approach where the filters are not "features" but rather an HLS spec/draft version that the player is compatible with. 

My thinking is to provide both. Note that solely providing a spec/draft version is insufficient because a client might implement aspect of the spec (interstitials) but not another (low-latency).

Roger.