Subject: [Hls-interest] Token Auth support

From: "Law, Will" <[email-masked]>
Date: Wed, 24 Mar 2021 18:51:58 +0000

A number of CDN & workflow operators are concerned that the current design of HLS Content Steering (even with the QUERY-PARAMS object just unveiled this morning) is incompatible with the wide variety of token auth implementations used in practice today in multi-CDN implementations. We had an ad-hoc meeting last week and the notes are publicly available<https://docs.google.com/document/d/1Umhr7ioyZ0Svjp93UDajYJRb88MKvTCabNsAQuHnXGA>. Rather than simply reporting the problem, we also attempted to propose a solution. This we term ‘Redundant Stream Steering” and it is documented here<https://docs.google.com/document/d/1f4_9hU_4DXPiKusF1QMUX3OY2DRbYnEWimGcj7T1-ZE/>. Discussion is ongoing. We wanted to surface this to Apple sooner rather than later, as we appreciate that the window for design changes in HLS Content Steering is growing smaller. As part of a larger concern, questions on the robustness of content steering across Server-Side Ad Insertion were also raised. The SVA are looking in to these issues under the guidance of Glenn and will report separately.

Will, on behalf of

David Brown - Amazon CloudFront
Peter Chave - Akamai
Phil Cluff - Mux
Andrew Crowe - Limelight Networks
Glenn Goldstein - Lumen
Trevor Hunsaker - Verizon Media
Jessica Obermark - Verizon Media
Rob Roskin - Lumen
Zac Shenker – Fastly
PJ Speelmans - Theoplayer
Nicolas Weil - AWS Elemental

---

From: Ray van Brandenburg <[email-masked]>
Date: Wed, 24 Mar 2021 23:06:58 +0100

Interesting discussion, thanks for sharing the notes Will. 

While I’m not sure it will resolve all issues listed in the meeting minutes, I just wanted to note that IETF is close to publishing an RFC for token auth across CDNs based on JWT, with HLS being one of the primary use cases. Full disclosure: I’m one of the co-authors, along with Phil Sorber who is working for Apple.

See https://tools.ietf.org/html/draft-ietf-cdni-uri-signing-21 for the latest version 

Best regards,

Ray

> On 24 Mar 2021, at 19:52, Law, Will <wilaw=[email-masked]> wrote:
> 
> 
> A number of CDN & workflow operators are concerned that the current design of HLS Content Steering (even with the QUERY-PARAMS object just unveiled this morning) is incompatible with the wide variety of token auth implementations used in practice today in multi-CDN implementations. We had an ad-hoc meeting last week and the notes are publicly available. Rather than simply reporting the problem, we also attempted to propose a solution. This we term ‘Redundant Stream Steering” and it is documented here. Discussion is ongoing. We wanted to surface this to Apple sooner rather than later, as we appreciate that the window for design changes in HLS Content Steering is growing smaller. As part of a larger concern, questions on the robustness of content steering across Server-Side Ad Insertion were also raised. The SVA are looking in to these issues under the guidance of Glenn and will report separately.
>  
> Will, on behalf of
>  
> -- 
> Hls-interest mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hls-interest

---

From: "Goldstein, Glenn" <[email-masked]>
Date: Thu, 25 Mar 2021 16:17:08 +0000

Thanks Will. The SVA is looking at HLS Content Steering Proposal as well as the Redundant Stream Steering that Will has mentioned, and should have responses to both by end of next week. The SVA group working on the response represents a diverse set of perspectives across the streaming industry, including content providers, CDNs, Ad Insertion vendors, QOS reporting vendors, multi-CDN orchestration vendors, and video player developers.

From: "Law, Will" <[email-masked]>
Date: Wednesday, March 24, 2021 at 2:51 PM
To: "[email-masked]" <[email-masked]>
Subject: [Hls-interest] Token Auth support

A number of CDN & workflow operators are concerned that the current design of HLS Content Steering (even with the QUERY-PARAMS object just unveiled this morning) is incompatible with the wide variety of token auth implementations used in practice today in multi-CDN implementations. We had an ad-hoc meeting last week and the notes are publicly available<https://imss91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f1Umhr7ioyZ0Svjp93UDajYJRb88MKvTCabNsAQuHnXGA&umid=86EA5988-BE4C-F605-8EA8-9679F9DE70A3&auth=19120be9529b25014b618505cb01789c5433dae7-0be4600458c2611fe1dfe195c2bb2248f22c30ff>. Rather than simply reporting the problem, we also attempted to propose a solution. This we term ‘Redundant Stream Steering” and it is documented here<https://imss91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f1f4%5f9hU%5f4DXPiKusF1QMUX3OY2DRbYnEWimGcj7T1%2dZE%2f&umid=86EA5988-BE4C-F605-8EA8-9679F9DE70A3&auth=19120be9529b25014b618505cb01789c5433dae7-a70c7d9f97b88304455dce59fb9b6bbdc2e3717f>. Discussion is ongoing. We wanted to surface this to Apple sooner rather than later, as we appreciate that the window for design changes in HLS Content Steering is growing smaller. As part of a larger concern, questions on the robustness of content steering across Server-Side Ad Insertion were also raised. The SVA are looking in to these issues under the guidance of Glenn and will report separately.

Will, on behalf of

This communication is the property of Lumen Technologies and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

---

From: Roger Pantos <[email-masked]>
Date: Thu, 25 Mar 2021 14:42:24 -0700

Hello Will (et al); thanks for the thoughtful discussion. Let me go through the questions you raised (as I see them) and offer some thoughts:

A - Different CDNs may need different auth tokens (at least until we reach the shiny future of cross-CDN URI signing or something similar)

Per-CDN auth tokens can be supplied by the steering server via QUERY-PARAMS. Which leads us directly to:

B - A steering server will need proper authorization to produce or obtain per-CDN auth tokens 

As noted, this will likely require that the steering manifest URI itself carry some kind of auth parameter.

C - Akamai modifies Master Playlist on the fly to generate tokens for Media Playlists

This is a bit of an odd duck, in that it requires that the Master Playlist come specifically from Akamai as opposed to a different CDN. Nonetheless, a few things to note:

- As it produces the Master Playlist, Akamai has an opportunity to provide the same token(s) to the steering server via the steering manifest URL
- Alternately, it seems that a properly-authenticated steering server (see point B) ought to be able to obtain the proper Akamai tokens from Akamai
- And obviously, in an Akamai-centric world, Akamai might be operating the steering server as well. That presumably makes things simpler (at least for Akamai).

D - Akamai modifies Media Playlists on the fly to generate tokens for Media Segments

I don’t think that this changes at all with Content Steering. Akamai will still receive the Media Playlist requests that are directed to it, and it can continue to rewrite the response to tokenize segments

E - Have we addressed redirects?

Not sure exactly the context around this question, but certainly I would expect that servers can return 302 etc. to redirect Steering Manifest requests.

F - Hulu SSAI servers Media Playlists from ad domains & program segments from Content Service Provider domain

No change here; all that still works.

There is a deeper discussion to be had around content steering for the ad content itself vs. the program content. Not steering ads is easy; steering ads + program content in the same direction (east coast or west coast for example) is fairly easy. Steering them independently gets complicated. (But is it necessary?)

G -  Should clients drive steering, instead of servers?

There are a couple of arguments against this. Clients exist across a spectrum of different implementations and different versions of those implementations, each of which tends to have slightly different behavior. That makes it difficult to manage load on any kind of aggregate basis: one set of clients may be steering in one direction while a different set steers in the opposite direction.

Also, providers are the ones who tend to have global visibility of load and outages. The providers I’ve spoken to want the servers to make the steering decisions, not the clients. The want all clients (or at least as many as possible) to follow the same simple set of directions, reliably.

H - If servers drive steering, isn’t the easiest way to do that to use DNS ?

DNS in general is too limited in terms of the client context it has to work with. (Imagine trying to solve the auth token problem above via DNS.) It’s also distributed, which means that the provider who is providing the steering inputs needs to coordinate those with the DNS of every CDN. And turning a DNS into a  content steering server (in other words making it a “smart” DNS) would impact its scalability.

Roger.

> On Mar 24, 2021, at 11:51 AM, Law, Will <wilaw=[email-masked]> wrote:
> 
> A number of CDN & workflow operators are concerned that the current design of HLS Content Steering (even with the QUERY-PARAMS object just unveiled this morning) is incompatible with the wide variety of token auth implementations used in practice today in multi-CDN implementations. We had an ad-hoc meeting last week and the notes are publicly available <https://docs.google.com/document/d/1Umhr7ioyZ0Svjp93UDajYJRb88MKvTCabNsAQuHnXGA>. Rather than simply reporting the problem, we also attempted to propose a solution. This we term ‘Redundant Stream Steering” and it is documented here <https://docs.google.com/document/d/1f4_9hU_4DXPiKusF1QMUX3OY2DRbYnEWimGcj7T1-ZE/>. Discussion is ongoing. We wanted to surface this to Apple sooner rather than later, as we appreciate that the window for design changes in HLS Content Steering is growing smaller. As part of a larger concern, questions on the robustness of content steering across Server-Side Ad Insertion were also raised. The SVA are looking in to these issues under the guidance of Glenn and will report separately.
>  
> Will, on behalf of
>