Subject: Re: [MBONED] mboned: draft-ietf-bier-multicast-http-response

From: "Holland, Jake" <[email-masked]>
Date: Mon, 24 Jan 2022 23:01:46 +0000

Hi,

Sorry for the very late response to this.

I read through draft-ietf-bier-multicast-http-response.  It was
an interesting solution described for an interesting use case.

My high-level overview is that I was a bit confused about its
relationship to BIER, and I suspect some restructuring or high-
level changes to the doc might make it more useful.

Some specific points jumped out at me as possibly worth raising,
which I think mostly illustrate why I landed on that high-level
summary, and I'll detail them below.

I'm offering this review as purely advisory, not an objection to
moving the doc forward.  I'm just trying to explain where I found
things confusing and point out things that I thought might be worth
considering more carefully, in hopes you find it helpful.  Please
feel free to use these comments or not, as you see fit.

1. This draft seems to be informational about a use case, but the
BIER charter currently says:

" 3) Use Case: The WG will produce one use-case document that clearly
articulates the potential benefits of BIER for different use-cases."

I guess this would come with a minor charter update to allow for multiple
use-case docs, unless it merges with the use cases doc?  (Unless this
falls under another charter point?  I didn't see one...)

Anyway, not really a problem with the doc per se, just kind of a
confusing thing I noticed.

2. This looks like a description of a system just like the one in this
paper, which I think shares an author with this draft?:

"fCDN: A Flexible and Efficient CDN Infrastructure without DNS
Redirection or Content Reflection"
(Al-Naday, Reed, Riihijarvi, Trossen, Thomos, Al-Khalidi)
https://arxiv.org/pdf/1803.00876.pdf

(I recently saw that paper because it was referenced from the latest
coinrg use cases doc.)

It seems most of the explanation of this system could be handled by
reference to that paper, rather than explaining it in the i-d, unless
there's some essential differences that escaped me?

Anyway, the point is that at a high level, I think this i-d would
benefit by focusing much more on why the kind of scalability BIER
provides is essential to this use case, ideally with a quantitative
model for how much more can be supported with BIER as compared to a
similarly provisioned system using traditional multicast.

3. Both the paper and the i-d seem to elide the way multicast IP is
used and mapped into BIER's bit index settings (or not?) in this
system.

I think this is describing a system where the SNAP does BIER
encapsulation based on the contents carried in the packets and which
CNAPs need to receive that content.  While left unsaid, it seems
designed so the BIER index mapping (and the corresponding set of
CNAP receivers) can be completely decoupled from the mapping of
content into its IP addressing, correct?

I wasn't clear from the docs whether the intent was to actually
share (S,G) and ports at the IP/UDP layer, and have different CNAPs
getting different content for sockets listening on the same address
depending on the bits set in the BIER header at the SNAP, (a feat
that only be accomplished by BIER, as it breaks the group addressing
model of IP multicast), or whether this is intended to have each
piece of content mapped to a specific (S,G) and for CNAPs to listen
on all the mapped UDP sockets dynamically for each piece of content
they're expecting, in addition to the SNAPs sending BIER bit-fields
that match the IP mapping of the content (a feat that could be done
in a fast-enough and scalable-enough multicast IP system, but can
benefit from the scaling benefits of BIER).

I also wasn't clear on if it's that 2nd one, whether the client thrash
on sockets per individual piece of content would have any similar
scaling problems as HTTP/1.0 in the CNAP that could impact its
suitability, if it has trouble reusing active sockets instead of
creating new ones.  (Or if it's the first, whether non-cacheable
pass-through content with the same path and differentiated by cookies
or something would have a risk of collision, or shouldn't be
transported by this system, or what exactly.)

4. I didn't see a description of how reliability is handled?  I did
see "We assume TCP friendly transport, which can assure reliable
delivery.", but I don't think that's true--TCP-friendly is a term I
only know from UDP congestion control (either TFRC or TFMRC), but I
believe systems that use it still can suffer loss.

Anyway, in DASH MABR I think reliability in the multicast transport
comes via either version of FLUTE (RFCs 6726 or 3926) (though I think
without the WEBRC congestion controller), and in the Cablelabs reference
I think it uses ROUTE, which IIRC uses something based on NORM (RFC
5740).

Of course HTTP transport will need reliability, but I didn't see an
explanation of what the supported options are for how this system achieves
it, or an explanation of how the different design choices impact the
performance characteristics (and in particular whether there's any impact
to the scalability considerations that BIER brings in cases where there's
loss in different parts of the network). 

5. This bit from section 5 seems to misrepresent the way that systems like
DASH MABR work:
        ... As soon
   as there is at least one Client connected to a CNAP for one
   particular content, the CNAP would join all 11 multicast groups for
   this content.

Usually you would just join the one (or 2, if audio is separated from
video) relevant stream for the single client's bitrate, and the systems
that understand that it's DASH content as opposed to generic random HTTP
would also understand that being joined to a particular bitrate implies a
likelihood of remaining joined to the same bitrate.  (Or at least: I
haven't seen implementations that would automatically join all bitrates
given a subscription to any of them?).

Unlike the transparent reactive HTTP approach outlined in this doc,
behavior for things like DASH and ROUTE are closer to a server push
to pre-load a cache either embedded in the receiving app or on a
caching server adjacent to it.

6.  The architecture for supporting the security model probably would
benefit from a section explaining it earlier in the doc than the security
considerations section at the end.

This whole setup relies on the BIER domain understanding the contents,
so this system is only applicable for a CDN-like system where the CNAPs
(and SNAPs) have full knowledge of the contents, and HTTPS is only used
downstream from the CNAPs and upstream from the SNAPs.  It was hard for
me to read thru the whole rest of the doc with that hanging over my
thoughts, and thinking you were only going to apply to non-https.

Whether or not you move that info to clarify the architecture's approach
to https to an earlier section, the security considerations section for
a system like this (outside of the design of having CDN-like key sharing
to support https downstream) probably also should either reference
something like RFC 6707's security considerations that gives an overview
of CDN security considerations or copy some of its contents that would
apply to this system.

But more importantly, it should probably say more about the considerations
NOT covered by that doc.  For example, it should cover the requirement
to control all the forwarding nodes in the bier domain to ensure no
attacker-controlled on-path forwarders have an opportunity to change the
packets (or mention the recommended methods to prevent changing of the
packets by on-path forwarders).

Nits:
2nd paragraph on 5.2 doesn't end with a period, and I'm not sure the
sentence was completed.

On 12-20, 12:17 PM, "Toerless Eckert" <[email-masked]> wrote:

Dear Mboned WG

I would like to invite WG members to review, and comment to [email-masked]
subject draft. It is very much about aspects of using BIER that fall
into the scope of what i think should be very much of interest to Mboned 
member.

Here is a a motivational summary of the purpose of this document:

  Adaptive bitrate streaming (such as today DASH) for live-streaming or short-tail can
  use Multicast to improve network efficiency by eliminating duplicate transmission
  of the same data segments to multiple users requiring the same bitrate. This
  was shown and used in products as early as 200x (e.g.: Digital Fountain). 
  These solutions never proliferated because of the intrinsic signaling problems
  of IP Multicast causing severe performance/scale/operational issues.

  These IP Multicast limitations can today be overcome with the novel BIER
  multicast mechanism. This document describes an application solution layer
  architecture in which BIER is used as the multicast mechanism for HTTP
  adaptive bitrate streaming including the use of BIER-TE for including
  path steering optimizations into the solution. This document is solely
  informational and does not intend to establish any standards for the IETF.

Thank you so much
    Toerless (for the authors)

_______________________________________________
MBONED mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mboned

---

From: Dirk Trossen <[email-masked]>
Date: Thu, 27 Jan 2022 08:56:38 +0000

Hi Jake,

Many thanks for the response and apologies for the delay in coming back you on this.

Please see inline.

Best,

Dirk

-----Original Message-----
From: Holland, Jake [mailto:[email-masked]] 
Sent: 25 January 2022 00:02
To: Toerless Eckert <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: [MBONED] mboned: draft-ietf-bier-multicast-http-response

Hi,

I read through draft-ietf-bier-multicast-http-response.  It was an interesting solution described for an interesting use case.

My high-level overview is that I was a bit confused about its relationship to BIER, and I suspect some restructuring or high- level changes to the doc might make it more useful.
[DOT] Good point

Some specific points jumped out at me as possibly worth raising, which I think mostly illustrate why I landed on that high-level summary, and I'll detail them below.

I'm offering this review as purely advisory, not an objection to moving the doc forward.  I'm just trying to explain where I found things confusing and point out things that I thought might be worth considering more carefully, in hopes you find it helpful.  Please feel free to use these comments or not, as you see fit.

1. This draft seems to be informational about a use case, but the BIER charter currently says:

" 3) Use Case: The WG will produce one use-case document that clearly articulates the potential benefits of BIER for different use-cases."

I guess this would come with a minor charter update to allow for multiple use-case docs, unless it merges with the use cases doc?  (Unless this falls under another charter point?  I didn't see one...)

Anyway, not really a problem with the doc per se, just kind of a confusing thing I noticed.
[DOT] it's indeed confusing. The use case was first captured in the use case doc, so all well (Section 3.10 in its latest version). But given that we also wanted to highlight a possible solution over BIER, we went for a separate draft. So I'm not sure this is a charter issue since the draft is meant to outline technological solution aspects which a use case does not have. But I leave this point ultimately to the chairs to consider.

2. This looks like a description of a system just like the one in this paper, which I think shares an author with this draft?:

"fCDN: A Flexible and Efficient CDN Infrastructure without DNS Redirection or Content Reflection"
(Al-Naday, Reed, Riihijarvi, Trossen, Thomos, Al-Khalidi) https://arxiv.org/pdf/1803.00876.pdf

[DOT] Yes, indeed, I'm one of the co-authors, having worked with my colleagues at Essex University within an EU-funded project on the base technology, which the paper above then utilizes to outline an improved way for handling content delivery in CDNs. Historically, the baseline technology was SDN (using wildcard matching over the pathID information to perform linespeed stateless forwarding; the idea is highlighted in the ICC2016 paper on "Stateless multicast switching in software defined networks" (https://www.semanticscholar.org/paper/Stateless-multicast-switching-in-software-defined-Reed-Al-Naday/e42eec0a4cb8119af5e9b5dab66f37707a0605bf) with overlapping co-authors to the fCDN paper. The aforementioned inclusion into the use case document started our mapping onto BIER, which the draft discussed here then captured. But indeed, there are strong similarities since the baseline assumptions are the same (and similarities exist in the realization). 

(I recently saw that paper because it was referenced from the latest coinrg use cases doc.)
[DOT] Yes, I added it there, being a co-author on that draft. Thanks for reading the paper. I enjoyed the work even though we never pushed it beyond the initial submission (and rejection) to final publication. 

It seems most of the explanation of this system could be handled by reference to that paper, rather than explaining it in the i-d, unless there's some essential differences that escaped me?
[DOT] You are right that using this reference better to explain the key components would help. There are other references that capture the NAP-based realization to map HTTP (or IP communication) onto a path-based forwarding architecture, including the idea to realize dynamic multicast on the return path. A first demo of this idea was given at MWC2016, still showcasing smaller gain since the 'catchment interval' was kept at the response time of the server; when increasing to some 30% of the chunk length, this increases quite significantly but may lead to the problems with DASH-based control since the mechanism interprets the artificial delay as loss in the network, lowering the quality of the stream as a countermeasure.

Anyway, the point is that at a high level, I think this i-d would benefit by focusing much more on why the kind of scalability BIER provides is essential to this use case, ideally with a quantitative model for how much more can be supported with BIER as compared to a similarly provisioned system using traditional multicast.

[DOT] We discussed (back when more actively pursuing this) to add a quantitative part to it, including some simulation results we have. As a structure, it may indeed help to focus on the key part of BIER that play key part of the solution, before outlining the solution itself. I think implicitly we tried doing so through the requirements section, which is then followed by two possible realizations. 

3. Both the paper and the i-d seem to elide the way multicast IP is used and mapped into BIER's bit index settings (or not?) in this system.

I think this is describing a system where the SNAP does BIER encapsulation based on the contents carried in the packets and which CNAPs need to receive that content.  While left unsaid, it seems designed so the BIER index mapping (and the corresponding set of CNAP receivers) can be completely decoupled from the mapping of content into its IP addressing, correct?
[DOT] That is correct, indeed.

I wasn't clear from the docs whether the intent was to actually share (S,G) and ports at the IP/UDP layer, and have different CNAPs getting different content for sockets listening on the same address depending on the bits set in the BIER header at the SNAP, (a feat that only be accomplished by BIER, as it breaks the group addressing model of IP multicast), or whether this is intended to have each piece of content mapped to a specific (S,G) and for CNAPs to listen on all the mapped UDP sockets dynamically for each piece of content they're expecting, in addition to the SNAPs sending BIER bit-fields that match the IP mapping of the content (a feat that could be done in a fast-enough and scalable-enough multicast IP system, but can benefit from the scaling benefits of BIER).

[DOT] CNAPs forward requests to content (entirely unicast based) and receive responses to said requests via possible multicast responses, forming value unicast response towards the client. So there's no different content received than what the client requests; it may just arrive (at its CNAP) as a multicast response. The feat here that BIER enables is that you can group responses to the same content (chunk), delivered over individual CNAP-SNAP paths into a single multicast response by simply combining the individual paths together. This is fast and may yield a different result for any (multicast) response, depending on the synchronicity of the clients, including clients simply stopping sending a forward request. 

I also wasn't clear on if it's that 2nd one, whether the client thrash on sockets per individual piece of content would have any similar scaling problems as HTTP/1.0 in the CNAP that could impact its suitability, if it has trouble reusing active sockets instead of creating new ones.  (Or if it's the first, whether non-cacheable pass-through content with the same path and differentiated by cookies or something would have a risk of collision, or shouldn't be transported by this system, or what exactly.)

4. I didn't see a description of how reliability is handled?  I did see "We assume TCP friendly transport, which can assure reliable delivery.", but I don't think that's true--TCP-friendly is a term I only know from UDP congestion control (either TFRC or TFMRC), but I believe systems that use it still can suffer loss.
[DOT] Yes, the reliability part needs further work but we left it out here since it's not a BIER matter. The challenge here lies in the dynamic nature since your group may not be stable. Not sure about the formulation of that sentence above, I can either blame one of my co-authors or admit some sort of fatigue when writing it :-)

Anyway, in DASH MABR I think reliability in the multicast transport comes via either version of FLUTE (RFCs 6726 or 3926) (though I think without the WEBRC congestion controller), and in the Cablelabs reference I think it uses ROUTE, which IIRC uses something based on NORM (RFC 5740).
[DOT] One of the problems we encountered, which may be the most interesting interplay between transport and the lower layer multicast mechanism is that 'delaying' responses is a good thing, which in turn is interpreted by many reliability mechanisms as loss/congestion in the network. Say, if you have a chunk length of 2s, delaying your response after the first request received by 1s is likely enough for the multicast response to arrive in time but it's now a 1s latency measured by the receiving end! The upside is that you've now got 1s time for other requests to arrive, which may increase your (dynamic) group size.

Of course HTTP transport will need reliability, but I didn't see an explanation of what the supported options are for how this system achieves it, or an explanation of how the different design choices impact the performance characteristics (and in particular whether there's any impact to the scalability considerations that BIER brings in cases where there's loss in different parts of the network). 

5. This bit from section 5 seems to misrepresent the way that systems like DASH MABR work:
        ... As soon
   as there is at least one Client connected to a CNAP for one
   particular content, the CNAP would join all 11 multicast groups for
   this content.

Usually you would just join the one (or 2, if audio is separated from
video) relevant stream for the single client's bitrate, and the systems that understand that it's DASH content as opposed to generic random HTTP would also understand that being joined to a particular bitrate implies a likelihood of remaining joined to the same bitrate.  (Or at least: I haven't seen implementations that would automatically join all bitrates given a subscription to any of them?).

Unlike the transparent reactive HTTP approach outlined in this doc, behavior for things like DASH and ROUTE are closer to a server push to pre-load a cache either embedded in the receiving app or on a caching server adjacent to it.
[DOT] Yes, this indeed seems wrong and I must admit that the details on why this was written escape me at the moment but it would need to be addressed.

6.  The architecture for supporting the security model probably would benefit from a section explaining it earlier in the doc than the security considerations section at the end.

This whole setup relies on the BIER domain understanding the contents, so this system is only applicable for a CDN-like system where the CNAPs (and SNAPs) have full knowledge of the contents, and HTTPS is only used downstream from the CNAPs and upstream from the SNAPs.  It was hard for me to read thru the whole rest of the doc with that hanging over my thoughts, and thinking you were only going to apply to non-https.

Whether or not you move that info to clarify the architecture's approach to https to an earlier section, the security considerations section for a system like this (outside of the design of having CDN-like key sharing to support https downstream) probably also should either reference something like RFC 6707's security considerations that gives an overview of CDN security considerations or copy some of its contents that would apply to this system.

But more importantly, it should probably say more about the considerations NOT covered by that doc.  For example, it should cover the requirement to control all the forwarding nodes in the bier domain to ensure no attacker-controlled on-path forwarders have an opportunity to change the packets (or mention the recommended methods to prevent changing of the packets by on-path forwarders).
[DOT] agree on your points above on item 6, plain as that. Thanks for pointing them out!

Nits:
2nd paragraph on 5.2 doesn't end with a period, and I'm not sure the sentence was completed.

Dear Mboned WG

I would like to invite WG members to review, and comment to [email-masked] subject draft. It is very much about aspects of using BIER that fall into the scope of what i think should be very much of interest to Mboned member.

Thank you so much