Subject: [tsvwg] Multicast suggestion for draft-ietf-tsvwg-le-phb-04

From: Toerless Eckert <[email-masked]>
Date: Fri, 6 Apr 2018 04:08:54 +0200

a) Would ike to bring up a multicast specific issue with LE and
multicast: 

Since i was first exposed to fountain type codecs that can
surive arbitrary loss and its benefit in conjunction with
multicast transmission, i thought that LE traffic classes
would be an ideal use case for IP multicast in conjunction
with these codecs and LE. You just send this stream, it only gets
replicated when there is available bandwidth not used by best
effort traffic, jujust wait long enough until you have enough
packets for your decoder to reconstruct your original data. Great!
Kinda 2004'ish... thinking (first digital fountain products or the like 
around that time).

Alas, what this nice but too simple thought about LE ignores
with IP multicast is the problem that queue drops can only
happen after replication inside a network device, so any
non-congestion controlled multicast LE traffic does not only
take away unnecessary bandwidth on the incoming interface (like unicast),
but it does constitute an intentional or unintentional attack
vector against a resource that in most network platforms
can not even be managed at all, but is just shared at box/ASIC/NPU
level: replication performance. With unicast, there is
often some impicit or explicit fair allocation of NPU resources
vs. input interfaces, but when you start having some multicast
flows with a lot of outgoing interface receivers (lots of
replications, and most of them discarded), you can quickly
run into starvation of other multicast or even unicast flows
because you started trying to replicate and forward million
more packets thinking that they woudn't cost resources because
they'd get dropped inernally anyhow being LE. Especially when
the multicast replication performance is not separate from inicast,
but you just got a single resource eg: NPU whose performance
is simply measured in total number of egres packets it supports,
you can get quite surprised how your aggregate egress packet rate
goes down the drain because the poor NPU spends most of its time
replicating and dropping LE packets to slower congested interfaces.

With BE traffic you would not even think of designing multicast
solutions, where the aggregate amount of egres traffic was larger
than what the outgoing interfaces support. With LE traffic and
network/found codecs you can esily run into the misconception that
such a setup would work perfectly well, because you are not aware
that replication and large amount of dropping does wase
precious resources.

Aka: I would highly recommend to make a statement that 
multicast LE traffic (outside controlled networks) MUST
be congestion controlled using the options summarizzed in 
RFC8085, 4.1. - and add explanatory text about the reasons
(as above).

If this is something you think is useful, let me know, i could
help rewrite to fit the draft.

b) I really had a hard time finding this draft because i am
always looking for "scavenger" is there any way to smuggle that
word into the draft (some side remark) so that future google searches
will find it ?

I don't think i am the only one that remembers this traffic
class by that name, even if the official IETF term is different (LE).

Cheers
    Toerless

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 6 Apr 2018 08:54:23 +0200 (CEST)

> RFC8085, 4.1. - and add explanatory text about the reasons

Agreed.

> class by that name, even if the official IETF term is different (LE).

Yes, there should have a section that just states "in other documents LE 
is referred to as 'Less than best effort (LBE), scavenger class..." (add 
text for other things it might have been called over time).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 06 Apr 2018 08:03:53 +0100

>> RFC8085, 4.1. - and add explanatory text about the reasons
I think that this would be good to add.
> Yes, there should have a section that just states "in other documents 
> LE is referred to as 'Less than best effort (LBE), scavenger class..." 
> (add text for other things it might have been called over time).
>
Me too, I think it is important that this is found.

Gorry

---

From: Roland Bless <[email-masked]>
Date: Fri, 6 Apr 2018 17:39:58 +0200

Hi Toerless,

see inline comments.

> Kinda 2004'ish... thinking (first digital fountain products or the like 

Yes, interesting use case...

> replicating and dropping LE packets to slower congested interfaces.

Interestingly, the LE idea was motivated by a DiffServ multicast
problem, see RFC3754.
I think I understand the problem, so I try to rephrase
in my own words:
Packet replication happens before the network node can know
that there is congestion at a particular output
queue, so the replication to that output queue was
not necessary. In my point of view this is an interesting
implementation-related optimization problem, but it is not really
specific for the LE PHB. The same problem exists for BE traffic,
as also some output interfaces may be congested, whereas others
would not.
At the time when we wrote RFC3754, there was a comment that on
some router architectures it is possible to directly duplicate
a multicast packet on the backplane of the router, so that all
outgoing interfaces read the packet in parallel. In this case
there may not be such a replication performance impact.

> that replication and large amount of dropping does wase

I'm not sure about this one...see next comment.

> RFC8085, 4.1. - and add explanatory text about the reasons

I don't see how congestion control would help to
avoid the internal resource waste by unnecessary
replication. A congestion signal by packet loss
or ECN mark would only affect a particular multicast
branch and also receiver-based congestion control
would not change the situation.

> class by that name, even if the official IETF term is different (LE).

Point taken, but let me explain a little bit:
The QBone Scavenger Service (QBSS) Definition was published
March 16, 2001, whereas our "Lower than best effort" idea was first
published in September 1999 as draft-bless-diffserv-lbe-phb-00:
http://www.ietf.org/mail-archive/web-old/ietf-announce-old/current/msg05168.html
or
https://tools.ietf.org/html/draft-bless-diffserv-lbe-phb-00

>From this time on, we developed it further within the DiffServ WG.
So strictly speaking, the IETF term has been changed from ""Lower than
best effort" to "Lower Effort"
(https://tools.ietf.org/html/draft-bless-diffserv-pdb-le-00).

I think that the QBSS definition was an Internet2 implementation
of our LBE PHB, but the QBSS definition never mentioned that origin.
So most people think that the scavenger service definition was the
original idea, but indeed our lower than best-effort was published
one and a half years before the QBSS specification.
So I'll try to provide that linkage to scavenger service for those
who are only familiar with the Internet2 side of the story :-)

Best,
 Roland

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 6 Apr 2018 20:00:13 +0200

> as also some output interfaces may be congested, whereas others

On a typical network device, the agggregate ingres bandwidth
is the same as the aggregate egres bandwidth (asymmetric, eg: DSL
is a longer discus). So the performance requirements raised against
forwarding is roughly against being able to fill all interface
on input and output. And then somewhat more performance as icing
on the cake.

So when you would start to use badly designed LE, eg: fountain
codec over UDP unicast, the penalty on a network device is not
worse than inelastic BE unicast. Its severely limited in its
"badness" by the need to get all those bad packets into the device
before they can unnecessarily be dropped: You can never make
the box drop more packets than you receive in unicast. 

Likewise, badly designed LE multicast can be a lot worse because
it can be as bad as badly designed inelastic BE multicast. Except
that nobody understands the impact of real bad inelastic multicast,
because all the deployments of multicast go through a
lot of trouble to provide guaranteed bandwidth for inelastic
multicast traffic and protect it against attacks, such as in IPTV
and maket data networks. And in theory, there is elastic multicast
as required by in RFC8085, but only few companies know how to
do it and have working code for this (*sigh*).

So the risk is really high that LE with multicast opens a door
of badly designed multicast solutions ignoring the realities
of network devices because they do not understand how they
operate and assume that a lot of replication and dropping
is not a big problem - when in fact it is.

> there may not be such a replication performance impact.

Yes, the are buses and multicast capable fabrics and other
ways how unnecessary replications vary in their impact. But
they can equally make the problem bigger: Imagine 8 port
GigE swich. 4 input ports, each sources 1 Gbps multicast into
one group, 4 ports receivers, all joining all 4 groups. Ports
connected via eg: 8 Gbps internal Bus. So each of the
4 egres port receives 4 Gbps and has to drop them. Aka: The
bus can make such a platform perform worse for these type
of problems than other designs.

In all platforms i have seen over 20 years, the aggregated
replicated "virtual" numbrer of egres packets (before drop)
needs to be at most <= 110% of the aggregate number of
physcial output packets of the box, or you get big problems
(creating victim packets). And when LE systems are designed
to operate under higher loss than BE, you can very easily
run into those situations in well loaded networks.

Of course, it would be a lot of fun to design network devices
for which this is not an issue, and its certainly possible,
but i could never persuade a HW designer to do all the tricky
HW to do this. And besides: Without a lot of clear use case
business value for LE with a lot of loss, nobody would invest
for this. 

> branch

Not correct.

> and also receiver-based congestion control

Not correct.

RFC8085 4.1, first bullet:

feedback based is you "congestion signal". It
throttles the sender until no receiver has significant
loss. Effective against waste of replication, but really
just useful for no-congestion (randomn) loss or
limited congestion scenario (well defined reliable
lower bound availabel bandwidth). Otherwise you have
to kick off bad receviers from the tree. The RFC8085
text is a bit to wiggly IMHO. Anyhow: Totally useless
for LE.

RFC8085 4.1, second bullet:

This is the only useful CC option for multicast LE
unless you run in controlled networks where you can
clearly manage the replication-overhead on every
replication device. Receivers will drop themselves
off the tree when they receive loss and join a tree
with a lower bitrate. Aka: Sender needs to offer
content at a variety of bitrates and receiver can
join to a subset of those groups/channels matching
the bitrate it can receive without significant loss.
With streaming data / fountain codecs this is a
great, piece-of-cake design. With video its somewhat
complex SVC designs, but anything LE is not realtime
IMHO, so fountain codecs plus this receiver based
CC are IMHO a great option for LE. With the
only limitation being that you can not go overboard with
the amount of loss acceptable, but really keep it below IMHO
1..3% (based on above discussion of replication overhead).
If you experience higher loss, you need to downspeed
(join different group). 

This is pretty much inline with typical multicast
designs that do this for SVC, so the main sthing to
make sure is that we do not let LE today think it
can live with higher loss than acceptable for BE
receiver rate multicast CC. Funnily enough, that
rule would be a good rule just in case LE gets into BE
queue anyhow, but it also helps against the
multicast replication issue.

As said, let me know if you want me to help out on text
around this...

Oh, and btw: If we would use BIER end-to-end we
could do a lot more with LE. Might be worth another
sentence ;-)

> https://tools.ietf.org/html/draft-bless-diffserv-lbe-phb-00

Hah, there we go, copying that type of history into the draft
will get the needed word "scavenger" back into the draft ;-)

> who are only familiar with the Internet2 side of the story :-)

History is always written by the winner, so as long as you
write this draft, and think you can defend the statements
against IESG challenges, i would be very happy to see the
above reformulated into a history section on the RFC. 

Just see my other CC/bleaching text and the last section where i
think it becomes clear why the word scavenger is really a lot
more natural to me. Ideally i i would like to see a title like
this:

"A Lower Effort Per-Hop Behavior (LE PHB) and DSCP for Scavenger Traffic"

Aka: LE describes what the network does (less), scavenger
nicely describes what the traffic needs to do. Quite
complementary. And with all those words (LE, PHB, scavenger,
DSCP) in the title, everybody googling for this stuff will find it.

(many operators know DSCP, but have no idea what PHB means, thats why i would add it to the title).

-- 
---
[email-masked]