Subject: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

From: Toerless Eckert <[email-masked]>
Date: Fri, 6 Apr 2018 18:03:45 +0200

Some thought about CC and bleaching for LE (no multicast this time)

> LE traffic SHOULD be congestion controlled (i.e., use a congestion
> controlled transport or implement a congestion control method [RFC8085])

a) What would be the current summarized TSV IESG review requirement
against a CC statement for BE ? I thought it was something like

  "MUST be CC controlled unless known to be operating across a controlled
  network, and even then it SHOULD use a circuit breaker [RFC8084]".

Whatever the right text is - IMHO it would be great if we knew that text,
and then explained in the draft any difference in CC expectation for LE
vs. these BE rules. I for once do not understand why a simple "SHOULD" would
suffice. My starting point is that LE CC = BE CC, and then - see below.

b) 8085 has UDP in the title. It would be good if the CC text was
amended to make it clear the requirement applies whatever the transport
is. Not sure what the best magic text (or better reference) for non-UDP
based transports is.

c) I couldn't figure out from rfc6297 if all the observed LE protocol
(profiles) would actually automatically receive less share of bandwidth
if simply run together with "normal" TCP traffic in the same BE queue.

To me its somewhat logical that an LE transport protocol that knows
it will get LE treatment could and should actually be somewhat more
aggressive than a BE transport because it has to recover from more loss,
or if ECN is used it needs more aggressive upspeeding to best utilize 
the likely a lot more fluctuating ressources in LE. 

Even if so far the researched LE CC mechanisms have not explored this
"more aggressive" behavior, i think it would be great if we could
define the right option for it, to make LE even more useful with
future network profiles (see point f below). But of course the
challenge then is how to make sure such more aggressive LE would
never kill BE in the Internet:

d) The document only mentions what happens when LE traffic gets BE
treatment in passing:

  [about SHOULD do LE CC]:
  "but is also essential if LE traffic is mapped to the default PHB
   in DS domains that do not support LE"

I think wrt. to CC requirements against LE traffic, we should make
it front and center to define LE CC behavior in comparison to
BE CC behavior - else we will get all type of unpredictable results
when folks start to use various LE protocols:

I) LE CC must ensure that if operating across a congested hop that
is treating LE as BE, LE CC MUST NOT give LE traffic more bandwidth share
than the competing BE traffic, and it SHOULD give it less share.

So i hope thats the uncontentuous part. Here is the interesting part:

II) LE CC MAY be more aggressive than BE CC (give it more share than
BE traffic if LE traffic is treated as BE) only if known to operate
across a fully LE enabled path (every hop supporting LE). In this case, LE CC
MUST fall back to I) when it can suspect that the path is not fully LE capable.

But how could LE CC ever meet the "MUST fall back" expectation ?

e) I don't think LE CC can ever figure out the "MUST fall back" except
through bleaching (or other to be invented signaling).  But the
"do not bleach" rule of the current text is really the most useful rule
for the Internet to make introduction of LE into Internet nodes as
attractive as possible.

II-bleach) LE CC MAY be more aggressive than BE CC (give it more share than BE flows)
only if known to operate across a controlled network where the path
is expected to support LE on every hop. Paths meant to support
more aggressive LE CC SHOULD bleach LE to BE on hops that do not
support LE. LE CC MUST fall back to be equal or less aggrssive than
BE CC when receiving bleached packets (DSCP set to BE).

So, this allows more aggressive than BE, but only for controlled networks.

f) Here is an example where i think more aggressive LE will be quite
valuable: IMHO, we may see a lot more non-TCP friendly traffic
in their own queues, the more inelastic and short-term bursty
services start to appear on SP networks. I would think especially
in 5G/6G and metro networks, DetNet style traffic or really high
bitrate/bursty AR/VR traffic will want to have a lot of low-latency
trafic that is little, or not congestion controlled. And this can 
nicely be stuffed into separate priority queues and get what it wants
(Benefit of having a large SP network).

So this is then a very ragged base load on the links. Then i add
BE traffic, which CCs to mid to longer term available remaining
bandwidth. That leaves a lot of shorter term ragged cliffs left
over which can only be grabbed by more aggressive CC than BE
willing & able to deal with this highly variable leftover bandwidth.

Aka: Taking bandwidth unused in a pure BE network is simple and
works with less aggressive CC than BE-CC. Scavenging remaining bits
from more interesting "multi-service" network loads is the real scavenging
job requiring more aggressiveness.

Cheers
    Toerless

---

From: <[email-masked]>
Date: Mon, 9 Apr 2018 07:13:15 +0000

Hi Toerless,

the design of LE leading to the current draft allows for a domain
- to consume a DSCP which may pass a domain not supporting LE and this DSCP isn't bleached.
- to support LE internally without a requirement to negotiate LE support with other domains.

This works, if LE supports CC as default class traffic does. That would require the SHOULD related 
to CC in the current draft to be replaced by a MUST, if the LE traffic is sent across an 
interconnection link and there's no LE SLA in place.  I'd be rather unhappy to receive "aggressive" 
streaming traffic from a peer and be told that it is behaving correctly, 
cause it's marked LE and my domain could support an LE class to deal with it.

There's another option beside bleaching unknown DSCPs at domain boundaries, which is to drop or 
bandwidth limit traffic. Both options may be more viable, if the traffic which is limited or dropped 
is more aggressive than the kind of traffic which is covered by some SLA (like default class traffic 
assuming CC coupled with a best effort transport).

I don't favor solutions where a DSCP re-write is used for signaling. You expect an ECN like 
mechanism to be in place, where the receiver returns an information reporting a received DSCP to the 
sender. If you require an ECN like behavior, why not using ECN? In your case: combined with 
the LE DSCP. That may allow to signal "no support" for aggressive load expansion by ECN, rather 
than by DSCP rewrite.

My general view is: don't link determined end to end transport behavior to DSCP values bearing no 
end to end significance.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg [mailto:[email-masked]] Im Auftrag von Toerless Eckert
Gesendet: Freitag, 6. April 2018 18:04
An: [email-masked]
Cc: [email-masked]
Betreff: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> controlled transport or implement a congestion control method 

a) What would be the current summarized TSV IESG review requirement against a CC statement for BE ? I thought it was something like

Whatever the right text is - IMHO it would be great if we knew that text, and then explained in the draft any difference in CC expectation for LE vs. these BE rules. I for once do not understand why a simple "SHOULD" would suffice. My starting point is that LE CC = BE CC, and then - see below.

b) 8085 has UDP in the title. It would be good if the CC text was amended to make it clear the requirement applies whatever the transport is. Not sure what the best magic text (or better reference) for non-UDP based transports is.

c) I couldn't figure out from rfc6297 if all the observed LE protocol
(profiles) would actually automatically receive less share of bandwidth if simply run together with "normal" TCP traffic in the same BE queue.

To me its somewhat logical that an LE transport protocol that knows it will get LE treatment could and should actually be somewhat more aggressive than a BE transport because it has to recover from more loss, or if ECN is used it needs more aggressive upspeeding to best utilize the likely a lot more fluctuating ressources in LE. 

Even if so far the researched LE CC mechanisms have not explored this "more aggressive" behavior, i think it would be great if we could define the right option for it, to make LE even more useful with future network profiles (see point f below). But of course the challenge then is how to make sure such more aggressive LE would never kill BE in the Internet:

d) The document only mentions what happens when LE traffic gets BE treatment in passing:

  [about SHOULD do LE CC]:

I think wrt. to CC requirements against LE traffic, we should make it front and center to define LE CC behavior in comparison to BE CC behavior - else we will get all type of unpredictable results when folks start to use various LE protocols:

I) LE CC must ensure that if operating across a congested hop that is treating LE as BE, LE CC MUST NOT give LE traffic more bandwidth share than the competing BE traffic, and it SHOULD give it less share.

II) LE CC MAY be more aggressive than BE CC (give it more share than BE traffic if LE traffic is treated as BE) only if known to operate across a fully LE enabled path (every hop supporting LE). In this case, LE CC MUST fall back to I) when it can suspect that the path is not fully LE capable.

e) I don't think LE CC can ever figure out the "MUST fall back" except through bleaching (or other to be invented signaling).  But the "do not bleach" rule of the current text is really the most useful rule for the Internet to make introduction of LE into Internet nodes as attractive as possible.

II-bleach) LE CC MAY be more aggressive than BE CC (give it more share than BE flows) only if known to operate across a controlled network where the path is expected to support LE on every hop. Paths meant to support more aggressive LE CC SHOULD bleach LE to BE on hops that do not support LE. LE CC MUST fall back to be equal or less aggrssive than BE CC when receiving bleached packets (DSCP set to BE).

f) Here is an example where i think more aggressive LE will be quite
valuable: IMHO, we may see a lot more non-TCP friendly traffic in their own queues, the more inelastic and short-term bursty services start to appear on SP networks. I would think especially in 5G/6G and metro networks, DetNet style traffic or really high bitrate/bursty AR/VR traffic will want to have a lot of low-latency trafic that is little, or not congestion controlled. And this can nicely be stuffed into separate priority queues and get what it wants (Benefit of having a large SP network).

So this is then a very ragged base load on the links. Then i add BE traffic, which CCs to mid to longer term available remaining bandwidth. That leaves a lot of shorter term ragged cliffs left over which can only be grabbed by more aggressive CC than BE willing & able to deal with this highly variable leftover bandwidth.

Aka: Taking bandwidth unused in a pure BE network is simple and works with less aggressive CC than BE-CC. Scavenging remaining bits from more interesting "multi-service" network loads is the real scavenging job requiring more aggressiveness.

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 Apr 2018 08:52:10 +1200

On 09/04/2018 19:13, [email-masked] wrote:
...
> My general view is: don't link determined end to end transport behavior to DSCP values bearing no 

+many. The fact that DSCP meanings are domain-specific is so fundamental
that coupling them to e2e transport behaviour is condemned to failure.
(Which we spent a lot of time explaining to the WebRTC folk.) Very different
from ECN.

    Brian

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 10 Apr 2018 09:44:17 +0200

> - to support LE internally without a requirement to negotiate LE support with other domains.

Agreed.

> interconnection link and there's no LE SLA in place.

Agreed. Thats why i was proposing text to indicate how CC for LE traffic must have
it get equal or less share when competing with BE traffic. Aka: "any CC" is not
sufficient, we wat it to be BE LE "compatible".

> I'd be rather unhappy to receive "aggressive" 

Right. which is why in the first place, it would be good to distinguish
between "BE compatible" and "not BE compatible" LE CC. 

> assuming CC coupled with a best effort transport).

Right.

Is there any explicit policy option that could be recommended ?
Traditionally the solutions are trying to throttle traffic on a per-flow
basis, but thats quite expensive. Not sure if any of the neweer AQM schemes could
result in the desired effect.

> sender.

Yes.

> If you require an ECN like behavior, why not using ECN? In your case: combined with 

Nice idea.

I guess this gets us to a point where we would need more actual testing/simulation
to figure out if LE CC would really need to be BE incompatible to be able to
scavenge all the bits that normal BE and BE compatible LE traffic could not
utilize.

Aka: I would like to use this proposal of yours if i knew whether it would work
or not. But i don't. And if no other prior work gives better insight, its
probbly best not to make any normative statements about how to deal with
"non BE compatible LE CC", but at best rather have informational text disussing
that option as open and to be explored.

> My general view is: don't link determined end to end transport behavior to DSCP values bearing no 

See reply to Brian.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: <[email-masked]>
Date: Tue, 10 Apr 2018 08:00:16 +0000

Hi Toerless

thanks, my comments are marked [RG].

Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Toerless Eckert [mailto:[email-masked]] 
Gesendet: Dienstag, 10. April 2018 09:44
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

On Mon, Apr 09, 2018 at 07:13:15AM +0000, [email-masked] wrote:
> streaming traffic from a peer and be told that it is behaving 
> correctly, cause it's marked LE and my domain could support an LE class to deal with it.

Right. which is why in the first place, it would be good to distinguish between "BE compatible" and "not BE compatible" LE CC. 

[RG] This may be a good idea.

> There's another option beside bleaching unknown DSCPs at domain 
> boundaries, which is to drop or bandwidth limit traffic. Both options 
> may be more viable, if the traffic which is limited or dropped is more 
> aggressive than the kind of traffic which is covered by some SLA (like default class traffic assuming CC coupled with a best effort transport).

Right.

Is there any explicit policy option that could be recommended ?
Traditionally the solutions are trying to throttle traffic on a per-flow basis, but thats quite expensive. Not sure if any of the neweer AQM schemes could result in the desired effect.

[RG] AQM would have to be deployed in the complete domain to 
limit the CC-unfriendly traffic at the point of congestion. That 
means to support LE within the domain. If a provider isn't interested 
in that, the decision might be to limit incoming traffic.

> If you require an ECN like behavior, why not using ECN? In your case: 
> combined with the LE DSCP. That may allow to signal "no support" for 
> aggressive load expansion by ECN, rather than by DSCP rewrite.

Nice idea.

I guess this gets us to a point where we would need more actual testing/simulation to figure out if LE CC would really need to be BE incompatible to be able to scavenge all the bits that normal BE and BE compatible LE traffic could not utilize.

Aka: I would like to use this proposal of yours if i knew whether it would work or not. But i don't. And if no other prior work gives better insight, its probbly best not to make any normative statements about how to deal with "non BE compatible LE CC", but at best rather have informational text disussing that option as open and to be explored.

[RG] I favor "open and to be explored". For the time being, I have no experience in operating LE (and co-existence with BE).

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 10 Apr 2018 11:00:33 +0200

> that coupling them to e2e transport behaviour is condemned to failure.

I don't  know how to read Ruediger and your statements. How is an LE
sender meant to indicate that the traffic is LE if not via an end-to-end DSCP
indicating LE ? 

> (Which we spent a lot of time explaining to the WebRTC folk.)

draft-ietf-tsvwg-rtcweb-qos ? Too bad it doesn't have changelogs. WOuld love
to know what was changed in it because of that feedback. No mentioning
of interdomain so not sure how this draft is relevant here...

Cheers
    Toerless

Cheers
    Toerless

---

From: <[email-masked]>
Date: Tue, 10 Apr 2018 09:17:59 +0000

Hi Toerless,

you wrote: "How is an LE sender meant to indicate that the traffic is LE 
if not via an end-to-end DSCP indicating LE ?"

There is no end-to-end DSCP indicating the same PHB unless all providers, 
departments, platform owners (and so on) along the end-to-end Transport 
chain agreed on the value. 
My take of your LE discussion was that the sending party doesn't care, whether 
there's a Diffserv agreement in place with a receiving domain. In that case, 
it is a no go to link any service or transport specification to the DSCP value. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Toerless Eckert [mailto:[email-masked]] 
Gesendet: Dienstag, 10. April 2018 11:01
An: Brian E Carpenter <[email-masked]>
Cc: Geib, Rüdiger <[email-masked]>; [email-masked]; [email-masked]
Betreff: Re: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

On Tue, Apr 10, 2018 at 08:52:10AM +1200, Brian E Carpenter wrote:
> > My general view is: don't link determined end to end transport 
> > behavior to DSCP values bearing no end to end significance.
> 
> +many. The fact that DSCP meanings are domain-specific is so 
> +fundamental

I don't  know how to read Ruediger and your statements. How is an LE sender meant to indicate that the traffic is LE if not via an end-to-end DSCP indicating LE ? 

> (Which we spent a lot of time explaining to the WebRTC folk.)

draft-ietf-tsvwg-rtcweb-qos ? Too bad it doesn't have changelogs. WOuld love to know what was changed in it because of that feedback. No mentioning of interdomain so not sure how this draft is relevant here...

Cheers
    Toerless

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 10 Apr 2018 13:00:37 +0200

> chain agreed on the value. 

Sure. 
a) thats why we write these drafts/RFC to establish such DSCP standards.
b) If there is no end-to-end DSCP, there is also no other way to
   indicate end-to-end PHB.

> it is a no go to link any service or transport specification to the DSCP value. 

No, its not about parties "not caring" about DiffServ agreement. Its just
that any new diffserv semantic we want to bring into "the Internet" needs to
have a good story-line how it can be be backward compatible with BE in case there is no
end-to-end agreement, and with LE we can IMHO easily do this. I don't think
"good fallback" is the same as "don't care".

E.g.: for me the logic is quite simple: Applications indicate the end-to-end
desired PHB/DB via a DSCP, the path/domains try to figure out if/how that
desire can be met. If such an app is deployed on the Internet, it also
needs to have some fallback to BE - Circuit breakers or BE compatible CC.

Cheers
   Toerless

-- 
---
[email-masked]

---

From: <[email-masked]>
Date: Tue, 10 Apr 2018 11:51:36 +0000

Hi Toerless

Some more points marked [RG].

Regards, Ruediger

-----Ursprüngliche Nachricht-----

> There is no end-to-end DSCP indicating the same PHB unless all 
> providers, departments, platform owners (and so on) along the 
> end-to-end Transport chain agreed on the value.

Sure. 

[RG] The DSCP may change if RFC 8100 or the like isn't applied. The idea is to 
get an end-to-end PHB.

> My take of your LE discussion was that the sending party doesn't care, 
> whether there's a Diffserv agreement in place with a receiving domain. 
> In that case, it is a no go to link any service or transport specification to the DSCP value.

No, its not about parties "not caring" about DiffServ agreement. Its just that any new diffserv semantic we want to bring into "the Internet" needs to have a good story-line how it can be be backward compatible with BE in case there is no end-to-end agreement, and with LE we can IMHO easily do this. I don't think "good fallback" is the same as "don't care".

E.g.: for me the logic is quite simple: Applications indicate the end-to-end desired PHB/DB via a DSCP, the path/domains try to figure out if/how that desire can be met. If such an app is deployed on the Internet, it also needs to have some fallback to BE - Circuit breakers or BE compatible CC.

[RG] Compatibility with default transport and Best Effort service is useful and alright in that case. To me no agreement on a Diffserv transport service means default transport (& bleaching) of traffic received at interconnection and I think this interpretation is backed by Diffserv standardization.

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 10 Apr 2018 14:59:17 +0200

Hi Toerless,

[sorry for the delay, too busy with other stuff and that
mail required some deeper reading effort and thoughts...].
See comments inline.

Am 06.04.2018 um 18:03 schrieb Toerless Eckert:
>   network, and even then it SHOULD use a circuit breaker [RFC8084]".> Whatever the right text is - IMHO it would be great if we knew that text,

Hmm, I also agree that LE CC at least covers BE CC requirements.
However, right now I think that some traffic may actually not be
CCed, such as short UDP transactions, one request, one response
(DNS is a typical example). This is then not a UDP flow and maybe not
really eligible for the LE class, however, I can imagine some uncritical
telemetry data such as infrequently polling of temperature sensors
to use LE. That's why I used a SHOULD. However, I already have a
revision of text on CC requirements on my TODO list for the next
I-D revision anyway. So the above text uses "LE traffic" that also
includes packets that cannot be strictly denoted as "flow".

BTW, RFC8085 also says SHOULD:
   It is important to stress that an
   application SHOULD perform congestion control over all UDP traffic it
   sends to a destination, independently from how it generates this
   traffic.

> is. Not sure what the best magic text (or better reference) for non-UDP

I deliberately did not put UDP into the text, but I probably can
clarify a bit that RFC 8085 describes UDP-based applications, but
that this may apply to other non-CCed transports as well.

> the likely a lot more fluctuating ressources in LE. 

Probably, but higher aggressiveness could also impact fairness and
stability within the LE class, but my main concern is that LE may
get elevated to BE by domains that don't support LE. In this case
BE flows may loose their "fair share" in presence of more aggressive
LE flows.

> never kill BE in the Internet:

Yep.

> when folks start to use various LE protocols:

It think that this leads too far. With respect to RFCs2474/2475
(DiffServ PHB and Architecture) a PHB definition
describes/determines only the packet forwarding behavior within a single
node (cf. Sec.3 of RFC2475 https://tools.ietf.org/html/rfc2475#section-3).
The single PHB is agnostic what is happening inside its DS class,  since
a DS node only operates on the whole behavior aggregate. So I
think it is reasonable to shortly discuss LE PHB CC requirements, but it
seems that we otherwise need a separate document for discussing all
the issues...

> So i hope thats the uncontentuous part. Here is the interesting part:

Nope, since it is probably not possible to fulfill this requirement:
if there is less BE traffic present (e.g., inelastic or all application
limited) at the bottleneck, say 20% of link capacity, and the rest is
used by LE traffic (i.e., 80%), so the MUST NOT is not sensible in this
case (would mean that LE traffic is throttled to 20% too). This is
exactly, what I don't want. LE should be able to use up all otherwise
unused resources.

> MUST fall back to I) when it can suspect that the path is not fully LE capable.

I think that this cannot be reliably detected by the end-systems only.
A separate codepoint could help that indicates LE elevation.

> for the Internet to make introduction of LE into Internet nodes as

Yep. As I said, a potential solution could be to use another DSCP
that indicates that some hop treated LE as BE, however, this needs
to be reflected to the sender somehow, then.

> job requiring more aggressiveness.

Thanks for discussing this, but I think that we should try to avoid
additional complexity.
There are many use different cases for LE and various scenarios where
LE traffic may share bottlenecks with other traffic etc. Putting
too much or too strict requirements onto LE CC could also lead to
ossification of CC algorithms, e.g., congestion control of
current Lower-than-Best-Effort transport protocols assume the presence
of current prevalent loss-based CC. If that changes (e.g., to
delay-based schemes) LBE CC may be more aggressive than BE CC
at some point in the future.

Cheers,
 Roland

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 10 Apr 2018 17:28:39 +0200

Hi Toerless,

see inline.

> sufficient, we wat it to be BE LE "compatible".

Besides the problems related to the LE/BE share I pointed out in an
earlier mail, this may have strong dependence on the behavior of current
prevalent congestion controls.

> between "BE compatible" and "not BE compatible" LE CC. 

While I think that I have a rough idea of what you mean by these terms,
I think it is difficult to define what this actually means - especially
without the danger of ossification of current CCs for BE.

> scavenge all the bits that normal BE and BE compatible LE traffic could not

That LE CC would then depend on the behavior of the dominating BE CCs.
I think that we should think of and allow also for CC evolution, e.g.
LEDBAT would work with currently dominating loss-based CCs, but probably
not with delay-based CCs.

Regards,
 Roland

-- 
PD Dr. Roland Bless -- WWW: http://tm.kit.edu/~bless
Institut für Telematik, Karlsruher Institut für Technologie (KIT)
Postf. 6980, 76049 Karlsruhe -- Zirkel 2, Geb. 20.20, 3.OG, Raum 358
Tel.: +49-721-608-46413 Mob.: +49-152-0160-1400 Fax: +49-721-608-46789

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Apr 2018 11:16:09 +1200

> sender meant to indicate that the traffic is LE if not via an end-to-end DSCP

DSCPs are by definition source-to-edge, so the strict answer to your
question is "they can't". All DSCPs are in principle rewritten at each
admin boundary. 
(Why? Because the ISPs active in the diffserv WG insisted on this.)
(Don't like this? Step 1, invent a time machine and set it to 1998.)
If and only if there is an SLA between adjacent networks, then
DSCPs mentioned in the SLA could be copied over.

If there was a BCP about diffserv mapping across domain boundaries,
that might amount to a sort of universal voluntary SLA. But RFC8100
is Informational. Maybe it will prevail.
 
> draft-ietf-tsvwg-rtcweb-qos ? 

No, RFC7657 (from the DART WG).

> Too bad it doesn't have changelogs. WOuld love

The real question is whether rtcweb deployment encourages deployment
of RFC8100 style mappings. I'm pessimistic.

Yes, this strictly irrelevant to LE, but the underlying question is
the same I think - whatever we put in our RFCs, what is the incentive
for carriers to deploy LE support, or RFC8100 support in general?

    Brian

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 03:25:44 +0200

On Tue, Apr 10, 2018 at 02:59:17PM +0200, Bless, Roland (TM) wrote:
> (DNS is a typical example).

Right. So the text i was proposing somehwere else on the thread
stating that LE traffic MUST also be good BE citizen (not get larger
share than BE traffic if forwarded via BE PHB) is probably an easier
way to describe the expectations against LE CC mechanisms - it avoid
having to specify what really our (IETF) BE CC expectations are by just
referring to them.

> This is then not a UDP flow and maybe not

Well, by just referring to BE LE as suggested about this would
be resolve: If you're allowed to not use CC for stuff like
DNS in BE, then you're therefore also allowed to do it in LE 
(this does not imply whether or not it would make sense in LE)

Referring to BE LE is somewhat lame, it would be nicer if we had a
comprehensive BE CC requirements spec we could point to, but i have
to admit i don't know which doc does include this right now. On
the other hand, CC expectations for BE may evolve over time,
and referring to them the way suggested above makes LE automatically
adjust as well to maybe changing BE CC requirements.

Wrt to UDP/Telemetry: I think it would make LE a lot more attractive
if we had some positive examples how to use it for real relevant
use-cases. There is a list of nice use cases by name, but i fear
that most of them can't really live with a completely
unpredictable, opportunistic traffic class like LE alone.

I was think that most applications for LE will require at least
some form of deadline: require a deadline. Movie/software data files
could be made available 48 hours before release, but only at
release would the key for decrypt be distributed. Periodic actions
like backup, where deadline is the period, etc., pp.

The generic solution to this is to use BE/LE in a hybrid fashion.
Calculate "min rate" to finish transfer by the deadline. 
Transfer via LE, but whenever rate falls below min rate,
transfer remaining rate via BE.

IMHO, One could use MP-TCP for this, one LE, one BE flow.
But i would only include text about MP-TCP after approval of
this disussing on MPTCP mailing list.

Telemetry could work similarily. If you need some minimm reliable
rate, you send that with BE and add higher rate with LE. But
whether or not either BE or LE rate is permissible if no further
CC is applied is another discuss.

>    sends to a destination, independently from how it generates this

Not quite sure about the history on the MUST/SHOILD part of the
doc. The explanatory details are pretty good though ;-)

> that this may apply to other non-CCed transports as well.

Something like that, yes.

> BE flows may loose their "fair share" in presence of more aggressive

Lets take the discussion about the current aggressiveness of the various
LE CC mechanisms known to ICCRG mailing list first, if there is no
clear better knowledge, i'd at best suggest that that option is for further
research. Which we may want to note in the doc.

> seems that we otherwise need a separate document for discussing all

Right. I'd like to continue this discuss, especially with Ruediger
being knowledgable about this, but at this point in the discussion
i wouldn't continue to suggest the above text (d).

> exactly, what I don't want. LE should be able to use up all otherwise

I think I) is right AND you are right, so either i need to
propose a better wording to make I) better readible, or you need to
read it again ;-)

I) only describes the necessary LE CC behavior if the LE
traffic was ( unknowing to the LE CC mechanism)  treated as BE,
not if it is really treated separately from BE as LE.

In your case, all real BE traffic occupies just 20% of the link, so
its not really a congestion point at all. Additional BE traffic on that
link that wanted arbitrary bandwidth could therefore get 80%, therefore
according to my requirement, so could the LE traffic, so your 
example result is met. 

More interestingly, lets say you've got 10 BE flows that want
arbitrary bandwidth, on a 10 Mbps link each gets 1 Mbps. We add
10 LE CC flows and also treat them in the BE queue (without them
knowing of course). The result is that each of the LE CC flows
should now get at most 0.5 Mbps, because now we have 20 flows
competing on the link, and if those where all BE CC then each one
would get 0.5 Mbps each.

This is really the core example, because we primarily care about
LE CC to not be unfair to BE traffic IF the LE traffic is treated as 
BE (somewhere along the path).

Now, in reality, these 10 LE flows are put into an LE queue with
0% bandwidth share, so unfortunately, they will not get anything,
because those 10 BE flows get all the 100% bandwidth. Or maybe the
operator actually gives LE some 5% weighted fair queue to LE on
the link, in which case all LE flows would only get those 5%.
Hoever many LE flows there are. On a link with an actually
congested BE queue.

I do think that there is of course no way for an LE CC
flow to know that it is NOT competing with BE CC traffic unless
we add anything new to the mix (which we don't), so the LE CC
flows will always have to compete amongst each other as if they
compete also with BE flows. Therefore no need to explicitly make
a requirement specifically about the case where LE CC flows are
NOT in the same queue with BE CC.

lastly: I think at least some of the existing LE CC mechanisms
are actually meant to compete with BE CC and then actually get 
_less_ share of the BE queue than competing BE CC traffic. I
think they would all get the actually free 80% in your example,
but in my congested example of 10 BE plus 10 LE flows in the
BE queue, i think they would get less than 5% of bandwidth
each. Which is fine. Which might be something we could write
up as a requirement, but i am not sure that we should. Primarily
because i think it would be a very difficult target to describe.

> A separate codepoint could help that indicates LE elevation.

See above - lets consider point II) canned right now and
let me try to figure out some more input from ICCRG and Ruediger
before i have a new opinion about "more aggressive than BE CC" options.

Cheers
   Toerless

-- 
---
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Apr 2018 13:54:14 +1200

...
 
>>> So i hope thats the uncontentuous part. Here is the interesting part:

Why is this needed when the text already says:

   A packet forwarded with the LE PHB SHOULD have lower precedence than
   packets forwarded with the default PHB, i.e., in the case of
   congestion, LE marked traffic SHOULD be dropped prior to dropping any
   default PHB traffic.  Ideally, LE packets SHOULD be forwarded only if
   no packet with any other PHB is awaiting transmission.

BE and LE PHBs should talk about queueing and dropping behaviour, not about
capacity share, in any case. It's clear that on a congested link, LE
is sacrificed first - peak hour LE throughput might be precisely zero,
which is not acceptable for BE.

     Brian

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 04:33:05 +0200

> (Don't like this? Step 1, invent a time machine and set it to 1998.)

Yes, without having read your email in before, i was watching that 
time machine episode of Big Bang Theory today on the plane, and that
thought crossed my mind.

But $800 is a lot of money to fix the DiffServ architecture. (hint: joke)

I just had a lot of bad experience with this diffserv architecture.
Primarily loss of information of N:1 DSCP remarkings (and not the good ones
like AF N+1 remarkings). We tried to figure out how a poor enterprise
getting back remarked packets from its L3VPN service could still figure
out what those maybe only 4 DSCP should mean for the following 12 class
DSCP queuing system in the enterprise. Luckily more and more enterprises
had to encap their traffic on CE anyhow, e.g.: for security purpose,
not trusting the SP with their plaintext, so for those customers, the
problems of SP remarking went out of the window.

Ideally IMHO, you really want two DSCP fields to support SPs desires to
provide arbitrary DSCP behavior/remarkings. An end-to-end DSCP and
in another field (encap ?) the DSCP for the next domain.

> is Informational. Maybe it will prevail.

Yes, its a good step in the right direction. Note that rfc2475 is
informational too, only rfc2474 is normative. Leads to intereesting
results i think.

rfc8100 is written with an SP (and interdomain SP) perspective in
mind. It hink it would be great to write a document with the
applications in mind.

Aka: describe an architecture where DSCP
is end-to-end using standard PHB default DSCP. Describe
requirements that SP networks would need to meet to make this
work. Describe how intelligent apps could read received DSCP and
intelligently react to e.g.: N+1 downgraded AF codepoints.
Aka: there is good remarkings in networks, namely the one default
in the standard PHB RFCs using default codepoints. 

This would not contradict rfc2474, it just defines a stricter
profile of rfc2475 (requiring rfc8100 behavior and maybe more).

Even just the expectation for applications to be configurable to
send non-default PHB DSCPs that we have today is already hilarious.
Without even any IETF or other standards bodys definition of (abstract)
APIs to determine the DSCP semantics supported. 

> No, RFC7657 (from the DART WG).

Ah, educational documentation. Nice. Had forgotten about that one.

> of RFC8100 style mappings. I'm pessimistic.

Forget interdomain issues. The question is whether anyone can propose
a useful QoS setup for draft-ietf-tsvwg-rtcweb-qos even within
an enterprise leveraging the three priorities. If there is
a good recipe to operationalize this in an enterprise, we
can look at figuring out if/how to use the same recipe in 
L3VPN. I am quite pessimistic about the three priority model.

I think its not even clear what the desirable behavior under
contention across those priorities is. I think the goal was
just to provide nerd-knob heaven, aka: make RTCweb apps configurable
so one could start experimenting in enterprises with priorities
and DSCP and see what QoS setup might make sense. Which is
not a bad idea. Its just not a complete solution, so i would
not dare to recommend anything to SPs for this today.

> for carriers to deploy LE support, or RFC8100 support in general?

Indeed. As said at the end of one of my emails, a specific benefit
of "more aggressive LE CC" could be to even scavence more bits from
multiservice links with bursty high priority traffic than possible
today with BE CE. But thats researchy.

For the immediately applicable LE CC schemes, i think the
main benefit for SPs would be to reduce pressure during peak
consumption hours in their network by supporting application/
solutions to start using LE. Networks need to be sized for
busy hour and often get a lower and lower average utilization
because of increasing ratio between peak and and min utilization.
Aka: Slow down need for network growth and increase revenue
by smoothing out network loads via LE applications.

The other one is pricing differentiation. Especially when you
go mobile on the edge, you will have volume pricing. Volume
pricing free LE traffic class support could be the biggest 
incentive for LE applications to be developed and drive the more
cost-effective network utilization. Think of streaming netflix
to phone with LE (maybe hybrid LE/BE as explained in other email)

Mobile networks are quite complex today though, so there is
a lot more to analyze before you can make a complete
pitch how LE traffic is sufficiently free to the operator
provide all these benefits. 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 05:00:38 +0200

>    no packet with any other PHB is awaiting transmission.

I) is meant as a requirement against LE CC. If formally a traffic classes
CC requirements are not part of the PHB, thats fine. If we need a second
document with a one paragraph LE CC requirement becasue we should not
specificy CC requirements and PHB together in one document that would be silly,
but IMHO then we need such a second 2 page document. Or where else would
you define the CC requirements ?

Note that the biggest issue with the above text is that it
does not help any SP that ha no idea what LE is and therefore
his routers along the path will treat it as BE. Only BE compatible
LE CC will help them and allow to safely let LE traffic onto the Internet.

Beside that:
I havent really commented on that text segment in before, but i am a
bit worried that it could be interpreted as if LE could perfectly
be "supported" in a single queue with BE by just assigning it a different
drop profile, and from past experience trying to figure out the impact
of different drop profiles during contention, thats a black magic and
does not lead to operationally predictable outcomes.  But of course i
have been burned with this, so i am paranoid.

In fact i think the text is correct and would NOT permit BE and LE to
be in a single queue, but i would certainly prefer more directive text - 
aka: something saying LE should ideally use a separate lower priority
queue from any other traffic class queue, but that a weighted
fair share queue with a share of 0 is fine too. This can be informational,
but IMHO we should strive to make adoption easy and not a guesswork.

> capacity share, in any case.
>
> It's clear that on a congested link, LE

Right. which is another point i raised in another mail: If we
really make LE go to 0 for long periods, we really should have
informational text about hybrid BE/LE use, or we won't get
any adoption. Otherwise apps using only LE will be written,
and then they won't work well enough, and especially in enterprises,
network opeators might give LE some traffic share > 0, and then that
share number becomes yet another magic parameter you need to plan
and update based on your LE traffic profiles.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 12 Apr 2018 08:39:25 +0200 (CEST)

On Thu, 12 Apr 2018, Brian E Carpenter wrote:

> BE and LE PHBs should talk about queueing and dropping behaviour, not 
> about capacity share, in any case. It's clear that on a congested link, 
> LE is sacrificed first - peak hour LE throughput might be precisely 
> zero, which is not acceptable for BE.

I have received questions from operational people for configuration 
examples for how to handle LE/BE etc. So I did some work in our lab to 
give some kind of example.

So my first goal was to figure out something that'd do something 
reasonable on a platform that'll only do DSCP based RED (as this is 
typically available on platforms going back 15 years). This is not 
optimal, but at least it would be deployable on lots of platforms 
currently installed and moving packets for customers.

The test was performed with 30ms of RTT, 10 parallel TCP sessions per 
diffserv RED curve, 800 megabit/s access speed (it's really gig, but in my 
lab setup I have some contraints that meant if I set it to gig I might get 
some uncontrolled packet loss due to other equipment sitting on the same 
shared link, so I opted for 800 megabit/s as "close enough").

What I came up with that would give LE ~10% of access bandwidth compared 
to BE, and a slight advantage for anything that is not BE/LE (goal was to 
give this traffic a lossless experience) was this:

This is a Cisco ASR9k that without this RED configuration will buffer 
packets up to ~90 milliseconds, resulting in 120ms RTT (30ms path RTT and 
90ms buffer-bloat).

  class class-default
   shape average 800 mbps
   random-detect dscp 1,8 1 ms 500 ms
   random-detect dscp 0,2-7 5 ms 1000 ms
   random-detect dscp 9-63 10 ms 1000 ms

This basically says that for LE and CS1, start dropping packets at 1ms of 
buffer fill. Since some applications use CS1 for scavanger, it made sense 
to me to treat CS1 and LE the same.

For BE (which I made to be DSCP 0,2-7), start dropping packets at 5ms 
buffer fill, less agressively compared to LE.

For the rest, don't start dropping packets until 10ms buffer fill, giving 
it slight advantage (thought here being that gaming traffic etc should not 
see much drops even though they will see some induced RTT because of BE 
traffic).

This typically results in LE using approximately 30-50 megabit/s when 
there are 10 LE TCP sessions and 10 BE TCP sessions, all trying to go full 
out. The BE sessions then get ~750 megabit/s. The added buffer delay is 
around 5-10ms as that's where the BE sessions settle their BW usage. 
Platform unfortunately doesn't support ECN marking.

If I were to spend queues on this traffic instead of using RED, I would do 
this differently. I will do more tests with lower speeds etc, this was 
just initial testing for one use-case, but also to give an example of what 
can be done on currently shipping platforms. I know there are much better 
ways of doing this, but I want this into networks NOW, not in 5-10 years. 
So the easier the advice, the better chance we get this into production 
networks.

I don't think it's a good idea to give CS1/LE no bandwidth at all, that 
might cause failure cases we can't predict. I prefer to give LE traffic a 
big disadvantage, so that it might only get 5-10% or something of 
bandwidth, when there is competing traffic.

I will do more testing, I have several typical platforms available to me 
that are in wide use.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: <[email-masked]>
Date: Thu, 12 Apr 2018 06:56:03 +0000

Hi Toerless,

My comments marked [RG]...

Regards, Ruediger

-----Ursprüngliche Nachricht-----

.............
> of RFC8100 style mappings. I'm pessimistic.

Forget interdomain issues. The question is whether anyone can propose a useful QoS setup for draft-ietf-tsvwg-rtcweb-qos even within an enterprise leveraging the three priorities. If there is a good recipe to operationalize this in an enterprise, we can look at figuring out if/how to use the same recipe in L3VPN. I am quite pessimistic about the three priority model.

I think its not even clear what the desirable behavior under contention across those priorities is. I think the goal was just to provide nerd-knob heaven, aka: make RTCweb apps configurable so one could start experimenting in enterprises with priorities and DSCP and see what QoS setup might make sense. Which is not a bad idea. Its just not a complete solution, so i would not dare to recommend anything to SPs for this today.

[RG] I'm not sure whether I understand you. Do you say, the DSCPs of draft-ietf-tsvwg-rtcweb-qos are by and large reserved to distinguish 3 priorities? I must admit that my interest in drafts reserving a multitude of DSCPs for unclear purposes is limited. I believe in aggregated scheduling. Deploy separate classes, if this proves to be of advantage ("prove", not "believe"). Someone has to configure and operate all this scheduling, and sometimes debugging is required. If something is new, a basic standard set which is expandable to me is more favorable, than a flexible (aka complex) toolset.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 09:43:09 +0200

The RTCweb DSCP draft does not introduce any new DSCP,
it just recommends how to use existing DSCP for
the combination of RTCewb media types and 3 priorites.
These are still traffic aggregates, but its quite unclear
to me how a typical deployment would configure the classes
and how apps would assign priorities, and what the
final user experience would be (especially under congestion).

Given how this is in RFC ed queue, there is no point
arguing this draft now, Brian and I somehow derailed into
the topic. IMHO the only relevant part for the LE draft
discussion is that i really prefer DiffServ standards work that
is not only specification of magic parameters but has
some well understood working deployment options and benefits.
And ideally these are documented too in RFCs.

So i think we're kinda saying the same thing.

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 10:05:15 +0200

Thanks Mikael, interesting data

We have done some pretty advanced work in the past with different
drop profiles in single queues, see eg: draft-lai-tsvwg-normalizer,
and i got out of this experience thinking that these drop profiles are very
magic parameter based tools. Results change widely with the
different congestion control and short term burstyness of flows.
RED profiles will also not meet the current written requirement in
the draft to not send LE packets when non-LE packets are queued,
but of course to really do this, you would need strict
priority queuing with a lowest prio queue for LE. I think thats
very uncommon on routers.

I would hope today we could get a separate WFQ for LE on all
platforms which looks to me like the much easier and easier to
predict solution. If you tell me thats not the case, my
interest to delve again into RED profiles will raise again *sigh*

Wrt to give LE more than 0% bandwidth share: In another mail i
made the argument that the percentage you assign becomes yet
another magic parameter where its unclear what the best value is,
and worst case we figure out it should be consistent across paths.

Therefore IMHO it would be better if we could persuade apps to
use BE/LE in a hybrid fashion, preferring LE, but resorting to LE
to guarantee meeting deadlines - especially when LE gives you
actual zero for longer periods. That way we have a very simple,
non-magic parameter in the network. LE is really only leftover.

Admittedly this approach puts some more complexity into the app,
so i am not saying you're wrong, i am just saying N > 0 makes
LE more compex for the network operator and i wouldn't know
a simple recipe how to optimize N.

And both approaches would of course be complementary.

Cheers
   Toerless

On Thu, Apr 12, 2018 at 08:39:25AM +0200, Mikael Abrahamsson wrote:
> I have received questions from operational people for configuration examples
> for how to handle LE/BE etc. So I did some work in our lab to give some kind
> of example.
> 
> So my first goal was to figure out something that'd do something reasonable
> on a platform that'll only do DSCP based RED (as this is typically available
> on platforms going back 15 years). This is not optimal, but at least it
> would be deployable on lots of platforms currently installed and moving
> packets for customers.
> 
> What I came up with that would give LE ~10% of access bandwidth compared to
> BE, and a slight advantage for anything that is not BE/LE (goal was to give
> this traffic a lossless experience) was this:
> 
> buffer fill. Since some applications use CS1 for scavanger, it made sense to
> me to treat CS1 and LE the same.
> 
> For BE (which I made to be DSCP 0,2-7), start dropping packets at 5ms buffer
> fill, less agressively compared to LE.
> 
> For the rest, don't start dropping packets until 10ms buffer fill, giving it
> slight advantage (thought here being that gaming traffic etc should not see
> much drops even though they will see some induced RTT because of BE
> traffic).
> 
> This typically results in LE using approximately 30-50 megabit/s when there
> are 10 LE TCP sessions and 10 BE TCP sessions, all trying to go full out.
> The BE sessions then get ~750 megabit/s. The added buffer delay is around
> 5-10ms as that's where the BE sessions settle their BW usage. Platform
> unfortunately doesn't support ECN marking.
> 
> this differently. I will do more tests with lower speeds etc, this was just
> initial testing for one use-case, but also to give an example of what can be
> done on currently shipping platforms. I know there are much better ways of
> doing this, but I want this into networks NOW, not in 5-10 years. So the
> easier the advice, the better chance we get this into production networks.
> 
> big disadvantage, so that it might only get 5-10% or something of bandwidth,
> when there is competing traffic.
> 

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 12 Apr 2018 10:14:00 +0200 (CEST)

> interest to delve again into RED profiles will raise again *sigh*

There are too many platforms and they're all different as you already 
said. Common platforms will give you the possibility of having a few 
queues, so I will also investigate what the result would be of having 
three queues, one for LE, one for BE and one for rest, where LE gets 5% 
and the two other queues share the rest (getting half the bw each if both 
are congested).

This kind of setup would result in no congestion seen by rest of traffic 
even if BE and LE are fully congested.

If I set the LE drop profile to 1ms/2ms then it stopped working when there 
was BE congestion. I think this is not what one wants to happenn for 
normal customers, instead LE traffic should get some bandwidth even if 
it's a smallish single digit percentage of the total.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Apr 2018 11:12:35 +0200

On Thu, Apr 12, 2018 at 10:14:00AM +0200, Mikael Abrahamsson wrote:
> There are too many platforms and they're all different as you already said.
> Common platforms will give you the possibility of having a few queues, so I
> will also investigate what the result would be of having three queues, one
> for LE, one for BE and one for rest, where LE gets 5% and the two other
> queues share the rest (getting half the bw each if both are congested).
> 

Right. Whats "the rest" ?

Stuff like IPTV/voice/guaranteed L3VPN service traffic or the like ?

> was BE congestion. I think this is not what one wants to happenn for normal
> customers, instead LE traffic should get some bandwidth even if it's a
> smallish single digit percentage of the total.

Argument in favour of some small > 0 share of bandwidth:

It does allows for many apps to continue in LE working without change
because they would just become just terribly slow - but would not need to
be restarted/rewritten which is necessary if the rate would go to
0 for periods longer than common TCP timeouts.

The other interesting question for the LE queue is what impact
the queue size has. If i remember correctly, the queue size
is not really in msec even if it can be specified in msec, right ?

Instead it would be a translation of msec to an actual #packet/#byte amount
for the nominal share of the queue, right ? (i am not quite sure, would
need to check docs).

I am asking, because if that interpretation is right, and queue size
of LE is e.g.: 20 msec at 5% share of the link, but LE actually
get 100% of link when BE is unused, then queue size in that sitution
would be just 1 msec, and that might have a negative impact.

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 13 Apr 2018 08:28:37 +1200

Mikael,

Very interesting, always nice to hear from the real world. You do
prove one of my favourite sayings: "You can't beat queueing theory".

> bandwidth, when there is competing traffic.

Firstly, of course the downside of misusing the CS1 codepoint for LE
is that you might damage traffic intended to use precedence 1 (i.e. better
than default). But secondly, indeed 100% loss of LE will cause
application layer failure - LE applications are *supposed* to fail
when the network is too busy. However, without a router having specific
handling for LE, what you describe seems pretty logical.

Regards
   Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 13 Apr 2018 06:14:45 +0200 (CEST)

On Thu, 12 Apr 2018, Toerless Eckert wrote:

> Right. Whats "the rest" ? Stuff like IPTV/voice/guaranteed L3VPN service 
> traffic or the like ?

These are packets with all DSCP values that are not BE and LE.

> The other interesting question for the LE queue is what impact the queue 
> size has. If i remember correctly, the queue size is not really in msec 
> even if it can be specified in msec, right ?

Correct. I re-discovered this, you can see my email in aqm on this. So I 
will be bufferbloating LE but I imagine this will be fine.

> for the nominal share of the queue, right ? (i am not quite sure, would

No, that's correct.

> would be just 1 msec, and that might have a negative impact.

Correct.

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 13 Apr 2018 19:22:45 +0200

Thanks, Mikael

Let me try to restate in a form that hopeffully gets us closer to
candidate text for the draft (or another draft if explanations
are not desired in this draft - i am trying to figure out what
we need for an actual LE service, the draft may only be allowed
to cover subsets of this).

To simplify discussion:
Start with a link 100% BE. BE queue has M bytes and whatever form
of AQM, e.g.: RED or newer.

Add LE as queue with WFQ or similar. BE 95%, LE 5% share.

BE queue keeps M bytes, after all, therre might not be any LE traffic,
so BE should still work well at 100%.

To allow LE traffic to get exactly the same behavior as BE traffic
when there is no BE traffic, LE queue would need to also have M bytes.

Contention: current ASIC based queuing systems can not share
LE and BE queues, so a toal of 2*M byte buffer space would be needed.
(please someone tell me my contention is wrong ;-).

Any form of current AQM supported by ASICs will have AQM parameters
effectively in bytes/bit or packets, even if config interfaces may
make it look like time. In result, with this setup, worst case
BE gets 95% share, and LE 5%, but then LE queue and AQM (e.g.:
red parameters) are 20 times longer than what they've been calculated
for with 100% link assumed. Result: buffer-bloat in LE.

Alternative: LE queue and AQM parameters are actually sized for M/20,
so LE will receive same treatment as BE when BE maxes out its
95% and LE gets its 5%. But now in the absence of BE traffic, LE
will get 1/20 the amount of buffer size when going up to 100%
link utilization.

Q1: What would be the impact to BE traffic if one would
scale BE queue-size and AQM parameters to 1/20th of what they
ideally should be ? I guess some performance impact, but any
useful estimate how big that impact would be ? Worse than
overall performance, would it create more unfairness between
short RTT and long RTT flows ? (i fear so).

Q2: Any knowledge if dedicated LE flow-control and congestion
control algorithms would fare better under lower amount of
buffering available ? E.g.: i would assume that rate based flow
control thats is shaping the sender such as in BBR would
require less buffering because it is less bursty.

I would hope we can figure out a good recommendation for LE
for the LE B/20 queue config. If that setup will give reasonable
performance with existing TCP algorithms, e.g: in the absence
of BE traffic LE with QUBIC TCP could get to > 50% link,
that would be a good start and keep LE requirements reasonable
(one more queue per link, 5% more bufferspace need per link).

If specific LE CC/flow-control recommendations can get
performance up to 100% that would be ideal. Of course this
would require more work on app/endpoint side to select / use
the appropriate CC and/or even new transport proto.

Q: How likely is it that M*2 buffer space requirement would
be a challenge in deployments ?

The less a problem it is, the easier we could skip the
M/20 setup for LE and would only need to worry about the
bufferbloat questions for LE - which should be much less
of an issue. Just from the perspective of evaluating the lowest
feasible additionalr esource seetup for LE i would still like
to see if there are answers for the M/20 setup.

Cheers
    Toerless

But first question to be would be how badly LE traffic would berform
if there is no BE traffic. 
I'd start

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 14 Apr 2018 08:08:49 +0200 (CEST)

On Fri, 13 Apr 2018, Toerless Eckert wrote:

> Q1: What would be the impact to BE traffic if one would scale BE 
> queue-size and AQM parameters to 1/20th of what they ideally should be ? 
> I guess some performance impact, but any useful estimate how big that 
> impact would be ? Worse than overall performance, would it create more 
> unfairness between short RTT and long RTT flows ? (i fear so).

In my testing with 10 TCP flows, 30ms RTT, hundreds of megabits/s of 
access speed configured, the 0.5ms LE buffer before drop probability went 
non-zero (5% LE 10ms configured, resulting as you say in 0.5ms actual when 
there is no competing BE) meant the LE traffic would not utilise the link 
speed but would sawtooth between 75%-95% of used capacity (1 second 
interval shown in iperf3).

> I would hope we can figure out a good recommendation for LE for the LE 
> B/20 queue config. If that setup will give reasonable performance with 
> existing TCP algorithms, e.g: in the absence of BE traffic LE with QUBIC 
> TCP could get to > 50% link, that would be a good start and keep LE 
> requirements reasonable (one more queue per link, 5% more bufferspace 
> need per link).

When I configured LE to insted have 50ms before RED curve (resulting in 
2.5ms when there is no other traffic) the above test fared a lot better. 
This however caused ~50ms of buffer bloat when there was competing BE 
traffic.

This is also heavily dependent on access speed, the red curves need to be 
different depending on if the access speed is 10 megabit/s, 100 or 1000. 
The lower the speed, the more ms you need in order for the flows to behave 
more smoothly.

> The less a problem it is, the easier we could skip the M/20 setup for LE 
> and would only need to worry about the bufferbloat questions for LE - 
> which should be much less of an issue. Just from the perspective of 
> evaluating the lowest feasible additionalr esource seetup for LE i would 
> still like to see if there are answers for the M/20 setup.

My results show decent performance even at very low queue depths for LE 
traffic, at least at higher access speeds.

My main interest in this anyway is to solve the problem with access speeds 
in the 10-1000 megabit/s range and come up with solutions that are 
deployable on most providers existing BNG (and equivalent) devices.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 14 Apr 2018 18:50:43 +1200

> This however caused ~50ms of buffer bloat when there was competing BE 

If buffer bloat is a problem for an "LE" application, it seems to me
that the application is badly designed.

Designing applications and transports to make good use of LE
seems like a good PhD topic to me. I certainly wouldn't expect
off-the-shelf FTP and TCP to do well, for example.

Regards
   Brian

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 16 Apr 2018 20:07:47 +0200

> that the application is badly designed.

Right. I was asking the question mostly because of the need for
buffer space. With the results Mikael showed (thanks, great detail, Mikael),
i would say the recommendation to LE could be:

-> Afford as much buffer space to LE queue as you can based on HW
   capabilities of the device, up to the same buffer space used for
   BE. This might end you up using 2 times the amount of buffer space
   as you needed with only BE though.

   If you can afford these large amount of buffer space for the LE
   queue, you will create buffer-bloat for LE traffic when three is
   a lot of BE traffic competing - and LE therefore reduced to its
   weighted fair share of the link. Unlike BE traffic that often
   carries latency sensitive traffic such as real-time audio or
   even video, this buffer bloat should never be a problem for LE
   traffic. LE never promises any specific delay. Applying LE
   PHB to latency conscious traffic is wrong.

-> If you can not afford more buffer space on a device for LE, simply
   scale the LE queue respective to its weighted fair share of the
   link, for example only 5% of the BE queue if BE has 5% share.
   In this case, BE traffic will likely not able to fully utilize the
   link in the complete absence of BE traffic, but due to flow
   control issues max out at a lower rate. An example test showed
   the following results:
   effective LE buffer size before start of RED drop: 0.5 msec
   10 TCP flows (QUBIC), 1 Gbps link: 75..90% utilization max.
   Results would expected to be worse with slower links, better with
   faster links.

Now we'd just need to find a good place/draft to document these
type of BCP operational aspects of LE. Not sure if the PHB draft
is the right place. Would make life easier of course if it was.

Cheers
    Toerless

WARNING: only rants following, proceed at your own risk.

> off-the-shelf FTP and TCP to do well, for example.

Have the PHD requirements dropped to the floor  since i left
university ?

I think even the hybrid BE/LE algorithm i suggested to document
for deadline based LE applications would at best qualify for a
bachelor thesis. If you do a lot of simulations to measure/simulate
queuing behavior LE vs. others using different applications, that
would qualify IMHO for a master thesis due to sheer amount of
work. 

For PhD the work itself would need to come up with something new.

While i do like novely in general, i also know the history of
young PhD joining the industry, pushing their novel scheduling
or aqm thesis into products, and then lots of poor operators
suffering with more and more incompatible nerd-knobs to manage
QoS in their network using multiple type of products. So...
i would be somewhat more careful here.

The root cause btw is the pressure of doing something novel raised
against PhD students often leads to just doing something unnecessarily
different because there just isn't that much "better" possible.
Finished PhD trying to apply their thesis at work because they have
made themselves believe it is better. Oh well, thats just missing
operational experience. And leaves a lot of really important
incremental improvement work unsolved because it's not novel enough
for research but still to difficult to productice due to dependencies
in the industry.

---

From: "Black, David" <[email-masked]>
Date: Wed, 18 Apr 2018 19:48:24 +0000

With <WG Chair Hat=OFF>, I wanted to +1 this conclusion:

> See above - let's consider point II) canned right now and

I strongly support both this separation of the two approaches and not taking
up approach II) in the current LE PHB draft.   Two quick reactions/comments:

a) Approach II) to LE CC seems only feasible in a fully controlled environment,
	at least for the network to which the sending endpoint is attached.
	Diffserv "in the wild" deployments can be much less disciplined.

b) Approach II) also feels like a different PHB by contrast to the LE PHB where
	there's an expectation that endpoints will reduce LE traffic in the face
	of competition with BE traffic (see SHOULD/MUST discussion on
	level of requirement for use of LBE transport in another thread).

Thanks, --David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Toerless Eckert
> Sent: Wednesday, April 11, 2018 9:26 PM
> To: Bless, Roland (TM)
> Cc: [email-masked]
> Subject: Re: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 19 Apr 2018 08:38:08 +0200 (CEST)

On Thu, 12 Apr 2018, [email-masked] wrote:

> advantage ("prove", not "believe").. Someone has to configure and 
> operate all this scheduling, and sometimes debugging is required. If 
> something is new, a basic standard set which is expandable to me is more 
> favorable, than a flexible (aka complex) toolset.

I've had some discussions with other people in the operational community, 
and multiple questions have come up.

1. If I do not bleach at all, should I do something based on this new 
draft? (there are people with MPLS networks that do their "bleaching" by 
assigning all traffic to BE on their transport (802.1p=0 / MPLS EXP=0), so 
they actually do not bleach customer diffserv in the IP header.

2. If I do bleach, should I remap CS1 to the new LE CP?

So basically, what should the setup be for ISPs for different scenarios? 
Towards the customer? Towards other ISPs?

There are two major camps out there that I see, those who just bleach 
everything, and those who do not bleach at all and are just transparent to 
customer traffic. We need guidance to both of these camps (I'm sure there 
are people doing different things as well, but if we give guidance to both 
of these cases then I think the people with hybrid setups can figure out 
what they need to do).

There are also two dimensions, what should they do with 
remapping/bleaching, and what should the PHB be (if any)? So let me write 
some suggestions and let's discuss.

So let's start with the operator that does no bleaching:

1. PHB should be that CS1 and LE is treated the same, ie lower priority 
than BE.
2. Do not bleach from customer.
3. Re-map CS1 to LE at edge towards other ISPs by default, but ISPs can 
agree on different behaviour.

For the ones that do bleaching:

>From customer/other ISP.
1. Remap CS1 to LE.
2. Do not bleach LE.
3. Bleach rest to BE.
4. PHB behaviour should be LE gets lower priority than BE.

Thoughts?

---

From: <[email-masked]>
Date: Thu, 19 Apr 2018 07:33:48 +0000

Hi Mikael,

my views marked [RG].

Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg [mailto:[email-masked]] Im Auftrag von Mikael Abrahamsson
Gesendet: Donnerstag, 19. April 2018 08:38
An: [email-masked]
Betreff: Re: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

On Thu, 12 Apr 2018, [email-masked] wrote:

> something is new, a basic standard set which is expandable to me is 
> more favorable, than a flexible (aka complex) toolset.

I've had some discussions with other people in the operational community, and multiple questions have come up.

1. If I do not bleach at all, should I do something based on this new draft? (there are people with MPLS networks that do their "bleaching" by assigning all traffic to BE on their transport (802.1p=0 / MPLS EXP=0), so they actually do not bleach customer diffserv in the IP header.

[RG] There's no obligation to bleach. The current standards allow bleaching if a domain wants to protect internal 
DiffServ transport from traffic which is received with Diffserv markings without having been negotiated. One reason is, 
that in that case, the meaning of the DSCPs is simply unknown to the receiver.

[RG] Its perfect to be liberal with what you receive - the conservative part is sending. Let's assume my domain is following on yours. My OP-folk receives packets marked with a DSCP from your domain and asks, which PHBs are linked to these DSCPs? An honest response would be "dunno".

[RG] Only if the sender advised you, that packets marked by CS1 indeed classify for LE! We've discussed this more than once: DSCPs do not carry any end to end information. A DSCP which is received from a neighbor domain, which is not bothering to inform you about the PHB linked to their DSCP markings, to your domain is an unknown DSCP. Your staff may degrade traffic to LE by the remap action you describe.

[RG] I'm not sure whether we've checked that all domains sending CS1 today actually classify it as LE.

[RG] The carrier I work for supports Diffserv.

.........

1. PHB should be that CS1 and LE is treated the same, ie lower priority than BE.
2. Do not bleach from customer.
3. Re-map CS1 to LE at edge towards other ISPs by default, but ISPs can agree on different behaviour.

[RG] As above, don't agree as long as I don't know whether you assume the PHB linked to a DSCP to be known and agreed by both interconnected parties.

1. Remap CS1 to LE.
2. Do not bleach LE.
3. Bleach rest to BE.

[RG] As above, don't agree as long as I don't know whether you assume the PHB linked to a DSCP to be known and agreed by both interconnected parties. Further, a remap rather than bleaching means that there's an agreement on the PHB linked to CS1 between the interconnected networks. Correct?

[RG] You only discuss the case where two interconnected networks both support LE, correct?

Thoughts?

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 19 Apr 2018 10:27:38 +0200 (CEST)

On Thu, 19 Apr 2018, [email-masked] wrote:

> [RG] Only if the sender advised you, that packets marked by CS1 indeed 
> classify for LE! We've discussed this more than once: DSCPs do not carry

I am talking residential customers who are using their mobile devices and 
regular PCs sitting on a residential Internet access.

Saying "this is not defined" is not helpful. I'd like to try to define 
some kind of minimum "Internet standard" here. If CS1 is not agreed to be 
LE, then we need to say "only 000001 is LE unless you know otherwise" and 
treat it as such.

So I would like us to end up with 000001 actually being LE and some kind 
of BCP on how to try to tie it all together with recommended PHB.

I know this has failed before, but is there a reason why this newly 
defined LE can't actually have a recommended PHB?

If the answer is "it can't" then I see no purpose in progressing this 
draft, then it's useless to define it.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 19 Apr 2018 10:30:05 +0200 (CEST)

> [RG] The carrier I work for supports Diffserv.

In what parts of the network? Residential broadband? Mobile? Enterprise 
VPN? Enterprise Internet-access? Internet transit? What's the PHB for each 
diffserve CP in each of these network parts? Is it the same in all of 
them?

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 19 Apr 2018 10:54:39 +0200

Hi Mikael,

see inline.

Am 19.04.2018 um 08:38 schrieb Mikael Abrahamsson:
> I've had some discussions with other people in the operational
> community, and multiple questions have come up.

Thanks, this is really valuable...

> assigning all traffic to BE on their transport (802.1p=0 / MPLS EXP=0),
> so they actually do not bleach customer diffserv in the IP header.

Ok, so _bleaching_ for me would be to set the DSCP to 0. The subsequent
domains have no clue then that it was actually a different PHB before
(yes, cross DS domain semantics of PHBs have to be negotiated according
to the Diffserv architecture, but let's assume that we use a standard
DSCP). So, yes domains could put the LE traffic into their BE aggregate,
but then the whole advantage of LE for the provider is gone. The current
draft recommends to keep the LE DSCP in case one maps the LE PHB into
the BE PHB, so that subsequent domains can at least treat it as intended
again.

> 2. If I do bleach, should I remap CS1 to the new LE CP?

This is a bit confusing. What do you mean by "bleach" here, treat
LE as BE? Remarking of DSCPs is a somewhat orthogonal problem since
Diffserv requires a flexible mapping from DSCP -> PHB. So my
recommendation is to remark CS1 to LE for domains that formerly used
CS1 as LE codepoint. Otherwise, you cannot be sure whether CS1 actually
is CS1 in its original meaning or whether it should be treated as LE.
Removing that ambiguity is the main motivation for my LE PHB draft.

> everything, and those who do not bleach at all and are just transparent
> to customer traffic. We need guidance to both of these camps (I'm sure
> there are people doing different things as well, but if we give guidance
> to both of these cases then I think the people with hybrid setups can
> figure out what they need to do).
> 
> remapping/bleaching, and what should the PHB be (if any)? So let me
> write some suggestions and let's discuss.
> 

Yep, however, bilateral agreement on different behavior is covered by
the Diffserv architecture in general.

> For the ones that do bleaching:

I don't like this fuzzy use of bleaching, see above.
So you basically mean here map DSCP XYZ to the default PHB (BE)?
For me, bleaching includes remarking to DSCP 0, too.

> 1. Remap CS1 to LE.

Nope, only in cases where CS1 was used before in the LE meaning.

> 2. Do not bleach LE.

Yes, keep LE DSCP, no matter how you treat it internally.

> 3. Bleach rest to BE.

What is the rest?

> 4. PHB behaviour should be LE gets lower priority than BE.

Yes, otherwise an ISP looses the advantage of protecting BE
traffic from LE traffic.

Regards
 Roland

---

From: <[email-masked]>
Date: Thu, 19 Apr 2018 10:02:23 +0000

IP/MPLS networks and products operated from Germany are supported. I can't speak for other European branches or the US branch. Customer communication may consume up to 4 PHBs. I've heard from other carriers also supporting 3-4 PHBs. Best Effort and EF with the standard DSCP are widespread. RFC 8100 is my preferred interconnection option.

Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Donnerstag, 19. April 2018 10:30
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> [RG] The carrier I work for supports Diffserv.

In what parts of the network? Residential broadband? Mobile? Enterprise VPN? Enterprise Internet-access? Internet transit? What's the PHB for each diffserve CP in each of these network parts? Is it the same in all of them?

---

From: <[email-masked]>
Date: Thu, 19 Apr 2018 10:05:46 +0000

I think you are free to do what you prefer on a residential access. Maybe unless your residential products are regulated. 

Your initial statement doesn't point out, that you are discussing residential access issues only. Thanks for adding that information. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Donnerstag, 19. April 2018 10:28
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> classify for LE! We've discussed this more than once: DSCPs do not 

I am talking residential customers who are using their mobile devices and regular PCs sitting on a residential Internet access.

Saying "this is not defined" is not helpful. I'd like to try to define some kind of minimum "Internet standard" here. If CS1 is not agreed to be LE, then we need to say "only 000001 is LE unless you know otherwise" and treat it as such.

So I would like us to end up with 000001 actually being LE and some kind of BCP on how to try to tie it all together with recommended PHB.

I know this has failed before, but is there a reason why this newly defined LE can't actually have a recommended PHB?

If the answer is "it can't" then I see no purpose in progressing this draft, then it's useless to define it.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 19 Apr 2018 12:52:08 +0200 (CEST)

> Ok, so _bleaching_ for me would be to set the DSCP to 0. The subsequent

Ok, what should I call the following (all values decimal for the 6 DSCP CP 
bits)

Look at diffserv CP values 0-63.
If CP in range 2-63, SET CP=0
If CP == 1 pass unchanged
If CP == 0 pass unchanged

This bleaches 2-63 to 0 and leaves 1 unchanged. What do we call this?

> LE as BE? Remarking of DSCPs is a somewhat orthogonal problem since

For me there are two things an ISP does depending on DSCP values. It has a 
policy that (might) inspect CP values and change them. It also has a PHB 
depending on the CP values.

The simplest policy is "SET CP = 0 on all packets". This seems to be what 
you call bleaching.

> Removing that ambiguity is the main motivation for my LE PHB draft.

If the domain previously just bleached everything and now wants to deploy 
support for LE (both allowing the CP to enter the ISP network and also 
creating different PHB for LE vs BE), what should it do regarding CS1?

Saying "it depends" is not helpful. This is the Internet with billions of 
devices and users. What should be the default behaviour for non-managed 
devices run by normal people that they bought from the electronics store. 
I am not talking about managed enterprise networks etc. I am talking about 
the most common network out there, the "oh, I need Internet access to 
watch cat pictures on my own device and it's default out of box configured 
from DSCP point of view".

> What is the rest?

CP between 2 and 63. Not 0 and 1.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 19 Apr 2018 12:57:25 +0200 (CEST)

On Thu, 19 Apr 2018, [email-masked] wrote:

> Your initial statement doesn't point out, that you are discussing 
> residential access issues only. Thanks for adding that information.

For me it's obvious that when I use the term "ISP" it means "BYOD style 
Internet access with default out-of-the-box-from-the-electronics-store 
behaviour of applications and devices".

What term should I use in the future to avoid this confusion in the 
future?

---

From: <[email-masked]>
Date: Thu, 19 Apr 2018 11:27:13 +0000

For me "residential access" is a useful information.

Regards, Ruediger 

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Donnerstag, 19. April 2018 12:57
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: AW: [tsvwg] CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> residential access issues only. Thanks for adding that information.

For me it's obvious that when I use the term "ISP" it means "BYOD style Internet access with default out-of-the-box-from-the-electronics-store
behaviour of applications and devices".

What term should I use in the future to avoid this confusion in the future?

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Apr 2018 08:34:17 +1200

How on earth can an operator blindly map CS1 to LE? Maybe the source
means CS1 as defined in RFC2474.

Unless you have an explicit agreement with the source, there is no correct
answer. Remarking CS1 to default is somewhat closer to "first, do no harm",
since it either slightly downgrades traffic intended for CS1 or slightly
upgrades traffic intended for LE but falsely labelled as CS1.

Regards
   Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Apr 2018 06:47:28 +0200 (CEST)

> means CS1 as defined in RFC2474.

Lots of maybes in this area it seems. RFC8325 says to send CS1 to 
background queue.

> upgrades traffic intended for LE but falsely labelled as CS1.

Ok, so just to make things perfectly clear. I am trying to create a BCP 
for residential broadband for PHB and CP treatment. This means BYOD, no 
contracts, no knowledge. This includes mobile, wifi, wireline etc. I would 
like whatever we come up with to be RFC8325 compatible and at least not 
conflicting with it. My goal is to go to the operator community and try to 
get as wide adoption of this as possible. If I succeed, this will be the 
new default behaviour for residential broadband access.

So, from this discussion it seems to me that what we would like to happen 
that would do least harm is this pseudocode.

CP is decimal representation of diffserv CP 0 to 63.

If CP => 0 then pass; # do not bleach anything

The PHB is set up like this:

If CP == 0 goto queueBE;
If CP == 1 goto queueLE;
If CP => 2 goto queueRest;

The queues are set up to drain with current ratios:

queueBE 	10
queueRest 	10
queueLE 	1

.. or something along those lines, meaning queueLE gets ~5% of the 
capacity when the other queues are full.

If operator wants to bleach, then pseudocode changes to:

If CP == 1 then pass; # allow LE
If CP => 2 then set CP = 0; # set 2-63 to 0

The PHB is set up like this:

If CP == 1 goto queueLE;
goto queueBE;

The queues are set up to drain with current ratios:

queueBE 	20
queueLE 	1

This sums up into roughly same thing but just 2 queues and LE gets ~5%.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Apr 2018 17:23:29 +1200

Mikael,

> If CP == 1 goto queueLE;

Sure, if IANA assigns the code point that the draft requests.

> I would 
> conflicting with it.

If you mean 8325 *as updated by draft-ietf-tsvwg-le-phb* we are fine.
But it will be very important to get that update into everybody's head.

   Brian

Regards
   Brian Carpenter

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Apr 2018 08:28:04 +0200 (CEST)

> Sure, if IANA assigns the code point that the draft requests.

Of course.

> But it will be very important to get that update into everybody's head.

Can you please enlighten me why you think it's ok to put CS1 into the same 
queue as LE (background, as per 8325), but it's not ok to remap CP 
CS1->LE?

Because my proposal for the second scenario in my earlier email would 
otherwise have been (adding remap of CS1 -> LE):

If CP == 1 then pass; # allow LE
If CP == 8 then set CP = 1; # remap CS1 to LE
If CP => 2 then set CP = 0; # set 2-63 to 0

I think this is an important distinction and we need to decide how to 
handle this case. Is CS1 more likely to be equivalent to BE or LE? Because 
in this scenario, it's going into one of these two. What's the least harm?

---

From: <[email-masked]>
Date: Fri, 20 Apr 2018 06:40:22 +0000

Mikael, by "residential broadband" you mean the consumer edge device or the home gateway - a piece of equipment which today isn't located within provider premises? 

The direction of traffic at the residential broadband gateway is "to the Internet" and congestion is caused by sources within the home network. All of that is BYOD equipment (in some countries including the residential broadband gateway). 

Should that be correct, I support your idea of giving LE queuing guidance to the residential broadband gateway vendors. 

If your proposal is meant to cover other pieces of equipment, like Broadband Network Gateways, please clarify. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----

Ok, so just to make things perfectly clear. I am trying to create a BCP for residential broadband for PHB and CP treatment. This means BYOD, no contracts, no knowledge. This includes mobile, wifi, wireline etc. I would like whatever we come up with to be RFC8325 compatible and at least not conflicting with it. My goal is to go to the operator community and try to get as wide adoption of this as possible. If I succeed, this will be the new default behaviour for residential broadband access.

So, from this discussion it seems to me that what we would like to happen that would do least harm is this pseudocode.

If CP == 0 goto queueBE;
If CP == 1 goto queueLE;

queueBE 	10
queueRest 	10
queueLE 	1

... or something along those lines, meaning queueLE gets ~5% of the capacity when the other queues are full.

If CP == 1 goto queueLE;
goto queueBE;

queueBE 	20
queueLE 	1

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Apr 2018 08:58:32 +0200 (CEST)

> If your proposal is meant to cover other pieces of equipment, like Broadband Network Gateways, please clarify.

This guidance is for CP handling/mapping ingress on BNG and on peering 
routers (ISP edge), and CP based PHB ingress/egress customer-facing for 
both BNG, HGW and all other equipment handling last-hop-to-customer 
traffic and intra-customer network for BYOD non-managed networks (such as 
a residential network).

So some examples:

What should the PHB be egress on BNG -> customer

What should the CP treatment be ingress on BNG (because HGW is customer 
controlled potentially).

If BNG is doing ingress shaping for instance, what should this PHB be 
(again, customer-controlled HGW).

What is the CP treatment between ISPs when nothing is discussed regarding 
CP treatment (Internet peering default). What should ISP ingress settings 
be towards an IX LAN for instance.

What is the CP based egress PHB towards an IX.

All of this is for Internet traffic, think cat pictures, online gaming, 
OTT streaming services etc. Generic Internet traffic.

In my solution my "BNG" does ingress and egress shaping, and that's what I 
have used so far in my examples. I don't consider the HGW a trusted 
device (it often is, but not all the time).

---

From: <[email-masked]>
Date: Fri, 20 Apr 2018 08:36:06 +0000

Mikael,

thanks. 

Regarding provider equipment, the options depend on the provider preferences and you describe a possible one. Another option is not to trust DSCPs sent by consumer devices.

At an interconnection interface, I side with Brian. To me, a provider sending DSCP marked packets to my domain without exchanging information about the PHBs linked to the DSCPs is not interested in ensuring end to end Diffserv support. General remarking to a codepoint resulting in default transport is viable interconnection router option in that case (if a domain supports Diffserv). I don't however object to other domains behaving differently. 

If a domain doesn't support LE internally, the ISP gateway may drop this traffic at the interconnection egress (requires the suitable DSCPs to be set). That seems to be of more interest in the direction of higher loads. If a domain prefers not to support LE internally, I don't think there's a point in deploying the configuration proposed by you on a backbone interface (it may make sense, if this backbone interface gets congested).

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Freitag, 20. April 2018 08:59
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] residential broadband BCP PHB and CP treatment Re: CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> If your proposal is meant to cover other pieces of equipment, like Broadband Network Gateways, please clarify.

This guidance is for CP handling/mapping ingress on BNG and on peering routers (ISP edge), and CP based PHB ingress/egress customer-facing for both BNG, HGW and all other equipment handling last-hop-to-customer traffic and intra-customer network for BYOD non-managed networks (such as a residential network).

So some examples:

What should the CP treatment be ingress on BNG (because HGW is customer controlled potentially).

If BNG is doing ingress shaping for instance, what should this PHB be (again, customer-controlled HGW).

What is the CP treatment between ISPs when nothing is discussed regarding CP treatment (Internet peering default). What should ISP ingress settings be towards an IX LAN for instance.

All of this is for Internet traffic, think cat pictures, online gaming, OTT streaming services etc. Generic Internet traffic.

In my solution my "BNG" does ingress and egress shaping, and that's what I have used so far in my examples. I don't consider the HGW a trusted device (it often is, but not all the time).

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Apr 2018 15:15:07 +0200 (CEST)

On Fri, 20 Apr 2018, [email-masked] wrote:

> At an interconnection interface, I side with Brian. To me, a provider 
> sending DSCP marked packets to my domain without exchanging information 
> about the PHBs linked to the DSCPs is not interested in ensuring end to 
> end Diffserv support. General remarking to a codepoint resulting in 
> default transport is viable interconnection router option in that case 
> (if a domain supports Diffserv). I don't however object to other domains 
> behaving differently.

I don't understand what this means in practical terms.

How do you expect Internet Tier1:s to know the expected PHBs of traffic of 
their customers customers customers customer?

As per my earlier email, I am trying to write BCP PHB/CP treatment for the 
Internet. This will by definition be very simple, and an alternative to 
"set CP=0 ;send to large FIFO" or "sent to large FIFO" (just ignore CP and 
don't touch it, treat everything as BE regardless of DSCP set) which is 
typically the two ways generally done today.

RFC8325 states lots of things, which I guess is actually implemented in 
products today, or we aim that to be the default. We have lots of history.

Remember, there are applications out there which actually sets DSCP and 
seems to expect PHB treatment, that are in wide use. These application 
developers did this because they thought it was the right thing to do. For 
instance openssh sets IPTOS_LOWLATENCY (as per earlier discussions). Now 
people are trying to change the defaults of ssh and looked at 
IPTOS_LOWLATENCY and said "hm, this looks like it should be AF21 instead". 
Is this sensible default? Should we just ignore these kinds of 
initiatives?

Can we come up with something that is marginally better than we have 
today? My 3 PHB queues was an attempt to do this.

We have an historic mess. Can we say what we think should be the future 
and come up with a recommendation for PHB that is not very harmful but 
still brings us away from "single large FIFO for everything"? It doesn't 
have to be perfect, it just has to be better than we have today and not 
provide too many downsides. This is what I aimed my 3 PHB queues proposal 
to be.

---

From: "Black, David" <[email-masked]>
Date: Fri, 20 Apr 2018 19:56:02 +0000

<WG Chair Hat=OFF>

I'm not Ruediger, but ...

> their customers customers customers customer?

I don't - if the adjacent network is sending other-than-BE traffic to my network that is not supposed to be bleached, then there is some agreement/understanding in place between the two networks about what the DSCPs mean and the resulting PHBs.   Absence of such agreements/understandings is a major cause of current bleaching behavior.

> products today, or we aim that to be the default. We have lots of history.

For this exercise, RFC8325 has too many traffic classes, as does RFC4594, upon which RFC8325 draws heavily.  The intent was always that actual networks would support a subset of the RFC4594 traffic classes, as stated in the introduction section of RFC4594:

   However, we expect that
   network administrators will implement a subset of these classes
   relevant to their customers and their service offerings.

As Ruediger has already mentioned, RFC 8100 is likely to be a better starting point for this exercise, to which one could add a few things like LE, as opposed to starting from RFC4594/RFC8325 and figuring out what large collection of things to remove.

> developers did this because they thought it was the right thing to do.

IMHO, it's very important to distinguish this endpoint-marking use case from the network interconnect use case.  For an example of endpoint marking guidance, see (IESG-approved, but not yet published as an RFC) 	
draft-ietf-tsvwg-rtcweb-qos (https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rtcweb-qos/).   For a network interconnect example, see RFC 8100.

> today? My 3 PHB queues was an attempt to do this.

Please consider RFC8100 as a plausible starting point.

Thanks, --David

> From: tsvwg [mailto:[email-masked]] On Behalf Of Mikael
> Abrahamsson
> Sent: Friday, April 20, 2018 9:15 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] residential broadband BCP PHB and CP treatment Re:
> CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Apr 2018 15:46:13 +1200

Hi Mikael,

> queue as LE (background, as per 8325), but it's not ok to remap CP 

Because RFC2474 clearly states that CS1 gets precedence over DF (=CS0).
(Which in fact goes back to RFC791 and RFC1349.)

"PHBs selected by a Class Selector Codepoint
 SHOULD give packets a probability of timely forwarding that is not
 lower than that given to packets marked with a Class Selector
 codepoint of lower relative order..." (RFC2474 section 4.2.2.2)

None of the other RFCs that use the CS1 codepoint for LE have formally
updated RFC2474; in other words they all violate the SHOULD above.
Specifically, RFC4594 did not formally update RFC2474.

A user who sets CS1 because they want better treatment than default
is doing exactly what an IETF standard says. So if you see CS1 on
incoming traffic, you have no idea whether the user wanted Precedence 1
or LE, unless you have a specific understanding with that user.

To me that means that the correct default behaviour is to map CS1 to
DF, as a compromise. Of course, if you *know* whether the user is
relying on RFC2474 or on RFC4594, you can configure the correct
mapping. But that's always true of diffserv.

   Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 21 Apr 2018 07:29:42 +0200 (CEST)

> mapping. But that's always true of diffserv.

Thanks for the explanation. Would it influence your opinion if there were 
data that showed that more applications use CS1 in the LE sense compared 
to using CS1 as higher preference over DF?

Basically my question is that if setting formal references/updates etc 
aside, what have people actually been doing in the past 20 years, should 
that affect our decision?

Discovering that for instance ssh uses IPTOS by default was a revelation 
to me.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 21 Apr 2018 07:46:04 +0200 (CEST)

On Fri, 20 Apr 2018, Black, David wrote:

> IMHO, it's very important to distinguish this endpoint-marking use case 
> from the network interconnect use case.  For an example of endpoint 
> marking guidance, see (IESG-approved, but not yet published as an RFC) 
> draft-ietf-tsvwg-rtcweb-qos 
> (https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rtcweb-qos/).  For a 
> network interconnect example, see RFC 8100.

For me 8100 is not applicable. 8100 talks about interconnects set up and 
agreed on remapping, PHBs etc. What I am trying to create is "Internet 
default", which is the exact opposite. My document would be about what 
happens when there is no agreement and no prior knowledge, neither between 
end-customer and ISP, or between ISPs.

However, draft-ietf-tsvwg-rtcweb-qos is highly applicable, as this speaks 
of what applications should do by default, and thus I want to try to make 
a proposal creating recommended Internet default (last-hop) PHBs that take 
this into account, but still also takes into account that anyone can 
source traffic into this so it shouldn't have strict priority for EF for 
example.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 22 Apr 2018 08:48:00 +1200

> to using CS1 as higher preference over DF?

It doesn't change the basic problem that we have actually published
inconsistent standards, sadly. I really feel bad that I didn't make a fuss
about this earlier, but it was a boiling frog situation, starting with
RFC3662 being only Informational and only "suggesting" a codepoint.

> that affect our decision?

It should certainly affect operational recommendations. So I think you're
correct that a BCP or nearly-BCP is needed, explaining the issues and
defining (a) default configuration and (b) the necessary configuration
options. I also think we need to be sure that we don't create yet more
ambiguity about the CS1 codepoint. The le-phb-04 text removes the treatment
of CS1 from RFC4594, but gives no guidance about CS1 as defined by RFC2474.
At least we could say that it's operationally ambiguous and should not
be used as a default setting in any host.

> Discovering that for instance ssh uses IPTOS by default was a revelation 

Yes. Of course, it was partly for compatibility with legacy code that
RFC2474 specified that all DSCPs were mutable at domain boundaries.

    Brian

---

From: <[email-masked]>
Date: Mon, 23 Apr 2018 08:26:43 +0000

My comments are marked [RG], regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Freitag, 20. April 2018 15:15
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] residential broadband BCP PHB and CP treatment Re: CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

On Fri, 20 Apr 2018, [email-masked] wrote:

> sending DSCP marked packets to my domain without exchanging 
> information about the PHBs linked to the DSCPs is not interested in 
> ensuring end to end Diffserv support. General remarking to a codepoint 
> resulting in default transport is viable interconnection router option 
> in that case (if a domain supports Diffserv). I don't however object 
> to other domains behaving differently.

[RG] RFC8100 should explain that. If RFC8100 requires clarifications, please 
point out the relevant issues.

      As per my earlier email, I am trying to write BCP PHB/CP treatment for the Internet. This will by definition be very simple, and an alternative to "set CP=0 ;send to large FIFO" or "sent to large FIFO" (just ignore CP and don't touch it, treat everything as BE regardless of DSCP set) which is typically the two ways generally done today.

      RFC8325 states lots of things, which I guess is actually implemented in products today, or we aim that to be the default. We have lots of history.

[RG] I'm fine with optimisations related to scheduling of wired/wireless traffic within home networks or at hot spots. This may also include BNG behavior. The case of a provider not supporting a PHB must be expected. Please don't read that as a recommendation not to support the PHB. I want a clear specification, covering all cases which are reasonable to expect.

      Remember, there are applications out there which actually sets DSCP and seems to expect PHB treatment, that are in wide use. These application developers did this because they thought it was the right thing to do. For instance openssh sets IPTOS_LOWLATENCY (as per earlier discussions). Now people are trying to change the defaults of ssh and looked at IPTOS_LOWLATENCY and said "hm, this looks like it should be AF21 instead". 
Is this sensible default? Should we just ignore these kinds of initiatives?

[RG] BCP is based on widespread support. Didn't hear of that in the example you mention. I'm not aware of any discussion of the openssh community within IETF, when they went to pick a codepoint. I can add standards of other SDOs which determine DSCPs and PHBs, but never were liaised with IETF. 

     Can we come up with something that is marginally better than we have today? My 3 PHB queues was an attempt to do this.

     We have an historic mess. Can we say what we think should be the future and come up with a recommendation for PHB that is not very harmful but still brings us away from "single large FIFO for everything"? It doesn't have to be perfect, it just has to be better than we have today and not provide too many downsides. This is what I aimed my 3 PHB queues proposal to be.

[RG] My experience is that those interested in Diffserv first determine their own architecture and deploy and operate that locally.  Then they figure out, that there's world outside of their LAN and Diffserv might be useful end-to-end. The usual conclusion is, the others may adapt to their local scheme, as they made up their mind before deploying and why should they adapt their deployment, rather than the others? I've settled a couple of these discussions, and sometimes partial agreement was the best I could reach.
To me, the Diffserv AF classes offer to much flexibility in terms of DSCPs and PHBs and too little guidance of when to deploy a specific DSCP and the PHB linked to it. We could try to clean up that, but I expect that we have to cope with multiple operational deployments all differing in some detail.

---

From: <[email-masked]>
Date: Mon, 23 Apr 2018 09:32:30 +0000

[RG] marks my comments.

-----Ursprüngliche Nachricht-----
Von: tsvwg [mailto:[email-masked]] Im Auftrag von Mikael Abrahamsson
Gesendet: Samstag, 21. April 2018 07:46
An: Black, David <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] residential broadband BCP PHB and CP treatment Re: CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

On Fri, 20 Apr 2018, Black, David wrote:

> IMHO, it's very important to distinguish this endpoint-marking use 
> case from the network interconnect use case.  For an example of 
> endpoint marking guidance, see (IESG-approved, but not yet published 
> as an RFC) draft-ietf-tsvwg-rtcweb-qos 
> (https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rtcweb-qos/).  For 
> a network interconnect example, see RFC 8100.

       For me 8100 is not applicable. 8100 talks about interconnects set up and agreed on remapping, PHBs etc. What I am trying to create is "Internet default", which is the exact opposite. My document would be about what happens when there is no agreement and no prior knowledge, neither between end-customer and ISP, or between ISPs.

[RG] RFC8100 is about deployment of end-to-end PHBs in the presence of DSCP remarking. Do I get that correct, you propose end-to-end DSCPs to support local or endpoint PHB deployment? 

        However, draft-ietf-tsvwg-rtcweb-qos is highly applicable, as this speaks of what applications should do by default, and thus I want to try to make a proposal creating recommended Internet default (last-hop) PHBs that take this into account, but still also takes into account that anyone can source traffic into this so it shouldn't have strict priority for EF for example.

[RG] draft-ietf-tsvwg-rtcweb-qos is restricted to home-networks, residential networks and local wireless networks. It says "There are cases where these DSCP markings do not help, but, aside from possible priority inversion for "less than best effort traffic" (see Section 5), they seldom make things worse if packets are marked appropriately." And later " WebRTC use of multiple DSCP values may encounter network blocking of packets with certain DSCP values."
Are you suggesting any changes to that? 

[RG] Interestingly, DSCP 46 / EF is the one next to default which can almost be linked to end to end standard PHB and DSCP - if sender and receiver have a service agreement in place. That holds for Germany (public telephony, L3 VPNs, business and wholesale products, wireless and IP sections of wireline packet transport).

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 23 Apr 2018 11:36:42 +0200 (CEST)

On Mon, 23 Apr 2018, [email-masked] wrote:

> [RG] RFC8100 should explain that. If RFC8100 requires clarifications, 
> please point out the relevant issues.

RFC8100 has TL;DR; problems for my use-case. It also doesn't give any 
concrete operational advice.

If someone comes to me and asks "hm, what service-policy should I 
configure towards my 500 internet peers and transit, and towards my 
residential access customers?" and I point them to RFC8100, do you think 
they'd be much wiser even if they forced themselves to read it all?

If they come to me and ask "oh, I want to IPv6-enable my home gateways, 
what should I think of" I can point them to RFC7084 which doesn't have any 
"TL;DR;" issues at all, it's all relevant to their question.

I want to create a BCP (or informational) document that gives concrete 
advice on what to do when you want to support Internet-facing QoS that 
might be slightly better than single FIFO and "set CP=0" (or ignoring it). 
RFC8100 will probably be referenced by this document, but most likely very 
few will read RFC8100 in the process of implementing what they want to 
implement.

Today the policy towards peers/IXes/transits probably looks like this if 
you're running a cisco-router (hand-typed so might not apply cleanly):

policy-map ingress-from-internet-peers
   class class-default
     set dscp 0

Potentially if they run a MPLS-enabled network, they might use "set mpls 
experimental imposition 0" and then just ignore the DSCP bits in their 
MPLS network. Similar towards residential customers.

I want to offer an alternative that's still sane trade-off between risks 
of DDOS-attacks from the Internet, and also that supports LE.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 23 Apr 2018 11:41:57 +0200 (CEST)

On Mon, 23 Apr 2018, [email-masked] wrote:

> [RG] RFC8100 is about deployment of end-to-end PHBs in the presence of 
> DSCP remarking. Do I get that correct, you propose end-to-end DSCPs to 
> support local or endpoint PHB deployment?

I feel like what my wife probably feels when she says "3G" and means 
"mobile data" and I correct her (I stopped doing that).

I have no idea what you just said if that's what I want.

> [RG] draft-ietf-tsvwg-rtcweb-qos is restricted to home-networks, 
> residential networks and local wireless networks. It says "There are 
> cases where these DSCP markings do not help, but, aside from possible 
> priority inversion for "less than best effort traffic" (see Section 5), 
> they seldom make things worse if packets are marked appropriately." And 
> later " WebRTC use of multiple DSCP values may encounter network 
> blocking of packets with certain DSCP values." Are you suggesting any 
> changes to that?

No. That text seems fine.

> [RG] Interestingly, DSCP 46 / EF is the one next to default which can 
> almost be linked to end to end standard PHB and DSCP - if sender and 
> receiver have a service agreement in place. That holds for Germany 
> (public telephony, L3 VPNs, business and wholesale products, wireless 
> and IP sections of wireline packet transport).

This is exactly why I proposed to send it into the DSCP 2-63 queue instead 
of being sent into BE or LE queues.

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 23 Apr 2018 11:49:46 +0200

Hi Mikael,

Am 19.04.2018 um 12:52 schrieb Mikael Abrahamsson:
> Ok, what should I call the following (all values decimal for the 6 DSCP
> CP bits)
> 

This is correctly stated, so I would always qualify bleaching with
the respective DSCPs that are bleached.

> For me there are two things an ISP does depending on DSCP values. It has
> a policy that (might) inspect CP values and change them. It also has a
> PHB depending on the CP values.
> 
> The simplest policy is "SET CP = 0 on all packets". This seems to be
> what you call bleaching.

Yep.

> If the domain previously just bleached everything and now wants to
> deploy support for LE (both allowing the CP to enter the ISP network and
> also creating different PHB for LE vs BE), what should it do regarding CS1?

> Saying "it depends" is not helpful. This is the Internet with billions
> of devices and users. What should be the default behaviour for
> non-managed devices run by normal people that they bought from the
> electronics store. I am not talking about managed enterprise networks
> etc. I am talking about the most common network out there, the "oh, I
> need Internet access to watch cat pictures on my own device and it's
> default out of box configured from DSCP point of view".

I think that Ruediger also stated that it depends on what was previously
the semantics of CS1 at that particular link. The problem is indeed at
app level, since there may be some apps using CS1 in LE semantics and
some probably in genuine CS semantics. So the choice is to map CS1->LE
or to bleach it unless you want to support genuine CS1. As a provider
it is your own risk and I think one could monitor the traffic volume.
If it is negligible, then using BE or elevated services for CS1 is no
problem, but otherwise this traffic may adversely affect other BE traffic.

> CP between 2 and 63. Not 0 and 1.

Ok, that is much more precise.

Regards
 Roland

---

From: <[email-masked]>
Date: Mon, 23 Apr 2018 11:23:55 +0000

Mikael,

and it's not easier for me to understand your discussion.

Let's focus on text related to drafts then.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Montag, 23. April 2018 11:42
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] residential broadband BCP PHB and CP treatment Re: CC/bleaching thoughts for draft-ietf-tsvwg-le-phb-04

> support local or endpoint PHB deployment?

I feel like what my wife probably feels when she says "3G" and means "mobile data" and I correct her (I stopped doing that).

> priority inversion for "less than best effort traffic" (see Section 
> 5), they seldom make things worse if packets are marked 
> appropriately." And later " WebRTC use of multiple DSCP values may 
> encounter network blocking of packets with certain DSCP values." Are 
> you suggesting any changes to that?

> and IP sections of wireline packet transport).

This is exactly why I proposed to send it into the DSCP 2-63 queue instead of being sent into BE or LE queues.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 23 Apr 2018 13:44:24 +0200 (CEST)

> and it's not easier for me to understand your discussion.

Let me try some schematics for a simplified network that's pertinent to 
what I want to achieve:

INTERNET<-1->PEERING ROUTER<-2->CORE ROUTER<-3->BNG<-4->HGW<-5->COMPUTER

1. This link connects the ISP to the Internet (peers, IXes, transit)
2. This link connects core/peering routers to each other.
3. This link connects core to BNG.
4. This link connects the residential access to HGW (DSL/PON/P2P eth/DOCSIS)
5. This link connects devices within the home (wifi/ethernet)

I want to document recommendations regarding PHB and CP (re-)mapping that 
can be done egress/ingress on 1 (peering router ingress/egress), 4 (BNG 
ingress/egress), 4(HGW ingress/egress) and 5 (HGW/hosts egress/egress, 
RFC8325) most likely based on IP level information such as DSCP.

In my experience it's very common that todays behaviour is to set CP=0 
ingress on 1 (peering router ingress) and 4 (BNG ingress) and PHB is all 
BE. This is what I want to change.

It can involve setting MPLS EXP/802.1p information based on DHCP CP (link 
2 and 3, potentially 4), but that's up to the individual ISP what they 
want to do (I am not interested in recommending values for that).

Does this help?

---

From: "Black, David" <[email-masked]>
Date: Tue, 24 Apr 2018 23:56:06 +0000

Mikael,

<WG Chair Hat= OFF>

> RFC8325) most likely based on IP level information such as DSCP.

The simplest (and special) case is Network Control traffic.  That should not be flowing from the COMPUTER to the operator's network, so the BNG has to bleach CS6 (and CS7) inbound on link 4, and the HGW ought to bleach those DSCPs inbound on link 5 if the HGW managed by the operator.  OTOH, BGP traffic using CS6 (and perhaps CS7) is to be expected on link 1 and bleaching CS6 when used for BGP traffic is a bad idea for obvious reasons ;-).

> In my experience it's very common that today's behaviour is to set CP=0

That was also a goal of RFC 8100, primarily for link 1 where the operator uses MPLS and aligns DSCP usage with the MPLS TC field ...

> want to do (I am not interested in recommending values for that).

It may not be that simple ... RFC 8100 was written for MPLS-based networks (where at least link 2 uses MPLS) where the operator aligns DSCP usage with the MPLS TC field.   One of the conclusions to be drawn from RFC 8100 is *in that operator's MPLS world*, link 1 is unlikely exhibit richness of DSCP usage approaching what is described in RFC 8325 or the rtcweb-qos draft.  This is because the number of DSCP values in use across MPLS links is limited when internal DSCP usage is aligned with the MPLS TC field.

If there are two different regimes for DSCP usage (link 1 vs. links 4 and 5), then the configuration concern that I think underlies this thread is how to map between them.  For RFC 8100, section 4 specifies traffic aggregates, including the DSCPs to use.   The mapping from the larger set of DSCPs that may be used on links 4 and 5 to the DSCPs in section 4 is then obtained by reading that section in concert with RFC 4594 ... and I won't disagree with the criticism that this is not easy for the uninitiated to figure out :-(.

It's also the case that LE/scavenger traffic isn't handled particularly well by RFC 8100 - this was raised as a concern while we were working on RFC 8100 (I'm an author), and at the time it was mostly deferred (RFC 8100 remarks it to BE at the interconnection).  I'll admit that one of my concerns at the time was that if we held up work in order to nail down all the considerations around LE/scavenger traffic, RFC 8100 would likely still not be published.

Thanks, --David

> Sent: Monday, April 23, 2018 7:44 AM
> INTERNET<-1->PEERING ROUTER<-2->CORE ROUTER<-3->BNG<-4->HGW<-5-
> >COMPUTER
>