Subject: Re: [tsvwg] draft diffserv-intercon: Handling of a scavenger class / CS1

From: <[email-masked]>
Date: Tue, 2 Jun 2015 11:14:59 +0200

Hi Mikael,

thanks for your proposal. Indeed, transport of LE(Scavenger) by a DSCP 000xx? at DiffServ-Intercon interfaces was the only suitable solution I imagined myself.

Brian quotes RFC3662 below. Now if an AF class is used to implement LE within a domain, that's still not resulting in a globally non ambiguous DSCP. This may justify a re-write to 000xx0 at a DiffServ-Intercon interface if the latter is enhanced by text at least reserving these DSCPs for LE. 

There should be no unknown DSCPs with AF like syntax, if DSCPs 000xx0 indicate LE at DiffServ-Intercon interfaces and unknown DSCPs face a rewrite of DSCP bits 0-2 to zero (bleaching). Then every unknown AF like DSCP will be made LE. 
I just note that then bleaching removes the necessity to rewrite the AF_DSCPs_used_for_LE at a sending interconnection interface.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg [mailto:[email-masked]] Im Auftrag von Mikael Abrahamsson
Gesendet: Freitag, 29. Mai 2015 23:11
An: Brian E Carpenter
Cc: [email-masked]
Betreff: Re: [tsvwg] draft diffserv-intercon: Handling of a scavenger class / CS1

On Sat, 30 May 2015, Brian E Carpenter wrote:

> Firstly, note that RFC 3662 is very intentionally not standards-track 
> and does not recommend a DSCP value. To me, this discussion seems to 
> mix up two orthogonal concepts:
>
> 1. What the PHB for a scavenger class actually is (or in Mikael's 
> suggestion to AQM, what the PHB set is, since mention of priority 
> implies several members of a PHB set, like AFxy).
>
> 2. What DSCP values happen to be used for this PHB set. Despite the 
> confusion introduced by RFC 3662, those values don't have to ones 
> already used for other purposes, especially not CSx. Actually the text 
> makes it clear that the DSCPs recommended for AFxy should be 
> considered:
>
>   If a CS PHB is used, note that this configuration will violate the
>   "SHOULD" of section 4.2.2.2 of RFC 2474 [RFC2474] since CS1 will have
>   a less timely forwarding than CS0.  An operator's goal of providing
>   an LE PDB is sufficient cause for violating the SHOULD.  If an AF PHB
>   is used, it must be configured and a DSCP assigned such that it does
>   not violate the "MUST" of paragraph three of section 2 of RFC 2597
>   [RFC2597] which provides for a "minimum amount of forwarding
>   resources".

I was only talking about what the 6 DSCP bits should be for this to ever start to work widely across the Internet. My suggestion is 000xx0 and the recommendation is to only bleach the first three bits at ISP interconnects instead of all 6 bits (which is the most commonly done at this time). This is backwards compatible with CS0/BE and it's also backwards compatible with default queuing behaviour when doing automatic IP precedence -> 802.1p and the recommended 4 queue ethernet prioritization (where CS1 and
CS2 is lower than CS0 and CS3).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Black, David" <[email-masked]>
Date: Tue, 16 Jun 2015 22:21:34 +0000

<WG chair hat OFF>

As the "designated victim" at the front of the room for this draft in Dallas
(and one of the draft authors), I understood the request to be for a single
scavenger PHB and DSCP. I concur that 3 AF-like drop probabilities could be
additional nice-to-have functionality, although I think what's essential here
is scavenging of resources otherwise unused for best-effort (or better)
forwarding, for which a single PHB and recommended DSCP would suffice.

We do have a PHB definition in RFC 3662 (LE PHB).  Unfortunately, RFC 3662's
selection of CS1 as the default DSCP has turned out to be a poor choice in 
20/20 hindsight- this text that Brian quoted from RFC 3662 wound up
foreshadowing the problems to come:

      If a CS PHB is used, note that this configuration will violate the
      "SHOULD" of section 4.2.2.2 of RFC 2474 [RFC2474] since CS1 will have
      a less timely forwarding than CS0.  An operator's goal of providing
      an LE PDB is sufficient cause for violating the SHOULD.

See the discussion of the LE PHB and CS1 in the DART draft (at the RFC Editor)
for a summary of the unfortunate state of the world in this area:

	https://tools.ietf.org/html/draft-ietf-dart-dscp-rtp-10

I like Mikael's suggestion of 000xx0 as the place to look for a recommended
DSCP to replace CS1, as that removes the obvious risks of priority inversion
for equipment that only pays attention to the most significant 3 bits of
the DSCP field.

For the Diffserv Intercon draft, I'd prefer to start with just one of those
codepoints, e.g., 000010, define it for the LE PHB for network interconnection
and suggest that the DSCP be preserved across each network domain (here,
"preserved" means to use that DSCP at domain egress, not necessarily within
the domain).  This would be in addition to the current traffic classes in
the draft, not a replacement.

Beyond that, I'd support a (hopefully short) draft that
	- updates RFC 3662 to change the recommended DSCP for the LE PHB
		from CS1 to 000010,
	- recognizes and allows for continued use of CS1 where deployed, and
	- updates other RFCs (e.g., 4592, 5127) accordingly as needed.
I don't think the Diffserv Intercon draft is a good vehicle for doing that
- a short free-standing draft would be better (IMHO), and that draft would
likely be referenced by the Diffserv Intercon draft if scavenger class text
is added to that draft.

</WG chair hat OFF>

Thanks,
--David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Tuesday, June 02, 2015 5:15 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] draft diffserv-intercon: Handling of a scavenger class /
> CS1
> 
> Hi Mikael,
> 
> thanks for your proposal. Indeed, transport of LE(Scavenger) by a DSCP 000xx?
> at DiffServ-Intercon interfaces was the only suitable solution I imagined
> myself.
> 
> Brian quotes RFC3662 below. Now if an AF class is used to implement LE within
> a domain, that's still not resulting in a globally non ambiguous DSCP. This
> may justify a re-write to 000xx0 at a DiffServ-Intercon interface if the
> latter is enhanced by text at least reserving these DSCPs for LE.
> 
> There should be no unknown DSCPs with AF like syntax, if DSCPs 000xx0 indicate
> LE at DiffServ-Intercon interfaces and unknown DSCPs face a rewrite of DSCP
> bits 0-2 to zero (bleaching). Then every unknown AF like DSCP will be made LE.
> I just note that then bleaching removes the necessity to rewrite the
> AF_DSCPs_used_for_LE at a sending interconnection interface.
> 
> Regards,
> 
> Ruediger
> 
> 
> I was only talking about what the 6 DSCP bits should be for this to ever start
> to work widely across the Internet. My suggestion is 000xx0 and the
> recommendation is to only bleach the first three bits at ISP interconnects
> instead of all 6 bits (which is the most commonly done at this time). This is
> backwards compatible with CS0/BE and it's also backwards compatible with
> default queuing behaviour when doing automatic IP precedence -> 802.1p and the
> recommended 4 queue ethernet prioritization (where CS1 and

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 17 Jun 2015 12:55:36 +1200

Hi David,
> likely be referenced by the Diffserv Intercon draft if scavenger class text

I think that is very reasonable technically, but there's a slight process
complication. 000010 is in the RFC2474 'standards action' range, so the
short draft would have to be standards track with a downref to RFC3662,
as well as formally updating it.

    Brian

---

From: [email-masked]
Date: Wed, 17 Jun 2015 08:11:00 +0100

See below.

> as well as formally updating it.
I'd be happy to contribute to this work.

As co-chair: I think the short separate standards-track draft would then
be the appropriate thing to do. This could be done quickly ***IF*** we get
a firm consensus  from the community to do this.

Gorry

>>> -----UrsprÃ¼ngliche Nachricht-----
>>> Betreff: Re: [tsvwg] draft diffserv-intercon: Handling of a scavenger
>>> class /
>>> CS1
>>>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 26 Jun 2015 08:37:15 +0200 (CEST)

On Wed, 17 Jun 2015, [email-masked] wrote:

> be the appropriate thing to do. This could be done quickly ***IF*** we 
> get a firm consensus from the community to do this.

If you want to reach out to the "community", I would suggest to for 
instance reach out to the nanog (https://www.nanog.org/list) mailing list 
and ask for feedback there. You'd be more likely to find people who will 
actually implement this there, than on this mailing list. You'd get a much 
wider audience who will tell you if this is deployable in the real world 
with real world equipment and real world configuration, and asking real 
world operators if this is something they could imagine implementing.

Those are the people who will decide what DSCP bit bleaching is done at 
peering borders and customer connections for instance, which is key to 
getting this deployed.

It would also be interesting to hear from equipment vendors, especially 
ones that make BNGs, which kind of bleaching functionality is available in 
existing equipment. Is it possible to just bleach precedence bits on most 
platforms? I'd imagine it is, but it would be interesting to know.

---

From: <[email-masked]>
Date: Fri, 26 Jun 2015 09:22:31 +0200

Hi Mikael,

I've asked my colleagues visiting RIPE and NANOG meetings for Diffserv Interconnection related contacts, but feedback was limited. I guess there's some truth in what you say - developers and more so operational staff hardly participates in IETF work.

If you are aware of DiffServ Interconnection related agreements of NANOG please let me know. I think there's nothing discussed by RIPE.

You wrote:

   It would also be interesting to hear from equipment vendors, especially 
   ones that make BNGs, which kind of bleaching functionality is available 
   in existing equipment. Is it possible to just bleach precedence bits 
   on most platforms? I'd imagine it is, but it would be interesting to know.

Bleaching of "Ip precedence bits" is not within scope of Diffserv Intercon and it will stay out of scope (it is an informational draft and it cannot and is not supposed to change standards).

Carrier representatives interested in a standard allowing to remark, say, ranges of DSCPs (bleaching by precedence is bad wording) should collectively draft a suitable document. I guess, if at all, this is only picked up by standardization if a significant number of provider representatives express interest. If you know some more provider representatives willing to co-author such a draft, please let me know. 

I've had some text related to the issue in very early versions of the draft. 

Regards,

Ruediger

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 26 Jun 2015 10:33:50 +0200 (CEST)

On Fri, 26 Jun 2015, [email-masked] wrote:

> I've asked my colleagues visiting RIPE and NANOG meetings for Diffserv 
> Interconnection related contacts, but feedback was limited. I guess 
> there's some truth in what you say - developers and more so operational 
> staff hardly participates in IETF work.

I don't understand what "Diffserv Interconnection related contacts" would 
be. This would most likely be handled by peering managers and IP 
network architects that do the rest of the network design.

> If you are aware of DiffServ Interconnection related agreements of NANOG 
> please let me know. I think there's nothing discussed by RIPE.

In the operational community things are done largely by interpersonal 
connections and discussions on mailing lists that then create de-facto 
ways of doing things.

> Bleaching of "Ip precedence bits" is not within scope of Diffserv 
> Intercon and it will stay out of scope (it is an informational draft and 
> it cannot and is not supposed to change standards).

Right now most ISPs will bleach all 6 bits for Internet traffic. My 
proposal was to come up with a solution that allowed them to bleach the CS 
bits.

> Carrier representatives interested in a standard allowing to remark, 
> say, ranges of DSCPs (bleaching by precedence is bad wording) should 
> collectively draft a suitable document. I guess, if at all, this is only 
> picked up by standardization if a significant number of provider 
> representatives express interest. If you know some more provider 
> representatives willing to co-author such a draft, please let me know.

I have already suggested what I think would be achievable in real life. I 
am willing to bring this to nanog and ask for opinions from the 
operational community.

> I've had some text related to the issue in very early versions of the draft.

I propose that what I want to do isn't applicable to your draft, I just 
realised I had jumped into the discussion without reading it. I now have 
quickly scanned through it, and your text isn't applicable to Internet 
peering at all.

So what I want to achieve would need a new document. I have no idea how it 
applies to all the other documents out there because there are just too 
many for someone not focusing on this area, to overlook. I have been 
architecting Internet backbones for 15 years.

I would like to see a document that applies to Internet traffic only, 
scavenger class and BE only, that basically proposes to bleach CS bits and 
leave the rest of the bits intact, and then suggests how to configure 
queuing behavior for Internet access to take this into account. It should 
be an operational document which includes deployment cases and how to 
configure this using commonly used equipment.

I think the disconnect here is that in the IETF DSCP and the rest is 
considered transport, but by most operators, DSCP and queuing behaviour is 
done by IP engineers and IP architects. These people go to v6ops, 
potentially to 6man, routing protocol WG:s and similar IP control and 
forwarding groups, they don't really attend what IETF calls transport. I 
also get the feeling that people in here don't really attend NANOG or 
other places where the people who actually forward packets and configure 
queues attend. So we have two groups who never interact.

So there is really a need for TSV-OPS or something, that actually talks 
about how to implement what's in the DSCP related documents, and the IETF 
needs to somehow try to attract operational people in there. I have non 
idea how to achieve this.

---

From: <[email-masked]>
Date: Fri, 26 Jun 2015 12:18:45 +0200

Hi Mikael,

if you attend the Prague IETF, let's meet there and discuss ways forward.

I note that DiffServ-Intercon doesn't meet your operational requirements. It is closer to reach mine (in Germany Ethernet and IP QoS interconnections are discussed with the regulator).
 
Starting new work with use cases certainly is a good idea. I share some of your interests, but I also note that we have differing views on others. So I'm happy to provide support where our interests overlap. That's bleaching and rewriting of DSCP ranges, both at interconnection.

IETF welcomes provider input also in TSVWG. I'm happy for every provider representative joining me in discussing operation related DiffServ issues in TSVWG. Once we are too many, a TSV OPS related WG may be helpful.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Freitag, 26. Juni 2015 10:34
An: Geib, Rüdiger
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] draft diffserv-intercon: Handling of a scavenger class / CS1

On Fri, 26 Jun 2015, [email-masked] wrote:

> there's some truth in what you say - developers and more so 
> operational staff hardly participates in IETF work.

I don't understand what "Diffserv Interconnection related contacts" would be. This would most likely be handled by peering managers and IP network architects that do the rest of the network design.

> If you are aware of DiffServ Interconnection related agreements of 
> NANOG please let me know. I think there's nothing discussed by RIPE.

In the operational community things are done largely by interpersonal connections and discussions on mailing lists that then create de-facto ways of doing things.

> Intercon and it will stay out of scope (it is an informational draft 
> and it cannot and is not supposed to change standards).

Right now most ISPs will bleach all 6 bits for Internet traffic. My proposal was to come up with a solution that allowed them to bleach the CS bits.

> collectively draft a suitable document. I guess, if at all, this is 
> only picked up by standardization if a significant number of provider 

I have already suggested what I think would be achievable in real life. I am willing to bring this to nanog and ask for opinions from the operational community.

> I've had some text related to the issue in very early versions of the draft.

I propose that what I want to do isn't applicable to your draft, I just realised I had jumped into the discussion without reading it. I now have quickly scanned through it, and your text isn't applicable to Internet peering at all.

So what I want to achieve would need a new document. I have no idea how it applies to all the other documents out there because there are just too many for someone not focusing on this area, to overlook. I have been architecting Internet backbones for 15 years.

I would like to see a document that applies to Internet traffic only, scavenger class and BE only, that basically proposes to bleach CS bits and leave the rest of the bits intact, and then suggests how to configure queuing behavior for Internet access to take this into account. It should be an operational document which includes deployment cases and how to configure this using commonly used equipment.

I think the disconnect here is that in the IETF DSCP and the rest is considered transport, but by most operators, DSCP and queuing behaviour is done by IP engineers and IP architects. These people go to v6ops, potentially to 6man, routing protocol WG:s and similar IP control and forwarding groups, they don't really attend what IETF calls transport. I also get the feeling that people in here don't really attend NANOG or other places where the people who actually forward packets and configure queues attend. So we have two groups who never interact.

So there is really a need for TSV-OPS or something, that actually talks about how to implement what's in the DSCP related documents, and the IETF needs to somehow try to attract operational people in there. I have non idea how to achieve this.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 26 Jun 2015 14:45:23 +0200 (CEST)

On Fri, 26 Jun 2015, [email-masked] wrote:

> IETF welcomes provider input also in TSVWG. I'm happy for every provider 
> representative joining me in discussing operation related DiffServ 
> issues in TSVWG. Once we are too many, a TSV OPS related WG may be 
> helpful.

Most operators have no idea this work is going on here. You can say you're 
welcoming participation all you want, but if you actually want 
participation then you have to reach out.

It also has to be relevant and understandable. Your document, whilst I'm 
sure it's perfectly correct and makes a lot of sense to matter experts, is 
TL;DR; (Too Long, Didn't Read), for most people who just want their 
networks to work and customers not complain.

So yes, let's meet up in Prague and discuss this. I think what we need is 
a different document than diffserv-intercon because most in there just 
isn't applicable to Internet peering interconnects.

---

From: <[email-masked]>
Date: Mon, 29 Jun 2015 09:00:46 +0200

Mikael,

it took a while to make progress on DiffServ Intercon. You say 

     So yes, let's meet up in Prague and discuss this. I think what we 
     need is a different document than diffserv-intercon because most 
     in there just isn't applicable to Internet peering interconnects.

Do you say:

- there's no need for interconnection based on QoS classes - just forward 
  limited DSCP information to an access and that's fine? Or is your 
  statement no DSCP or IP-Precedence based QoS differentiation in the IP 
  backbone (and only default forwarding there), limit QoS to the access?
- you'd like to support QoS interconnection, but set up individual DSCP 
  based PHB mapping tables per interconnection? In that case, no new 
  standard is required. 
- You just need bleaching to Default - that's a separate issue which has been 
  removed from DiffServ-Intercon as part of WG consent. It is a pre-requisite 
  of DiffServ-Intercon, but not an issue handled by it.
- You look for a rewrite of ranges of DSCPs ("treatment of IP precedence")
  - this is out of scope, as it is a standards track issue touching the
  fundaments of DiffServ. DiffServ Intercon also works well if IP-Precedence 
  rewrite occurs. 

General claims like "isn't applicable to Internet-peering" aren't helpful, if you don't tell us what is applicable to internet peering. DiffServ-Intercon has been applicable to the QoS interconnection agreements I've been involved in so far. 

If you are able to provide something which isn't TL;DR, I'd appreciate your submission before cut-off. I'd also appreciate your indication what needs to be removed from draft diffserv-intercon to increase op-folk attention.

I repeat that I've talked with my RIPE/NANOG colleagues to get feedback - they came back only with one or two names with whom to discuss QoS interconnection. And I never got any useful response (positive or negative) from those.

Regards,

Ruediger

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 29 Jun 2015 09:37:47 +0200 (CEST)

On Mon, 29 Jun 2015, [email-masked] wrote:

> I repeat that I've talked with my RIPE/NANOG colleagues to get feedback 
> - they came back only with one or two names with whom to discuss QoS 
> interconnection. And I never got any useful response (positive or 
> negative) from those.

This is the problem, that's not how to do this, the way to do this is to 
announce an idea on the nanog mailing list and see who might be 
interested.

As far as I read your document, it has nothing in it about how to 
incrementally get an Internet-wide scavenger class operationally deployed 
over time. To me it reads more like a document for business VPN 
Carriers-Carrier services style interconnect/interworking. It has no 
operational advice at all in it on how to deal with DDoS, how to configure 
access equipment, how to set up queuing strategies on different kinds of 
equipment, listing what equipment might do what, etc.

I suggested how to actually make this incrementally and widely deployable 
across the world, and that is to aim for a 000xxx style scavenger class. 
This seems to not work for some reason that is unclear to me, your 
document doesn't seem to be operational at all, it doesn't propose 
something that is operationally deployable on the wider Internet, so 
that's why I think we would need another document for this purpose. If 
that is not a goal we can agree on, then there is no need to write one.

So just to sum up: I'm sure your document is fine for what it intends to 
do, it just doesn't answer any of the concerns an IP network 
engineer/architect might have about implementing this on an Internet 
peering link where there is little or no control over what traffic will 
pass or what this traffic might do to the rest of their network. It 
suggests a way to do something with no operational analysis or risk 
mitigation at all.

The first thing that will pop into any good IP architects head nowadays 
will be "oh, what happens to my network when I get 200 gigabits/s of 64 
byte packets marked EF from that 5 million strong DDoS-drone network all 
across the world, aimed for my customer base?"

The answer to that is "I'll bleach DSCP it because that's the only way to 
handle it". So in order to get a scavenger class actually deployable, you 
have to come up with something that means they can relax the bleaching a 
little bit by some operators, but their infrastructure would still treat 
the traffic the same. You also have to take into account that a lot of 
core IP network equipment can't do DSCP mapping very well. This kind of 
functionality makes equipment more expensive and thus less likely to exist 
widely.

---

From: <[email-masked]>
Date: Mon, 29 Jun 2015 10:50:55 +0200

Hi Mikael,

ok, understood, I'm relaxed.

The relation of Diffserv intercon to scavenger is

- if there is a scavenger class, how should Diffserv intercon handle it 
  (will add text about a 000 xx0 DSCP)? Otherwise Diffserv Intercon 
  says, bleaching may occur for any unexpected DSCP.

- work on a scavenger class standard needs to be done in a 
  separate draft. That's WG consensus and Gorry Fairhurst said during 
  the last meeting:
  "Scavenger support document been discussed before, if there is 
  renewed interest, then we should take it up again. 
  As an individual contributor, I would be interested. 
  If you are also interested in this topic, please talk to the 
  WG chairs."

- operational issues may be relevant for a bleaching discussion. 
  A bleaching standard is out of scope of Diffserv Intercon (thats 
  WG consensus). I think to recall interest in work on bleaching 
  by others too.

Then I'm happy to meet you in Prague. 

Regards,

Ruediger 

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson [mailto:[email-masked]] 
Gesendet: Montag, 29. Juni 2015 09:38
An: Geib, Rüdiger
Cc: [email-masked]
Betreff: Re: AW: AW: AW: [tsvwg] draft diffserv-intercon: Handling of a scavenger class / CS1

> interconnection. And I never got any useful response (positive or

This is the problem, that's not how to do this, the way to do this is to announce an idea on the nanog mailing list and see who might be interested.

As far as I read your document, it has nothing in it about how to incrementally get an Internet-wide scavenger class operationally deployed over time. To me it reads more like a document for business VPN Carriers-Carrier services style interconnect/interworking. It has no operational advice at all in it on how to deal with DDoS, how to configure access equipment, how to set up queuing strategies on different kinds of equipment, listing what equipment might do what, etc.

I suggested how to actually make this incrementally and widely deployable across the world, and that is to aim for a 000xxx style scavenger class. 
This seems to not work for some reason that is unclear to me, your document doesn't seem to be operational at all, it doesn't propose something that is operationally deployable on the wider Internet, so that's why I think we would need another document for this purpose. If that is not a goal we can agree on, then there is no need to write one.

So just to sum up: I'm sure your document is fine for what it intends to do, it just doesn't answer any of the concerns an IP network engineer/architect might have about implementing this on an Internet peering link where there is little or no control over what traffic will pass or what this traffic might do to the rest of their network. It suggests a way to do something with no operational analysis or risk mitigation at all.

The first thing that will pop into any good IP architects head nowadays will be "oh, what happens to my network when I get 200 gigabits/s of 64 byte packets marked EF from that 5 million strong DDoS-drone network all across the world, aimed for my customer base?"

The answer to that is "I'll bleach DSCP it because that's the only way to handle it". So in order to get a scavenger class actually deployable, you have to come up with something that means they can relax the bleaching a little bit by some operators, but their infrastructure would still treat the traffic the same. You also have to take into account that a lot of core IP network equipment can't do DSCP mapping very well. This kind of functionality makes equipment more expensive and thus less likely to exist widely.