Subject: Re: [tsvwg] Status of ECN encapsulation drafts (i.e., stuck)

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Mar 2020 18:47:17 +0000

David,

I admit to curling up into a little ball and trying to ignore this 
controversy when it arose.
Let me try to sort this out now, for both ecn-encap-guidelines and 
rfc6040update-shim.

Back in Sep '19 (quoted at the end) you asked me not to use 
rfc64040update-shim to update RFC3168's fragmentation behaviour, even if 
it's the "right thing" to do, given I was saying that there were 
problems with the RFC3168 approach.

Background: Neither RFC3168 nor RFC6040 covered fragmentation & 
reassembly during encap and decap. So Joe Touch suggested 
rfc6040update-shim should fix that omission. Seems reasonable enough. 
However, it doesn't seem right to fix an omission by the stop-gap of:
1. requiring the approach in RFC3168 that we know is potentially 
problematic.
2. then planning to correct what we write, by updating it in a later RFC.

Let's call that approach (A). I don't like that at all. What if step #2 
never happens?
Fortunately, that's not the only way out of this. I can think of three 
other ways:

    B) The compromise text I've drafted below, which states the high
    level intent of a good mechanism as a SHOULD, and gives an example
    of how to do it. Then also allows the RFC3168 mechanism as a "MAY".
    C) Say nothing about fragmentation and reassembly in
    rfc64040update-shim or ecn-encap-guidelines. Then use a later RFC to
    update them both (stds track and BCP) with a considered 'correct'
    approach. ecn-encap-guidelines would still say include what it has
    always said about re-framing (which is a similar but different subject).
    D) Convince ourselves that fragmentation and reassembly during encap
    and decap is allowed to be different from fragmentation and
    reassembly without encapsulation.

Last night, I took approach (B), but with too little time left to 
discuss it on the list. I scrubbed the offending paras from 
rfc6040update-shim and replaced them with those below (also at 
https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#section-5 
).

Thinking about it further since last night, I'm now inclining towards 
approach (C).

    5. ECN Propagation and Fragmentation/Reassembly

    The following requirements updateRFC6040  <https://tools.ietf.org/html/rfc6040>, which omitted handling of
    the ECN field during fragmentation or reassembly.  These changes
    might alter how many ECN-marked packets are propagated by a tunnel
    that fragments packets, but this would not raise any backward
    compatibility issues:

    If a tunnel ingress fragments a packet, it MUST set the outer ECN
    field of all the fragments to the same value as it would have set if
    it had not fragmented the packet.

    During reassembly of outer fragments [I-D.ietf-intarea-tunnels  <https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#ref-I-D.ietf-intarea-tunnels>], if
    the ECN fields of the outer headers being reassembled into a single
    packet consist of a mixture of Not-ECT and other ECN codepoints, the
    packet MUST be discarded.

    As a tunnel egress reassembles sets of outer fragments
    [I-D.ietf-intarea-tunnels  <https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#ref-I-D.ietf-intarea-tunnels>] into packets, as long as no fragment
    carries the Not-ECT codepoint, it SHOULD propagate CE markings such
    that the proportion of reassembled packets output with CE markings is
    broadly the same as the proportion of fragments arriving with CE
    markings.

    The above statement describes the approximate desired outcome, not
    the specific mechanism.  A simple to achieve this outcome would be to
    leave a CE-mark on a reassembled packet if the head fragment is CE-
    marked, irrespective of the markings on the other fragments.
    Nonetheless, "SHOULD" is used in the above requirement to allow
    similar perhaps more efficient approaches that result in
    approximately the same outcome.

    InRFC 3168  <https://tools.ietf.org/html/rfc3168>  the approach to propagating CE markings during fragment
    reassembly required that a reassembled packet has to be be CE-marked
    if any of its fragments is CE-marked.  This "logical OR" approach to
    CE marking during reassembly was intended to ensure that no
    individual CE marking is ever lost.  However, an unintended
    consequence is that the proportion of packets with CE markings
    increases.  For instance, with the logical OR approach, once a
    sequence of packets each consisting of 2 fragments, has been
    reassembled, the fraction of packets that are CE-marked roughly
    doubles (because the number of marks remains roughly the same, but
    the number of packets halves).

    This specification does not rule out the logical OR approach ofRFC  <https://tools.ietf.org/html/rfc3168>
    3168  <https://tools.ietf.org/html/rfc3168>.  So a tunnel egress MAY CE-mark a reassembled packet if any of
    the fragments are CE-marked (and none are Not-ECT).  However, this
    approach could result in reduced link utilization, or bias against
    flows that are fragmented relative to those that are not.

Regards

Bob

On 15/09/2019 22:07, Black, David wrote:
> This email concerns draft-ietf-tsvwg-ecn-encap-guidelines and 
> draft-ietf-tsvwg-rfc6040update-shim, which are being handled together 
> for WG Last Call and RFC publication, and is posted in my role as 
> shepherd and responsible WG chair for these drafts.The current 
> situation is that both drafts are stuck due to a problem with the 
> fragementation text added to the rfc6040update-shim draft.   Section 5 
> on ECN Propagation and Fragmentation/Reassembly was added to that 
> draft in response to a WGLC comment, and it appears to have gone too 
> far in the direction of trying to do the proverbial “right thing”.
>
> The core of the problem is in these two paragraphs in Section 5 of 
> that draft 
> (https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-09#section-5):
>
>     As a tunnel egress reassembles sets of outer fragments
>     [I-D.ietf-intarea-tunnels] into packets, it SHOULD propagate CE
>     markings on the basis that a congestion indication on a packet
>     applies to all the octets in the packet.  On average, a tunnel egress
>     SHOULD approximately preserve the number of CE-marked and ECT(1)-
>     marked octets arriving and leaving (counting the size of inner
>     headers, but not encapsulating headers that are being stripped).
>     This process proceeds irrespective of the addresses on the inner
>     headers.
>
>     Even if only enough incoming CE-marked octets have arrived for part
>     of the departing packet, the next departing packet SHOULD be
>     immediately CE-marked.  This ensures that CE-markings are propagated
>     immediately, rather than held back waiting for more incoming CE-
>     marked octets.  Once there are no outstanding CE-marked octets, if
>     only enough incoming ECT(1)-marked octets have arrived for part of
>     the departing packet, the next departing packet SHOULD be immediately
>     marked ECT(1).
>
> Much as that may be the proverbial “right thing” to do, particularly 
> with the benefit of 20/20 hindsight, that text is inconsistent with 
> the following text from Section 5.3 of RFC 3168 
> (https://tools.ietf.org/html/rfc3168#section-5.3), as Markku Kojo has 
> pointed out:
>
>     ECN-capable packets MAY have the DF (Don't Fragment) bit set.
>     Reassembly of a fragmented packet MUST NOT lose indications of
>     congestion.  In other words, if any fragment of an IP packet to be
>     reassembled has the CE codepoint set, then one of two actions MUST be
>     taken:
>        * Set the CE codepoint on the reassembled packet.  However, this
>          MUST NOT occur if any of the other fragments contributing to
>          this reassembly carries the Not-ECT codepoint.
>        * The packet is dropped, instead of being reassembled, for any
>          other reason.
>     If both actions are applicable, either MAY be chosen.  Reassembly of
>     a fragmented packet MUST NOT change the ECN codepoint when all of the
>     fragments carry the same codepoint.
>
> The 6040update-shim draft is intended to update RFC 6040, and a number 
> of the tunnel protocol drafts, but it is not intended to update RFC 
> 3168, and hence the above new text (albeit well-intentioned) is a 
> showstopper. Changing ECN fragmentation behavior should be done in a 
> separate draft.
>
> Bob (as draft editor) – do you want to propose some new text to the 
> list, possibly after private email discussion with Marco and me to 
> figure out what it needs to say?
>
> Thanks, --David
>
> ----------------------------------------------------------------
>
> David L. Black, Senior Distinguished Engineer
>
> Dell EMC, 176 South St., Hopkinton, MA 01748
>
> +1 (774) 350-9323 *New *   Mobile: +1 (978) 394-7754
>
> [email-masked] <mailto:[email-masked]>
>
> ----------------------------------------------------------------
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Tue, 10 Mar 2020 20:35:08 +0000

Bob,

As draft shepherd, let me suggest an alternate way to think about fragmentation and reassembly that may result in simpler text.

The current situation arises in large part from viewing fragmentation and reassembly as being part of tunnel encapsulation and tunnel decapsulation respectively.

Instead, I suggest that we view all four of these as separate but related processes, specifically:

  *   A tunnel ingress encapsulates the inner packet and then fragments the resulting (encapsulated) outer packet if necessary for transmission.
     *   RFC 6040 and this document specify the ECN requirements for encapsulation.  RFC 3168 specifies ECN requirements for fragmentation.
  *   A tunnel egress reassembles the outer packet and then decapsulates the resulting (still encapsulated) outer packet to produce the inner packet.
     *   RFC 3168 specifies ECN requirements for reassembly.  RFC 6040 and this document specify the ECN requirements for decapsulation.

Optimized implementations may of course mix encapsulation and decapsulation processing with fragmentation and reassembly processing, respectively, but the results are required to be the same as if the above orders of processing were followed and that processing adhered to the requirements listed above.

I think the result will be clearer, and will also make it obvious that nothing new is being required, especially if "RFC 6040 and this document specify" can be changed to "RFC 6040 specifies" in both sub-bullets above.

What do you think of this approach?

Thanks, --David

From: Bob Briscoe <[email-masked]>
Sent: Tuesday, March 10, 2020 2:47 PM
To: Black, David; [email-masked]
Subject: Re: [tsvwg] Status of ECN encapsulation drafts (i.e., stuck)

[EXTERNAL EMAIL]
David,

I admit to curling up into a little ball and trying to ignore this controversy when it arose.
Let me try to sort this out now, for both ecn-encap-guidelines and rfc6040update-shim.

Back in Sep '19 (quoted at the end) you asked me not to use rfc64040update-shim to update RFC3168's fragmentation behaviour, even if it's the "right thing" to do, given I was saying that there were problems with the RFC3168 approach.

Background: Neither RFC3168 nor RFC6040 covered fragmentation & reassembly during encap and decap. So Joe Touch suggested rfc6040update-shim should fix that omission. Seems reasonable enough. However, it doesn't seem right to fix an omission by the stop-gap of:
1. requiring the approach in RFC3168 that we know is potentially problematic.
2. then planning to correct what we write, by updating it in a later RFC.

Let's call that approach (A). I don't like that at all. What if step #2 never happens?
Fortunately, that's not the only way out of this. I can think of three other ways:
B) The compromise text I've drafted below, which states the high level intent of a good mechanism as a SHOULD, and gives an example of how to do it. Then also allows the RFC3168 mechanism as a "MAY".
C) Say nothing about fragmentation and reassembly in rfc64040update-shim or ecn-encap-guidelines. Then use a later RFC to update them both (stds track and BCP) with a considered 'correct' approach. ecn-encap-guidelines would still say include what it has always said about re-framing (which is a similar but different subject).
D) Convince ourselves that fragmentation and reassembly during encap and decap is allowed to be different from fragmentation and reassembly without encapsulation.

Last night, I took approach (B), but with too little time left to discuss it on the list. I scrubbed the offending paras from rfc6040update-shim and replaced them with those below (also at https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#section-5 ).

Thinking about it further since last night, I'm now inclining towards approach (C).
5.  ECN Propagation and Fragmentation/Reassembly

   The following requirements update RFC6040<https://tools.ietf.org/html/rfc6040>, which omitted handling of

   compatibility issues:

   During reassembly of outer fragments [I-D.ietf-intarea-tunnels<https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#ref-I-D.ietf-intarea-tunnels>], if

   [I-D.ietf-intarea-tunnels<https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-10#ref-I-D.ietf-intarea-tunnels>] into packets, as long as no fragment

   markings.

   In RFC 3168<https://tools.ietf.org/html/rfc3168> the approach to propagating CE markings during fragment

   This specification does not rule out the logical OR approach of RFC<https://tools.ietf.org/html/rfc3168>

   3168<https://tools.ietf.org/html/rfc3168>.  So a tunnel egress MAY CE-mark a reassembled packet if any of

Regards

Bob
On 15/09/2019 22:07, Black, David wrote:
This email concerns draft-ietf-tsvwg-ecn-encap-guidelines and draft-ietf-tsvwg-rfc6040update-shim, which are being handled together for WG Last Call and RFC publication, and is posted in my role as shepherd and responsible WG chair for these drafts.The current situation is that both drafts are stuck due to a problem with the fragementation text added to the rfc6040update-shim draft.   Section 5 on ECN Propagation and Fragmentation/Reassembly was added to that draft in response to a WGLC comment, and it appears to have gone too far in the direction of trying to do the proverbial "right thing".

The core of the problem is in these two paragraphs in Section 5 of that draft (https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-09#section-5):

   headers.

   marked ECT(1).

Much as that may be the proverbial "right thing" to do, particularly with the benefit of 20/20 hindsight, that text is inconsistent with the following text from Section 5.3 of RFC 3168 (https://tools.ietf.org/html/rfc3168#section-5.3), as Markku Kojo has pointed out:

   taken:

        other reason.

The 6040update-shim draft is intended to update RFC 6040, and a number of the tunnel protocol drafts, but it is not intended to update RFC 3168, and hence the above new text (albeit well-intentioned) is a showstopper.   Changing ECN fragmentation behavior should be done in a separate draft.

Bob (as draft editor) - do you want to propose some new text to the list, possibly after private email discussion with Marco and me to figure out what it needs to say?

Thanks, --David
----------------------------------------------------------------
David L. Black, Senior Distinguished Engineer
Dell EMC, 176 South St., Hopkinton, MA  01748
+1 (774) 350-9323 New    Mobile: +1 (978) 394-7754
[email-masked]<mailto:[email-masked]>
----------------------------------------------------------------

--

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 10 Mar 2020 22:40:34 +0200

> On 10 Mar, 2020, at 10:35 pm, Black, David <[email-masked]> wrote:
> 
> 	• A tunnel ingress encapsulates the inner packet and then fragments the resulting (encapsulated) outer packet if necessary for transmission.
> 		• RFC 6040 and this document specify the ECN requirements for encapsulation.  RFC 3168 specifies ECN requirements for fragmentation.
> 	• A tunnel egress reassembles the outer packet and then decapsulates the resulting (still encapsulated) outer packet to produce the inner packet.
> 		• RFC 3168 specifies ECN requirements for reassembly.  RFC 6040 and this document specify the ECN requirements for decapsulation.
>  
> I think the result will be clearer, and will also make it obvious that nothing new is being required, especially if “RFC 6040 and this document specify” can be changed to “RFC 6040 specifies” in both sub-bullets above.
>  

It seems like a perfectly reasonable approach to me.  We can then focus on what the two existing documents say about their respective parts of the process, and whether they remain compatible with high-fidelity congestion control (and other modern circumstances).

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 11 Mar 2020 01:02:20 +0200

> On 10 Mar, 2020, at 8:47 pm, Bob Briscoe <[email-masked]> wrote:
> 
>   This specification does not rule out the logical OR approach of RFC3168.
> .  So a tunnel egress MAY CE-mark a reassembled packet if any of

As I have stated in the past on this subject, I disagree with this characterisation.  It misconstrues the meaning of CE and the definition of steady state involving it, both for conventional congestion control and for the high-fidelity version.  It also leads to a more complex reassembly implementation being "preferred" than is necessary, which may deter implementors.

For conventional congestion control, the steady state is defined in terms of the time interval (during which cwnd growth occurs) between RTTs containing at least one CE mark (and/or packet loss).  It is necessary for a single CE mark to be delivered promptly, ie. attached to the same packet as the marked fragment belonged to, in order to minimise growth overshoot and keep the control loop properly closed.  The control loop is not sensitive to the "proportion of bytes marked", only that a mark was encountered at a particular time.  It is well known, moreover, that adding delay to a control loop destabilises it - but this is exactly what an attempt to maintain the proportion of marked bytes would do.

For DCTCP-style congestion control, the steady state is defined in terms of the number of CE marks received per RTT.  Fragmentation does not change the RTT, only the number of packets passing a middlebox located on the tunnel path.  It would be reasonable to design an AQM expecting DCTCP traffic so that it produces exactly the correct number of CE marks per RTT at a particular queue depth.  But if the number of marks is effectively halved by a reassembly process that attempts to preserve the number of marked bytes, that queue will continue to grow past that designed ideal point.  We can therefore conclude that DCTCP is also insensitive to the proportion of marked bytes, and this is not a property worth preserving; rather, the total number of marked packets should be preserved.

Finally, I observe that Codel performs marking on a time schedule, not on a proportion of packets passing through.  This is entirely consistent with the above observations about transport behaviour, and further confirms the need to preserve the number of marked packets, not the proportion of marked bytes or packets.

The existing language in RFC-3168 succeeds in preserving the number of CE marks applied to a flow.  Any deficiencies we should consider are in relation to handling the distinction between ECT(1) and ECT(0), as this is what newly becomes significant with both the L4S and SCE proposals.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Mar 2020 02:53:48 +0000

David,

On 10/03/2020 20:35, Black, David wrote:
>
> Bob,
>
> As draft shepherd, let me suggest an alternate way to think about 
> fragmentation and reassembly that may result in simpler text.
>
> The current situation arises in large part from viewing fragmentation 
> and reassembly as being part of tunnel encapsulation and tunnel 
> decapsulation respectively.
>
>   * A tunnel ingress encapsulates the inner packet and then fragments
>     the resulting (encapsulated) outer packet if necessary for
>     transmission.
>       o RFC 6040 and this document specify the ECN requirements for
>         encapsulation.  RFC 3168 specifies ECN requirements for
>         fragmentation.
>   * A tunnel egress reassembles the outer packet and then decapsulates
>     the resulting (still encapsulated) outer packet to produce the
>     inner packet.
>       o RFC 3168 specifies ECN requirements for reassembly.  RFC 6040
>         and this document specify the ECN requirements for decapsulation.
>
Yes, this is how I hope everyone understands how outer 
fragmentation/reassembly relates to encap/decap. But that's not the 
point at issue.

I was hoping for something less like approach A = explicitly refer to 
RFC3168 for fragmentation/reassembly (even tho it's broken).
And something closer to approach C = don't say anything about 
fragmentation/reassembly.

I suspect most fragmentation/re-assembly code in encap/decap 
implementations doesn't even consider what to do with the ECN field. I 
don't want to explicitly flag to people at this stage that they should 
have implemented RFC3168, if we're likely to update it soon.

>      o
>
> Optimized implementations may of course mix encapsulation and 
> decapsulation processing with fragmentation and reassembly processing, 
> respectively, but the results are required to be the same as if the 
> above orders of processing were followed and that processing adhered 
> to the requirements listed above.
>

If I used the major bullets, I would just remove "then", "resulting" and 
the words in parenthesis, so the order or operations is non-specific. Viz:
     "A tunnel ingress encapsulates the inner packet and fragments the 
outer packet if necessary for transmission."
     "A tunnel egress reassembles the outer packet and decapsulates the 
outer packet to produce the inner packet."

But I'd rather just not say anything if there's nothing new to say.

> I think the result will be clearer, and will also make it obvious that 
> nothing new is being required, especially if “RFC 6040 and this 
> document specify” can be changed to “RFC 6040 specifies” in both 
> sub-bullets above.
>
If nothing new is required, we don't have to say anything. I'd much 
rather just delete S.5 of rfc6040update-shim.

For now, if anyone needs to work out which RFC tells them how to do 
fragmentation / reassembly with ECN, they can do that by typing "ECN 
fragmentation RFC" into any good search engine.

Less is more.

================================
Now for something completely different; ecn-encap-guidelines

Since the early days of the doc, there's been a section about 
re-framing. This is not about the mapping:
     1 packet <-> n fragments

It's about mapping ECN in these different cases:
     1 frame <-> n packets
     sequence of packets <-> byte stream <-> frames (oblivious to the 
original packet boundaries [e.g. DOCSIS])
     spraying pieces of packets into different frames (like 3G did).

The logical OR approach in RFC3168 was never written to cover these 
cases - it doesn't generalize to them anyway. If a large frame is 
congestion marked at a L2 AQM, you don't want all the packets inside to 
be marked, and none of the packets in other frames. Such bunching of 
congestion signals would not be good.

What do you think of the text I wrote on Monday night (see S.4.6):
http://www.bobbriscoe.net/projects/netsvc_i-f/consig/encap/draft-ietf-tsvwg-ecn-encap-guidelines-14-COULD-NOT-SUBMIT.txt
Even tho not editorially brilliant, I think it's technically sound.

When Joe Touch pointed out that this draft didn't mention fragmentation, 
I just added fragments to the list in the first para. I can take it out 
if you think it's not appropriate.

Here's the diff from the stuff in the previous draft (I removed the 
octet preserving congestion marking stuff):
http://www.bobbriscoe.net/projects/netsvc_i-f/consig/encap/draft-ietf-tsvwg-ecn-encap-guidelines-14a-DIFF-13.html

Bob
> *Fr"om:*Bob Briscoe <[email-masked]>
> *Sent:* Tuesday, March 10, 2020 2:47 PM
> *To:* Black, David; [email-masked]
> *Subject:* Re: [tsvwg] Status of ECN encapsulation drafts (i.e., stuck)
>
> [EXTERNAL EMAIL]
>
> David,
>
> rfc64040update-shim to update RFC3168's fragmentation behaviour, even 
> if it's the "right thing" to do, given I was saying that there were 
>     rfc64040update-shim or ecn-encap-guidelines. Then use a later RFC
>     to update them both (stds track and BCP) with a considered
>     'correct' approach. ecn-encap-guidelines would still say include
>     what it has always said about re-framing (which is a similar but
>     different subject).
>     D) Convince ourselves that fragmentation and reassembly during
>     encap and decap is allowed to be different from fragmentation and
>     draft-ietf-tsvwg-rfc6040update-shim, which are being handled
>     together for WG Last Call and RFC publication, and is posted in my
>     role as shepherd and responsible WG chair for these drafts.The
>     current situation is that both drafts are stuck due to a problem
>     with the fragementation text added to the rfc6040update-shim
>     draft.   Section 5 on ECN Propagation and Fragmentation/Reassembly
>     was added to that draft in response to a WGLC comment, and it
>     appears to have gone too far in the direction of trying to do the
>     proverbial “right thing”.
>
>     Much as that may be the proverbial “right thing” to do,
>     particularly with the benefit of 20/20 hindsight, that text is
>     inconsistent with the following text from Section 5.3 of RFC 3168
>     (https://tools.ietf.org/html/rfc3168#section-5.3), as Markku Kojo
>     has pointed out:
>
>     The 6040update-shim draft is intended to update RFC 6040, and a
>     number of the tunnel protocol drafts, but it is not intended to
>     update RFC 3168, and hence the above new text (albeit
>     well-intentioned) is a showstopper.   Changing ECN fragmentation
>     behavior should be done in a separate draft.
>
>     Bob (as draft editor) – do you want to propose some new text to
>     the list, possibly after private email discussion with Marco and
>     me to figure out what it needs to say?
>
>     Thanks, --David
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 13:03:15 +0000

Jonathan,

I have had to write a very long reply to explain why you are incorrect 
on this.
So, for the list, I'll summarize here at the top.

[@Jonathan, Please don't reply to the summary alone - not until you've 
read the explanation right through. If I've convinced you, please don't 
just go silent - please confirm that you were wrong, so that we can 
unstick this blockage quickly.]

Problem: RFC 3168 mandates a reassembly process for CE markings that is 
incorrect.
Solution: The proposed updated requirement corrects RFC3168 (for all 
these cases), but allows the original behaviour (avoids making existing 
implementations non-compliant).

This is orthogonal to the debates about scalable, classic, SCE, L4S, or 
whatever. RFC3168 is incorrect and the proposal is correct for all types 
of congestion controls and all meanings of ECN codepoints. [Your very 
last point about re-assembly with ECT1 does have a downside for SCE but 
not for L4S, so I have started a new thread for that.]

David Black and I are debating where a proposed correction to the 
reassembly requirement should go (these drafts?, a new draft?, etc). 
This email addresses your questioning of the correctness of the new 
requirement in the first place.

Specific problem: End-to-end, taking into account both fragmentation and 
reassembly, the "logical OR" reassembly process required in RFC3168 
roughly *doubles* *both* the number *and* the proportion of CE marked 
packets *and* the number and proportion of CE-marked bytes (relative to 
a parallel flow with slightly smaller packets that is not fragmented).

Specific solution: The proposed corrected approach *preserves all four* 
of the above properties: the number and proportion of CE marked packets 
and the number and proportion of CE-marked bytes, again taking into 
account both fragmentation and reassembly end-to-end. Actually it 
inherently slightly inflates all of them in proportion to the extra 
inner headers on the fragments, which is an even better property.

Also, a proposed example implementation is as simple, or simpler than 
needed for RFC3168 (it requires one additional state per interface, 
rather than per packet).

See explanation inline...

> As I have stated in the past on this subject, I disagree with this characterisation.  It misconstrues the meaning of CE and the definition of steady state involving it, both for conventional conges…
The wording also allows the "logical OR" approach if you believe it is 
simpler. Nonetheless, the two example approaches given in this and 
previous version of the draft both have the same or better simplicity 
than 'logical OR'.
>
> For conventional congestion control, the steady state is defined in terms of the time interval (during which cwnd growth occurs) between RTTs containing at least one CE mark (and/or packet loss).

When the network introduces ECN marking it doesn't know which of its 
marks will be considered by the transport to be within the same RTT. 
Indeed, in general, the network doesn't even know which marks will be 
seen by the same L4 flow (FQ+AQM is an exception). All an AQM has to do 
is increase or decrease the marking probability in response to growth or 
shrinkage of the queue. Decreasing or increasing the gap between marks 
is equivalent.

Your understanding about "the time interval between RTTs containing at 
least one CE mark/loss" is correct from the point of view of a 
conventional congestion controller. But it's wrong to say that "the 
control loop is not properly closed..." unless "...each single CE mark 
is attached to the same packet as the marked fragment belonged to".

If reassembly of marked fragments results in fewer marked packets, as 
long as timeliness is preserved (see next), the reassembly process just 
becomes a constant multiplier within the control loop. The control loop 
is no less closed as a result.

The Pythonesque "every congestion event is sacred" mindset would only 
apply if the AQM somehow knew what rate it wanted each flow to adopt, 
then applied a pre-calculated non-adaptive spacing between markings to 
make that so. That's (obviously) not how the Internet's control loops 
work. The whole point is they continually adapt.

> It is necessary for a single CE mark to be delivered promptly, ie. attached to the same packet as the marked fragment belonged to, in order to minimise growth overshoot and keep the control loop properly closed.

Yup. Timeliness is important - that's a given. The previous version gave 
the following example for how to ensure timeliness.

    marked octets.

I deleted a para about preserving octets (which included the above 
sentences), mainly because you didn't like the para. I intended to 
replace the above sentences with the sentence below. However, in one of 
the drafts it looks like I got distracted during this 2-phase commit. 
(Previous versions of the ecn-encap draft had similar sentences to those 
above, which I had already generalized to the one below):

    The mechanism for propagating congestion indications SHOULD ensure
    that any incoming congestion indication is propagated immediately,
    not held awaiting the possibility of further congestion indications
    to be sufficient to indicate congestion on an outgoing PDU.

I have now pasted this latter sentence into my local editor's copy of 
rfc6040update-shim, which was my intention, so it will appear in the 
next revision.

> The control loop is not sensitive to the "proportion of bytes marked", only that a mark was encountered at a particular time.

Both sides are true, not just one side: a conventional congestion 
response is dependent on a mark being encountered at a particular time, 
*which makes it* sensitive to the proportion of bytes marked.

If that were not true, AQMs would not work (because all they do is alter 
the marking probability). You must be aware that TCP's window depends on 
the marking probability.

I will try to explain below. But I don't know which part of the whole 
picture you're not grocking. So, ultimately, you need to work out for 
yourself how both these things can be true at once, rather than trying 
to argue that all the understanding in the congestion control community 
is wrong, just because you haven't yet reconciled it with the way you 
currently think. That's what reading papers is for.

> It is well known, moreover, that adding delay to a control loop destabilises it - but this is exactly what an attempt to maintain the proportion of marked bytes would do.

Here goes: an example.

Let's assume the tunnel fragments the packets of some IPv4 flows into 2 
fragments; one large (always size S to make the explanation simple), one 
small (always size s).  Sizes include inner headers.

Now, I'm going to assume the reassembly algo in the previous version of 
the draft that maintains a per-interface counter incremented by incoming 
marked octets and decremented by outgoing marked octets. It marks 
outgoing packets while (counter >= 0). That only uses per-interface 
state, whereas the "logical OR" algorithm uses per-packet state.

At reassembly, let's start from the point where a marked larger fragment 
arrives (counter rises to S), but its little sister fragment isn't 
marked. The algo marks this first reassembled packet (size S+s), and the 
counter goes negative (-s). The next time a marked fragment arrives it 
bumps up the counter, either to 0 if it's a small fragment or to (S-s) 
if it's a large one. Either way it marks the outgoing reassembled 
packet. So the counter drops to either (-s-S) or (-2s).

No delayed signals so far. But, between packets, the counter will 
continue to go more negative until the deficit is less than the size of 
an arriving marked fragment, which increases the counter but it's still 
not quite positive. So the algo doesn't propagate that congestion 
signal. That looks like a delayed signal, doesn't it? But there's more 
to this than meets the eye....

For explanation, let's take 2 types of flows, one that the tunnel 
doesn't have to fragment and one it does. And let's start from a 
position where they are all originally sent with the same packet rate 
(pre-fragmentation).

The tunnel ingress doubles the packet rate of the flows it fragments. 
Therefore, during any congestion event, the AQM marks twice as many 
packets from the fragmented flows as the others {Note 1}. Reassembly 
restores the packet rate to that at the origin sender. And it removes 
some of the extra congestion markings that fragmentation added.

For example, if 100 packets of the unfragmented flow received 2 marks, 
100 packets of the fragmented flow would turn into 200 fragments, 4 
would be marked, then reassembly would turn them back into 100 packets 
and a little more than 2 would be marked (because of the extra size of 
the inner headers).

If reassembly used "logical OR" instead, of the 100 forwarded packets, 4 
would be marked (3.96% to be precise).

With marked-byte-preserving, TCP will reduce the average packet rate of 
the fragmented flows in proportion to the extra headers. With "logical 
OR", TCP will roughly halve the rate of the fragmented flows.

Any change in the aggregate marking probability at the AQM due to 
fragmentation is just a constant multiplier in all the control loops, 
which will all just adapt accordingly. When there's a pulse in load, 
removing some signals at reassembly doesn't delay the signals any more 
than adding signals during fragmentation "undelays" signals. The system 
was already at a different operating point to start with, due to the 
presence of fragmentation. So it doesn't take any longer to signal a 
pulse in load.

In summary, it's wrong to focus only on the removal of some congestion 
signals, without also taking into account the addition of more in the 
first place.

This is the flaw in the "every mark is sacred" mindset. It selectively 
focuses on preserving one piece of the bigger picture, without noticing 
that the whole picture is different.

{Note 1}: Fragmentation doesn't alter the aggregate bit-rate (other than 
the extra headers). So there are not that many more marked packets 
overall, because the AQM isn't that much more congested. The markings 
are just biased more onto the fragmented flows.

> For DCTCP-style congestion control, the steady state is defined in terms of the number of CE marks received per RTT.  Fragmentation does not change the RTT, only the number of packets passing a middlebox located on the tunnel path.  It would be reasonable to design an AQM expecting DCTCP traffic so that it produces exactly the correct number of CE marks per RTT at a particular queue depth.  But if the number of marks is effectively halved by a reassembly process that attempts to preserve the number of marked bytes, that queue will continue to grow past that designed ideal point.
Again, this is because you weren't looking at the whole picture. You 
hadn't noticed that fragmentation at the ingress doubled the packet rate 
of the flows needing fragmentation to start with.

> We can therefore conclude that DCTCP is also insensitive to the proportion of marked bytes, and this is not a property worth preserving; rather, the total number of marked packets should be preserved.
Preserving the proportion of marked bytes at reassembly is done to 
preserve BOTH the proportion of marked packets AND the proportion of 
marked bytes over the whole path.

> Finally, I observe that Codel performs marking on a time schedule, not on a proportion of packets passing through.  This is entirely consistent with the above observations about transport behaviour…
Oh dear, this is not correct either. CoDel uses a time schedule, but it 
doesn't use a particular time for a particular degree of congestion. It 
just increases or decreases the time until it's right. So it's no 
different from increasing and decreasing the marking proportion until it 
gets it right. It's adjusting both time and proportion at the same time.

In general, the choice of unit is only important for a target, not for a 
control variable (linearity is important for a control variable though).

>
> The existing language in RFC-3168 succeeds in preserving the number of CE marks applied to a flow.
But reassembly halves the number of packets (typically). So RFC 3168 
doubles the /proportion/ of CE marked packets.

Over the whole path, the combination of fragmentation AND RFC3168 
reassembly doubles both the number and the proportion of CE marks 
applied to a flow.

> Any deficiencies we should consider are in relation to handling the distinction between ECT(1) and ECT(0), as this is what newly becomes significant with both the L4S and SCE proposals.
That is also incorrect, but it requires a different thread.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 16:03:37 +0200

> On 13 Mar, 2020, at 3:03 pm, Bob Briscoe <[email-masked]> wrote:
> 
> Specific problem: End-to-end, taking into account both fragmentation and reassembly, the "logical OR" reassembly process required in RFC3168 roughly doubles boththe number and the proportion of CE marked packets and the number and proportion of CE-marked bytes (relative to a parallel flow with slightly smaller packets that is not fragmented).

Before I attempt to address your arguments in detail, I want to clarify something fundamental.  I think this may be one good reason why David Black wants to separate the concerns of reassembly (covered by RFC-3168) and decapsulation (covered by RFC-6040) - a sentiment I completely agree with.

Let's consider two original, consecutive IP packets: A B.  They are encapsulated by a tunnel, producing TA TB.  These are both larger than the MTU, so are fragmented into TA1 TA2 TB1 TB2.  On the tunnel path, an AQM applies a CE mark to TB1.  Eventually these packets are decapsulated, prior to which they must be reassembled, and the CE mark must be preserved somehow.

What you seem to imply is that somehow, TA2 and TB1 can be reassembled into one CE-marked packet, leaving TA1 and TB2 separate, and that decapsulation may then apply the CE mark to both A and B.

But this does not jive with what I know about IP-layer fragmentation, in which TA1(ECT) TA2(ECT) always reassembles into TA(ECT), and TB1(CE) TB2(ECT) reassembles into TB, which is then CE marked by the RFC-3168 bitwise-OR rule.  We would then have A(ECT) and B(CE) transmitted onward, which seems to match the intent of the AQM - in particular, the number of CE marks is preserved, not doubled.

What am I missing from the context of your objections?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 15:35:58 +0000

Jonathan

> The existing language in RFC-3168 succeeds in preserving the number of CE marks applied to a flow.  Any deficiencies we should consider are in relation to handling the distinction between ECT(1) an…
* For SCE to give any benefit in the presence of fragmentation, it needs 
reassembly of ECT1 fragments to be changed.
* In contrast, L4S needs no change to reassembly of ECT1 fragments, 
because it keeps to the RFC3168 transitions.  So there will never need 
to be a packet consisting of a mix of ECT0 & ECT1 fragments.

Please confirm that this is only an issue with SCE, not L4S.
And please do not cast doubt over L4S by saying this affects L4S when it 
does not.

Before going on, I understand fragmentation in both IPv4 and IPv6 is 
better to be avoided by using PMTUD. But I believe fragmentation is 
still used, particularly where an IPv4 router ignores the DF flag, which 
I believe is particularly prevalent with tunnels.

Explanation of the issue:
An SCE AQM changes some ECT0 markings to ECT1. So, when these packets 
happen to be fragments (having already been fragmented), there will 
usually be a mix of ECT1 and ECT0 fragments to be reassembled into a 
packet. RFC3168 identifies this case and explicitly says it does not 
specify what to do, which would require a new specification. So current 
behaviour will be implementation-dependent.

IPv6:
If fragmentation is needed, IPv6 fragments packets at source, and 
reassembles at the receiver.
So, for SCE to provide any benefit if the IPv6 source is fragmenting, 
the receiver implementation of IPv6 will need to be updated (once a spec 
has been written, agreed and approved).

Until now, I thought that, at least in QUIC, SCE would get feedback 
without changing the receiver. But, if the sender is fragmenting, the 
receiver's IPv6 layer will need to be changed as well. It's possible 
some receiver implementations might happen to do roughly the right thing 
(once we agree what the right thing is).

IPv4:
RFC3168 advises to set the DF flag if a mix of ECT0 and ECT1 is expected.
However, many IPv4 tunnels ignore DF and fragment anyway, using "outer 
fragments" [draft-ietf-intarea-tunnels].
Therefore, the IPv4 reassembly behaviour will need to be specified. Then 
this ECT1 reassembly during tunnel decapsulation will need to be 
implemented.

As well as IP-in-IP, and IPSec, here's a list of the 14 IP-shim-(L2)-IP 
encapsulations that are widely deployed and whether they comply with 
RFC6040 (needed for SCE tunnel decap). To support SCE, they will also 
need fragment reassembly to be specified and implemented.

==Existing specs==

RFC3168 only specifies reassembly for IPv4, but explicitly not for the 
case with mixed ECT0 and ECT1 fragments. It does not seem to recognize 
that IPv6 e2e fragmentation exists, even tho IPv6 was published 3 years 
before.

    For example, if there is a malicious or [SCE or]
    broken entity in the path at or after the fragmentation point, packet
    fragments could carry a mixture of ECT(0), ECT(1), and/or Not-ECT
    codepoints.  The reassembly specification above does not place
    requirements on reassembly of fragments in this case.  In situations
    where more precise reassembly behavior would be required, protocol
    specifications SHOULD instead specify that DF MUST be set in all
    ECN-capable packets sent by the protocol.

DF does not exist in IPv6.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 16:42:04 +0000

Jonathan,

> Before I attempt to address your arguments in detail, I want to clarify something fundamental.  I think this may be one good reason why David Black wants to separate the concerns of reassembly (cov…
Yup. As I said in that conversation, that's not in question.
> What you seem to imply is that somehow, TA2 and TB1 can be reassembled into one CE-marked packet, leaving TA1 and TB2 separate, and that decapsulation may then apply the CE mark to both A and B.
Eh? Where do you think I've implied that? Nah.

I've just re-read, and maybe you've assumed that the single counter 
implies I'm just reassembling any fragment with any other. No. Obviously 
it only reassembles the fragments of one packet back together. I didn't 
think I'd have to say that!

Anyway, the implementation is only an example for people who find it 
easier to understand something concrete - the point is about the 
requirement, not how to implement it.

Nonetheless, just in case, I ought to say what does /not/ happen in the 
example implementation: A non-CE fragment arriving to be reassembled 
cannot cause the reassembled packet to be CE-marked. So there is no need 
to check the counter unless an arriving fragment is CE-marked. Also, 
regarding complexity, I've realized that both the "logical OR" and the 
"CE-byte-preserving" algorithms require per packet state (whether the 
re-assembled packet will be CE marked). The CE-byte-preserving algo adds 
one counter per-interface (probably two counters in practice, so writing 
by the incoming and outgoing processes can be decoupled).

> What am I missing from the context of your objections?
Well, I don't know, 'cos I don't know how you came to that 
interpretation (you'll need to point to the sentence that made you think 
that, if it's still not clear).

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 19:23:06 +0200

> On 13 Mar, 2020, at 6:42 pm, Bob Briscoe <[email-masked]> wrote:
> 

> Well, I don't know, 'cos I don't know how you came to that interpretation (you'll need to point to the sentence that made you think that, if it's still not clear).

I'm referring specifically to your claim that the existing specifications can result in a doubling of CE marks in the reassembled and decapsulated packets, relative to the number present in the encapsulated, fragmented packets arriving at the decapsulator.  I cannot at present identify any mechanism which permits this to occur, but it seems to form the basis of your justification for modified language.

So *one* of us has clearly misread the specs in question.  I just want to figure out which of us it is.  You could do so by quoting chapter and verse for such a mechanism, and we can work from there.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 13 Mar 2020 17:24:59 +0000

Hi Bob,

> that, if it's still not clear).

I'm also confused, so although I'm not speaking for Jonathan, I'll point to a sentence
that made me think the response raised an excellent and relevant point:
(From https://mailarchive.ietf.org/arch/msg/tsvwg/B2Gnb-jYPiZFYuA3YXNRhktKaT4/)

  'the "logical OR" reassembly process required in RFC3168 roughly doubles both the number
  and the proportion of CE marked packets and the number and proportion of CE-marked bytes
  (relative to a parallel flow with slightly smaller packets that is not fragmented).'

I thought the example succinctly suggested that one of the 4 things listed above can't be
doubled, namely:

- the number of CE-marked packets.

If there's a way for the number of CE packets to be doubled by the "logical OR" reassembly
process, I would like to see an example.  I haven't been able to think of one.

To me also, this seems a fundamental point worth clarifying.

(I do see that the other 3 metrics can be doubled under some specific assumptions about
the fragments and probabilities, as likewise demonstrated by the simple example.)

Best regards,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 17:45:34 +0000

Jonathan,

No, you have to read the email. The doubling comes at fragmentation, 
then the reassembly is meant to compensate for it.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 17:54:17 +0000

Jake,

Same answer as for Jonathan - I'm talking e2e, not just reassembly, 
taking account of frag and re-assembly relative to another 
non-fragmented flow. All carefully defined in the email (so pls don't 
question this sentence, 'cos I didn't want to re-write all the carefully 
defined stuff again). You have to read the build-up to the scenario too.

Of course, if I'm wrong, I'll admit it. However, not if someone just 
asserts that the summary is wrong without pointing to the flaw in the 
body of the argument.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 19:56:15 +0200

> On 13 Mar, 2020, at 7:45 pm, Bob Briscoe <[email-masked]> wrote:
> 
> No, you have to read the email. The doubling comes at fragmentation, then the reassembly is meant to compensate for it.

Ah, so the situation is that the CE mark occurs *before* the tunnel and the fragmentation.  But this should not cause any problem in the end, either, because the packets are reassembled before being interpreted by the transport receiver.  Reworking my earlier example:

Let's consider two original, consecutive IP packets: A B, both marked ECT.  B is then marked CE by an AQM.  They are then encapsulated by a tunnel, producing TA(ECT) TB(CE).  These are both larger than the MTU, so are fragmented into TA1(ECT) TA2(ECT) TB1(CE) TB2(CE).  Eventually these packets are decapsulated, prior to which they must be reassembled, and the CE marks on TB1 and TB2 must be preserved somehow.

TA1(ECT) TA2(ECT) always reassembles into TA(ECT), and TB1(CE) TB2(CE) reassembles into TB(CE) under RFC-3168 rules.  We would then have A(ECT) and B(CE) transmitted onward, which seems to match the intent of the AQM - in particular, the number of CE marks is preserved, not doubled.

So again, what context am I missing?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 17:58:42 +0000

Jonathan,

On 13/03/2020 17:56, Jonathan Morton wrote:
> Ah, so the situation is that the CE mark occurs *before* the tunnel and the fragmentation.

No.

Are you reading my recent emails, or the main email?

Bob

>   But this should not cause any problem in the end, either, because the packets are reassembled before being interpreted by the transport receiver.  Reworking my earlier example:
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 20:15:59 +0200

> On 13 Mar, 2020, at 7:58 pm, Bob Briscoe <[email-masked]> wrote:
> 

The recent ones, because I'm trying to establish this fundamental point first.  Until I can figure out what you *do* mean by CE marks being doubled, there's no point in discussing anything more subtle.

So again, I ask: what context am I missing?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 18:50:49 +0000

Jonathan,

> So again, I ask: what context am I missing?
I said "No." I.e. the AQM is not marking before the tunnel ingress. It 
is marking between tunnel ingress and egress.

How can I know what context you're missing if you haven't read the 
email? Presumably that means you're missing all the context given in the 
email.

Pls can you take these sort of emails off the IETF lists - I'm sure 
no-one cares for this sort of content-free conversation.

Bob

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 13 Mar 2020 19:28:29 +0000

Hi Bob,

> asserts that the summary is wrong without pointing to the flaw in the 

I guess I'll try again with the example from further down the body of the
email, which seems to contain a more explicit form of the claim in question:
(From https://mailarchive.ietf.org/arch/msg/tsvwg/B2Gnb-jYPiZFYuA3YXNRhktKaT4/ )

This seems mistaken to me.

If 2 of the 100 packets are marked pre-fragmentation, then fragmentation makes
it 4 of 200, then logical-OR reassembly will result in marking only for 2 packets
of the 100 post-reassembly.

There's no opportunity to increase the count of marked packets without receiving
new CE marks while fragmented because the reassembly doesn't cross the un-fragmented
packet boundaries.

It seems to me you're asserting that this would happen, but using a count-based
model (or maybe stream-based model?) that mistakenly fails to account for the way
reassembly avoids crossing IP packet boundaries of the pre-fragmented packets.

Also, since it seems to be an issue:  I don't think this line of inquiry is just
disagreeing with a summary or nitpicking at a minor point--the assertion that the
count of CE-marked packets can increase e2e because of fragmentation and reassembly
seems fundamental to the claim that logical-OR reassembly is broken.

I still leave open the possibility that I've misunderstood, so I'm asking for an
example of a packet sequence e2e that demonstrates how the count of CE-marked packets
can increase due to fragmentation and reassembly, because until I can understand that
it's possible, I tentatively conclude that the bulk of the argument presented was
built on a fundamental error.

I have read the email in its entirety and found no explanation of a fragmentation+
reassembly chain of events that allows a sequence of packets containing 2
pre-fragmented marked packets to turn into a sequence with 4 post-reassembly marked
packets, unless at least 2 extra marks were added while fragmented (the quoted part
above merely asserts it can happen, without an explanation of how anywhere I could
find).

If this is possible in a conformant network, I'm asking anyone who can explain it to
please do so, of course I don't limit my request to Bob.

Best regards,
Jake

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 21:45:22 +0200

> On 13 Mar, 2020, at 8:50 pm, Bob Briscoe <[email-masked]> wrote:
> 
> Pls can you take these sort of emails off the IETF lists - I'm sure no-one cares for this sort of content-free conversation.

No, because I think this *is* actually important to establish and understand.

>> I'm trying to establish this fundamental point first.  Until I can figure out what you *do* mean by CE marks being doubled, there's no point in discussing anything more subtle.
>> 

> I said "No." I.e. the AQM is not marking before the tunnel ingress. It is marking between tunnel ingress and egress.
> 
> How can I know what context you're missing if you haven't read the email? Presumably that means you're missing all the context given in the email.

As it happens, I think I now understand what you mean.

If you assume the AQM on the tunnel path is marking probabilistically, then doubling the packet rate over time (by fragmenting the original packets into two smaller ones each) also doubles the marking rate over time, and (assuming sufficiently independent probability of marking) the number of reassembled packets carrying marks also doubles.

You could have just *said* that, and not left me and Jake guessing for the past half-dozen e-mails.  I did not find such a clear statement in your earlier "main" comment either, despite its length.

I should also note that this effect does *not* occur with a time-schedule marking AQM such as Codel, in which the marking rate over time is independent of the packet rate, so if the packet rate is doubled the marking probability is actually halved.  Hence the number of reassembled packets carrying marks remains the same.

With *that* out of the way, I can now address the substance of your main argument.

First, on an RFC-3168 flow, the proportion of marked packets does not matter.  It only matters whether a later CE mark occurs before or after the CWR corresponding to an earlier one - ie. *when* it happens.  If before, then it is ignored.  If after, then it triggers a fresh ECE/CWR cycle.  Since CWR corresponds to the sender performing Multiplicative Decrease, the likelihood is that the queue will drain coincident with it, and no further CE marks will occur until the growth phase of the AIMD sawtooth completes again.

The situation is different for L4S because it performs, and relies on, precise feedback of the number of CE marks, regardless of their spacing.  In that case, an inflation of the marking rate would be expected to have some effect.  But if the control loop is properly closed, that effect will simply be to settle at either a shallower queue depth or a lower marking probability, depending on details of the AQM involved.  As noted above, Codel does the latter inherently.  I think a PI-based scheme would do so as well.

I therefore see no reason to complicate RFC-3168's handling of CE marks upon reassembly of fragmented packets.  We can have a separate conversation about ECT(1), in that other thread I haven't responded to yet.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 13 Mar 2020 21:19:11 +0000

On 3/13/20, 12:45 PM, "Jonathan Morton" <[email-masked]> wrote:
>    If you assume the AQM on the tunnel path is marking probabilistically,
> then doubling the packet rate over time (by fragmenting the original packets
> into two smaller ones each) also doubles the marking rate over time, and
> (assuming sufficiently independent probability of marking) the number of
> reassembled packets carrying marks also doubles.

Ah, thanks Jonathan.  I agree this is a good guess at the model.

I'll try my own phrasing to see if there's a chance I might be caught up now:

(1) If we consider 2 flows, one whose packets are all fragmented into 2 packets
each, and another that's unfragmented, but otherwise identically sending across an
identically congested AQM, an AQM that marks 2% of the packets in the unfragmented
one could end up causing marks on up to 4% of the packets in the fragmented flow,
as counted after reassembly.

If that successfully gets us all on roughly the same page, can we also agree
that logical-OR reassembly preserves the property below?

(2) If a receiver sees N CE-marked packets in a sequence of packets, the network
emitted at least N separate CE-marks while forwarding those packets.

To clarify one of my previous messages, (1) addresses what I was calling
"proportion of CE-marked packets" and (2) speaks to what I meant by "number of
CE-marked packets" in my response.  I was thrown by the "both number and proportion"
claim in the "long response" email.

In retrospect this looks like a semantics debate I'm happy to now abandon, though
I'm grateful for having my confusion about the discussion cleared up, if that
has indeed happened.

Anyway, coming back to the original thread: I certainly do think the (B) option
(from https://mailarchive.ietf.org/arch/msg/tsvwg/RFmIUiPb0vss7xt16yeK2TGJSJU/ )
as expressed in draft-ietf-tsvwg-rfc6040update-shim-10#section-5 takes positions that
don't seem backed by wg consensus.

I also think that David's suggested approach (in 
https://mailarchive.ietf.org/arch/msg/tsvwg/-EgX4VHHMLEGmj4IbeRNruLd86A/) sounds
like the best option on offer.

FWIW, I prefer David's suggestion to the (C) option of deleting any mention of
fragmentation in hopes that nobody implements something 3168-compliant, on the theory
that it's something we might later update.  Clarity about the existing relevant specs
seems just better to me, and this doc can also be updated if we do indeed update the
reassembly rules at some point.

Best regards,
Jake

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 23:22:12 +0200

> * For SCE to give any benefit in the presence of fragmentation, it needs reassembly of ECT1 fragments to be changed. 
> * In contrast, L4S needs no change to reassembly of ECT1 fragments, because it keeps to the RFC3168 transitions.  So there will never need to be a packet consisting of a mix of ECT0 & ECT1 fragments.

I'll agree with this: where the ECN codepoint is set at origin (or, equivalently, before the fragmentation point), the fragments will all carry identical codepoints and will be reassembled correctly under RFC-3168 rules.  These rules also handle the case of CE marks being set on IP fragments on the tunnel path.  The case of marking with ECT(1) is indeed left undefined, which is undesirable.

> Before going on, I understand fragmentation in both IPv4 and IPv6 is better to be avoided by using PMTUD. But I believe fragmentation is still used, particularly where an IPv4 router ignores the DF flag, which I believe is particularly prevalent with tunnels.

> DF does not exist in IPv6.

I think it would be more accurate to say that IPv6 behaves as though DF is *always* set.  There is no defined mechanism for a router to perform fragmentation.  Fragmented packets may be generated *at source* to eg. convey an oversize datagram; TCP would just reduce the MSS to fit, when notified of a reduced MTU on the path.  So I don't consider IPv6 fragmentation to be a significant problem.

Of course, you could end up tunnelling IPv6 packets through a fragmented IPv4 scheme.  But that mathematically reduces to the existing problem of fragmenting IPv4.

> An SCE AQM changes some ECT0 markings to ECT1. So, when these packets happen to be fragments (having already been fragmented), there will usually be a mix of ECT1 and ECT0 fragments to be reassembled into a packet. RFC3168 identifies this case and explicitly says it does not specify what to do, which would require a new specification. So current behaviour will be implementation-dependent.
> 
> IPv6:
> If fragmentation is needed, IPv6 fragments packets at source, and reassembles at the receiver.
> So, for SCE to provide any benefit if the IPv6 source is fragmenting, the receiver implementation of IPv6 will need to be updated (once a spec has been written, agreed and approved).
> 
> Until now, I thought that, at least in QUIC, SCE would get feedback without changing the receiver. But, if the sender is fragmenting, the receiver's IPv6 layer will need to be changed as well. It's possible some receiver implementations might happen to do roughly the right thing (once we agree what the right thing is).
> 
> IPv4:
> However, many IPv4 tunnels ignore DF and fragment anyway, using "outer fragments" [draft-ietf-intarea-tunnels].
> Therefore, the IPv4 reassembly behaviour will need to be specified. Then this ECT1 reassembly during tunnel decapsulation will need to be implemented.

ISTR, at some point in the past, interim language was suggested which would require taking the ECN codepoint from one of the fragments constituting the packet, with the behaviour being otherwise unspecified except by the existing rules.  This would be a worthwhile improvement from SCE's point of view, and is likely to match at least some existing implementations.

An implementation which performs a bitwise-OR across the ECN fields of the fragments would effectively convert partial SCE marks into CE marks (as the ECT codepoints are 01 and 10).  This is less than ideal, but at least some form of congestion control is maintained by this.

RFC-3168's existing recommendation to set DF seems like a good one to me, and is effectively automatic with IPv6.  Tunnels which perform outer fragmentation should be fixed to implement MTU Discovery support instead.  That seems like the easiest fix to me.

> As well as IP-in-IP, and IPSec, here's a list of the 14 IP-shim-(L2)-IP encapsulations that are widely deployed and whether they comply with RFC6040 (needed for SCE tunnel decap). To support SCE, they will also need fragment reassembly to be specified and implemented.

The list of 14 appears to be missing.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 23:37:58 +0200

> On 13 Mar, 2020, at 11:19 pm, Holland, Jake <[email-masked]> wrote:
> 

Yes, that's consistent with my current understanding.

> claim in the "long response" email.

Yes, I believe that's true.  I don't think either of the worked examples I gave are in dispute.

Ultimately I think the confusion arose from Bob thinking that marking proportion and marking rate are related in ways that, in fact, they are not (except under assumptions that are invalid in this case).

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 13 Mar 2020 23:54:22 +0100

Dear All,

so in this example we need an operator that operates a tunnel (encapsulation and decapsulation) and an AQM on the same tunnel and is considered to both (mis-)configure the tunnel to allow fragmentation and operate an SCE-AQM on hops along the tunnel's path. Is that really considered to be a sensible deployment that is worth catering to? I would guess having a tunnel that fragments/defragments alone would be painful/sub-optimal. Why would an operator that consciously accepted such a tunnel design (as there probably real world constraints that can justify such an arrangement) would then go and add a non-tunnel compatible AQM? Are fragmenting tunnels really such a big part of the commercial ISP world?

Best Regards
	Sebastian

> On Mar 13, 2020, at 19:50, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
>

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 14 Mar 2020 18:07:33 +0000

Sebastian,

On 13/03/2020 22:54, Sebastian Moeller wrote:
> Dear All,
>
> so in this example we need an operator that operates a tunnel (encapsulation and decapsulation) and an AQM on the same tunnel and is considered to both (mis-)configure the tunnel to allow fragmentation and operate an SCE-AQM on hops along the tunnel's path.
Tunnels are often constructed over the top of the network operator that 
runs the physical equipment (incl. any AQM). Overlays are one of the 
common motivations for tunnelling.

> Is that really considered to be a sensible deployment that is worth catering to? I would guess having a tunnel that fragments/defragments alone would be painful/sub-optimal. Why would an operator that consciously accepted such a tunnel design (as there probably real world constraints that can justify such an arrangement) would then go and add a non-tunnel compatible AQM? Are fragmenting tunnels really such a big part of the commercial ISP world?t
Short answer: I don't know.
Where an operator who sets up a tunnel can control the MTU of the links, 
I would imagine fragmentation is highly unlikely. But for overlays using 
tunnelling, most tunnel endpoints use PMTUD between themselves to find a 
good tunnel MTU, and I would imagine they use fragmentation for IPv4 
(because they have to, if their "packet too big" ICMPs are being 
blocked) for any incoming packets that don't fit once they add tunnel 
headers. But I don't know how often they actually have to fragment 
(depends on end-systems strategies are to keep the MTU below this point).

The only concrete data I can find is from the year 2000.
https://www.caida.org/research/traffic-analysis/fragments/
In this case all the packets on the link were fragments, and "A 
significant portion of the fragmented traffic ... is tunneled traffic." 
I don't know whether this link was chosen for this study because all the 
packets were fragments, or whether it would have been easy to find other 
links with a high proportion of fragments.

Finding the best max packet size is an area where neither IPv4 nor IPv6 
has ever found a good solution. Getting a tunnel to fragment and 
reassemble is indeed painfully sub-optimal, but all the other solutions 
have their own problems. It is possible the sub-optimality is often 
going on under-the-covers, just because it works. I do know that, for 
IPv4, the Don't Fragment (DF) flag is often ignored by tunnels, as a 
preferable alternative to just ditching the traffic.

In the absence of the IP layer solving this problem, Gorry's group found 
that about 20% of end-systems are just clamping the IPv4 TCP max segment 
size lower than necessary. And for IPv6 most are just using the min 
segment size. See:
https://iain.learmonth.me/stuff/pubs/UsableMTU2018.pdf
Again only for TCP, middleboxes can edit the MSS advertised in each 
packet to fool the hosts into using a smaller max segment size. You can 
configure a Cisco box (and I'm sure others) to do this when you're using 
it for tunnelling. The above study from Gorry's group didn't find much 
evidence of this though.

PLPMTUD solves this problem end-to-end (see [RFC4821] for TCP etc. and 
[draft-ietf-tsvwg-datagram-plpmtud] not yet published for UDP). However, 
where a tunnel already solves the problem (sub-optimally) for IPv4 using 
fragmentation, I can't imagine that anyone would disable it, because it 
is still necessary and it still works.

Further reading.
I just compiled a list, then realized everything is already cited here, 
in the context of minimizing latency:
http://bobbriscoe.net/projects/latency/latency_preprint.pdf#subsubsection.3.1.5

You could also try 
https://labs.ripe.net/Members/gih/evaluating-ipv4-and-ipv6-packet-fragmentation 
.

Bob
>
> Best Regards
> 	Sebastian
>
>
>
>
>> On Mar 13, 2020, at 19:50, Bob Briscoe<[email-masked]>  wrote:
>>
>> Jonathan,
>>
>>>> On 13 Mar, 2020, at 7:58 pm, Bob Briscoe<[email-masked]>  wrote:
>>>>

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sat, 14 Mar 2020 15:18:19 -0700 (PDT)

Bob,
	Questions and comments in line.

Regards
Rod

> (depends on end-systems strategies are to keep the MTU below this point).

My experience both personally and with helping others to
configure tunnels is that if PMTU is not setup correctly
the tunnels fail to carry packets >PMTU.  I only know
of one tunnel transport that has working fragmentation
and reassembly, but my exposure set is rather small.

> preferable alternative to just ditching the traffic.

What do you base your "know" on?
Can you please site an implementaiton?

> ..

-- 
Rod Grimes                                                 [email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 00:01:18 +0000

Jake,

On 13/03/2020 19:28, Holland, Jake wrote:
>> [BB] Same answer as for Jonathan - I'm talking e2e, not just reassembly,
> [JH] I guess I'll try again with the example from further down the body of the

[BB] No. As I said to Jonathan (before this email from you):
     "The AQM is not marking before the tunnel ingress. It is marking 
between tunnel ingress and egress. "

There are only ever any issues with the interaction between ECN and 
reassembly if different fragments of the same packet are marked 
differently (which doesn't happen with marking that was pre-fragmentation).

> [JH] There's no opportunity to increase the count of marked packets without receiving

[BB] No. There's no crossing of IP packet boundaries (of course).

I think you're reading more into it than I'm saying. It's totally simple.

To be unambiguous, I've provided a schematic here:
http://bobbriscoe.net/projects/netsvc_i-f/consig/encap/ecn-reassembly.pdf

[My hosting provider seems to have screwed up the DNS change-over when 
moving me to a new server on Thursday. I think my server is accessible 
to most of the world now, but apologies if it isn't.]

"For the record", I've included a text transcription at the end of this 
email (in case my schematic falls off the web in future).

Essentially, fragmentation inflates the number of packets but reassembly 
deflates it to the original number.
So if the packets are marked while they are fragmented, unless 
reassembly deflates the number of markings, fragmented flows will 
experience incorrectly inflated marking probability.

>
> [JH] Also, since it seems to be an issue:  I don't think this line of inquiry is just

[BB] I asked Jonathan not to respond to the summary without reading the 
whole email.

That was not aimed at you. It was to pre-empt what Jonathan always does: 
reads until the first point he disagrees with, then launches into a 
response, just repeating his assertions, and snips the other persons 
carefully written arguments against each of his earlier points.

Unsurprisingly, despite my request not to, he did it anyway, which is 
why this thread is now an incomprehensible mess.

If someone doesn't agree or doesn't understand, the fastest way to 
converge on understanding is for them to quote the text they disagree 
with or don't understand, like you do.

> [JH] I still leave open the possibility that I've misunderstood, so I'm asking for an

[BB] I hope it's clear now that this is not about marking before 
fragmentation (which is a non-issue).
And I hope the schematic helps clear this up.

Cheers

Bob
> Jake

=============================================
"For the record" text transcription of the schematic linked here:
http://bobbriscoe.net/projects/netsvc_i-f/consig/encap/ecn-reassembly.pdf

I'll repeat the above paras below and insert statistics at each point in 
the network, using the following terminology:
* Flow F (will be fragmented), with 1500B packets (including one IP 
header) before tunnelling.
* Flow N (will not be fragmented), with 1480B packets (including one IP 
header) before tunnelling.
In both cases, tunnel encap adds an extra 20B IPv4 header.
The MTU of the tunnel ingress is assumed to be 1500B.
Layer-2 headers are not counted.

I'll track 100 packets sent by each flow over the same duration in 
parallel along the same path.
But the metrics I give are accurate to decimal fractions of packets, 
even tho that's obviously not possible.

On leaving source or before arriving at tunnel ingress:

number of packets: F: 100; N: 100
number of CE marks: F: 0; N: 0
number of bytes: F: 150000; N: 148000

>     The tunnel ingress doubles the packet rate of the flows it fragments.

number of packets: F: 200; N: 100
number of CE marks: F: 0; N: 0
number of bytes: F: 154000; N: 150000

>     packets from the fragmented flows as the others {Note 1}.

Assume AQM applying 2% ECN marking

number of packets: F: 200; N: 100
number of CE marks: F: 4; N: 2
proportion of marked packets: F: 2%; N: 2%

number of bytes: F: 154000; N: 150000
number of CE bytes: F: 3080; N: 3000
proportion of marked bytes: F: 2%; N: 2%

CE-marked bytes are averages, assuming packets of any size are marked with the same probability.

>     Reassembly

After the proposed reassembly:

number of packets: F: 100; N: 100
number of CE marks: F: 2.03; N: 2
proportion of marked packets: F: 2.03%; N: 2%

number of bytes: F: 150000; N: 148000
number of CE bytes: F: 3040; N: 2960
proportion of marked bytes: F: 2.03%; N: 2%

Or after RFC3168 reassembly:

number of packets: F: 100; N: 100
number of CE marks: F: 3.96; N: 2
proportion of marked packets: F: 3.96%; N: 2%

number of bytes: F: 150000; N: 148000
number of CE bytes: F: 5940; N: 2960
proportion of marked bytes: F: 3.96%; N: 2%

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 15 Mar 2020 01:03:28 +0100

Hi Bob,

thanks! More questions below ;)

> On Mar 14, 2020, at 19:07, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> Tunnels are often constructed over the top of the network operator that runs the physical equipment (incl. any AQM). Overlays are one of the common motivations for tunnelling.

	[SM] Fair enough, my question was not that much about tunnels per se, as I can understand why tunnels are used, it really is about fragmenting/defragmenting tunnels in combination with a ECN-marking AQM. But I have not posed this question clear enough (for a change I tried to avoid being overly verbose).

> Where an operator who sets up a tunnel can control the MTU of the links, I would imagine fragmentation is highly unlikely. But for overlays using tunnelling, most tunnel endpoints use PMTUD between themselves to find a good tunnel MTU, and I would imagine they use fragmentation for IPv4 (because they have to, if their "packet too big" ICMPs are being blocked) for any incoming packets that don't fit once they add tunnel headers. But I don't know how often they actually have to fragment (depends on end-systems strategies are to keep the MTU below this point).
> 
> In this case all the packets on the link were fragments, and "A significant portion of the fragmented traffic ... is tunneled traffic." I don't know whether this link was chosen for this study because all the packets were fragments, or whether it would have been easy to find other links with a high proportion of fragments.

	[SM] I have found this as well, before asking, IMHO that is a case of a degenerate situation, where either the tunnel operator should have made sure that the tunneled path would carry MTU 1540 packets, or signaled the endpoints to reduce the payload. But I admit that there seems to exist no silver bullet that easily avoids fragmentation for IPv4...

> 
> 
> Finding the best max packet size is an area where neither IPv4 nor IPv6 has ever found a good solution.

	[SM] Question: Do you think that is because IPv4 fragmentation always allowed an easy way out here? I read a proposal once that it might have been more robust for each hop if encountering too large a packet to simply transfer a truncated packet, but I guess one would at least need to adjust the checksum (not the length field) to differentiate this from a partial packet loss on one of the transfers....

> Getting a tunnel to fragment and reassemble is indeed painfully sub-optimal,

	[SM] +1

> but all the other solutions have their own problems. It is possible the sub-optimality is often going on under-the-covers, just because it works. I do know that, for IPv4, the Don't Fragment (DF) flag is often ignored by tunnels, as a preferable alternative to just ditching the traffic.

	[SM] I know the perfect is the enemy of the good, but that seems more like one of these "one does not necessarily want to know/see how sausage is made" situation.

> 
> In the absence of the IP layer solving this problem, Gorry's group found that about 20% of end-systems are just clamping the IPv4 TCP max segment size lower than necessary. And for IPv6 most are just using the min segment size. See:
> Again only for TCP, middleboxes can edit the MSS advertised in each packet to fool the hosts into using a smaller max segment size.

	[SM] Yes, except given the issues with otherwise signaling too large packets a long a path, that might still be the cleanest solution...

> You can configure a Cisco box (and I'm sure others) to do this when you're using it for tunnelling. The above study from Gorry's group didn't find much evidence of this though.

	[SM] Thanks for the link!

> 
> PLPMTUD solves this problem end-to-end (see [RFC4821] for TCP etc. and [draft-ietf-tsvwg-datagram-plpmtud] not yet published for UDP). However, where a tunnel already solves the problem (sub-optimally) for IPv4 using fragmentation, I can't imagine that anyone would disable it, because it is still necessary and it still works.

	[SM] Well transient fragmentation (where the tunnel endpoints create the appearance of no fragmentation) comes at a considerable processing cost, and carrying fragments (especially in degenerate cases like your caida example above) skews the payload/overhead ratio badly (by artificially blowing up the packet rate), so I am confident tunnel operators would be happy to be able to drop that crutch in a heart-beat, IF they could be sure it would not be required anymore (or in a belts and suspender fashion, would keep the mechanism operational but would use PLPMTUD to make it unlikely that fragmentation actually is triggered)

> 
> Further reading.
> I just compiled a list, then realized everything is already cited here, in the context of minimizing latency:
> You could also try https://labs.ripe.net/Members/gih/evaluating-ipv4-and-ipv6-packet-fragmentation .

	[SM] Again, thanks for the references.

Best
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 10:50:44 +0000

Rod,

> Can you please site an implementaiton?

Here's a selection of implementations that allow you to configure 
clearing the DF flag in order to allow fragmentation prior to encapsulation:

  * Cisco IOS:
    https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_dplane/configuration/15-mt/sec-ipsec-data-plane-15-mt-book/sec-df-bit-ovride.html
  * Juniper JunOS:
    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/clear-dont-fragment-bit-edit-interfaces.html
  * Huawei:
    https://support.huawei.com/enterprise/en/doc/EDOC1100034238/a1bd518c/optional-configuring-the-df-flag-bit-for-gre-packets
  * Ericsson Redback:
    http://rbman.ito.expert/en_lzn7830011_1_r4a/3_19082-CRA1191170_1-V1Uen.G.html#CHAPTER1.59
  * Linux: iptables -t mangle -A PREROUTING -j DF --clear

While searching, I also found clear-df by accident in a firewall manual 
(not even doing any tunnelling)

And I even discovered "clear-df" in an (expired) individual Internet 
Draft from Ericsson for the Yang data model of an IPv4 tunnel:
draft-liu-intarea-ipipv4-tunnel-yang-02

Note, these are not to be confused with suppressing propagation of DF=1 
from the arriving IP header to the outer (so that a downstream router 
within the tunnel is allowed to fragment). In Linux, that would be: 
iptables -t mangle -A POSTROUTING -j DF --clear

So yes, I know this is often done, but I don't know how often.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 11:12:08 +0000

Sebastian,

Snipped all the comments you're agreeing with. Two responses below...

On 15/03/2020 00:03, Sebastian Moeller wrote:
>> [BB] In the absence of the IP layer solving this problem, Gorry's group found that about 20% of end-systems are just clamping the IPv4 TCP max segment size lower than necessary. And for IPv6 most are just using the min segment size. See:

[BB] I should add that PLPMTUD and clamping the max packet size 
complement each other.

PLPMTUD without clamping would kill latency.
Clamping avoids triggering fragmentation, but smaller than max packet 
sizes are inefficient.
Adding PLPMTUD eventually bumps up the MTU for each path to its max, 
removing the inefficiency.

> 	[SM] Well transient fragmentation (where the tunnel endpoints create the appearance of no fragmentation) comes at a considerable processing cost, and carrying fragments (especially in degenerate c…
[BB] Just need to point out that the /operator/ cannot use PLPMTUD to 
make triggering fragmentation unlikely. That has to be done at the 
origin sender. That's precisely why operators don't drop this crutch - 
because they can't be sure the sender has their own crutch.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 13:28:11 +0000

tsvwg,

I would like to draw the list's attention to this email, which is a 
classic example of the way Jonathan Morton spins untrue FUD (fear 
uncertainty and doubt) over L4S, even when the failure mode under 
discussion is actually a downside of SCE, which L4S was deliberately 
designed to avoid.

Whatever,... now to you Jonathan,

It's hardly necessary to be so underhand. Even without fragmentation, 
SCE markings applied within a tunnel usually won't reach the receiver 
anyway, as I've explained elsewhere. So in the sub-case of 
fragmentation, if SCE markings also don't survive reassembly, 
SCE-in-a-tunnel is just something that was already broken, getting 
broken again.

inline...

On 13/03/2020 21:22, Jonathan Morton wrote:
>>> [JM] The existing language in RFC-3168 succeeds in preserving the number of CE marks applied to a flow.  Any deficiencies we should consider are in relation to handling the distinction between ECT(1) and ECT(0), as this is what newly becomes significant with both the L4S and SCE proposals.
>> [BB]
> [JM] I'll agree with this: where the ECN codepoint is set at origin (or, equivalently, before the fragmentation point), the fragments will all carry identical codepoints and will be reassembled correctly under RFC-3168 rules.  These rules also handle the case of CE marks being set on IP fragments on the tunnel path.  The case of marking with ECT(1) is indeed left undefined, which is undesirable.

[BB] You rewrite my description of a downside of SCE in woffle-language 
that pads out the scenarios with irrelevant cases and obfuscates by 
removing all mention of the words SCE or L4S. Then you snip (reinstated 
below) where I asked you to retract your untrue assertion that L4S 
suffers from this as well.

>
>> [BB] Please confirm that this is only an issue with SCE, not L4S.
>> And please do not cast doubt over L4S by saying this affects L4S when 
>> it does not.
>
[BB] I just knew you were going to snip this. You're so predictable.

Moving on...

> [JM] I think it would be more accurate to say that IPv6 behaves as though DF is *always* set.  There is no defined mechanism for a router to perform fragmentation.  Fragmented packets may be generated *at source* to eg. convey an oversize datagram; TCP would just reduce the MSS to fit, when notified of a reduced MTU on the path.  So I don't consider IPv6 fragmentation to be a significant problem.

[BB] You've conveniently side-stepped oversize UDP datagrams over IPv6, 
where the IPv6 source has to do the fragmentation, and IPv6 reassembly 
logic at the destination would need *new logic* to propagate the SCE 
markings.

I drew attention to this in the IPv6 section of the email below (which 
you didn't respond to).

>> Until now, I thought that, at least in QUIC, SCE would get feedback without changing the receiver. But, if the sender is fragmenting, the receiver's IPv6 layer will need to be changed as well. It'…

[BB]
QUIC: 
https://tools.ietf.org/html/draft-ietf-quic-transport-27#section-14.1 
recommends (but does not mandate) that QUIC implementations clamp the 
max datagram size. Ideally it recommends using PLPMTUD as well.

The latest draft on RTP feedback over UDP mentions MTU issues but is not 
as concrete as QUIC:
https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message

> ISTR, at some point in the past, interim language was suggested which would require taking the ECN codepoint from one of the fragments constituting the packet, with the behaviour being otherwise un…

[BB] The language was in the previous revision:
https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-09#section-5

It was for PCN (pre-congestion notification), which is a standards track 
set of IETF RFCs that were implemented but not deployed, AFAICT. I 
removed all that text because you objected to the same language for CE. 
I generalized it into a requirement to achieve an externally observable 
effect, rather than saying how to do it.

I would be happy to reinstate it. But you can't have it both ways: you 
want standards track RFCs to cater for SCE, which isn't even chartered 
IETF work, but not to cater for other standards track RFCs.

You do not seem to be aware of the other WGs and other standards bodies 
(3GPP, IEEE802.1) that have been involved in this draft, and who are 
waiting for it to go to RFC (including an RFC being held in the RFC 
Editor's queue).

> An implementation which performs a bitwise-OR across the ECN fields of the fragments would effectively convert partial SCE marks into CE marks (as the ECT codepoints are 01 and 10).  This is less t…

[BB] We're talking about a standards track RFC-to-be here. You're asking 
for it to includes requirements to support SCE that isn't even chartered 
IETF work. If you want the IETF to give a signal to implementers of 
large numbers of existing layering protocols to update their code for 
you, you have to do a lot of work convincing people first.

> RFC-3168's existing recommendation to set DF seems like a good one to me, and is effectively automatic with IPv6.  Tunnels which perform outer fragmentation should be fixed to implement MTU Discove…
[BB] If you mean MTU Discovery between the tunnel endpoints, that 
doesn't help when a packet arrives that is too large for the discovered 
MTU. So you must mean supporting MTU Discovery by the endpoints (by the 
tunnel ingress sending a PTB ICMP). But most networks black-hole these 
nowadays.

So I can only conclude that you're masquerading as an expert again, and 
we should just move on. I wouldn't mind if you couched your objections 
with "I'm not sure but,..." or whatever.

Anyway, this is not how IETF RFCs work. They have to specify what to 
implement, in order to interoperate with what exists. Not what 
implementers and operators should have done. The only grey area is when 
what exists is non-compliant. But if non-compliance is widespread, it 
has to be catered for.

> The list of 14 appears to be missing.

[BB] Sorry [got distracted]:
https://datatracker.ietf.org/meeting/101/materials/slides-101-tsvwg-sessb-35-ecn-transport-00#page=4

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 15 Mar 2020 18:15:47 +0200

I had hoped that a thread with this title represented an opportunity to discuss the effects of fragmentation on L4S and SCE, *without* immediately descending into tribalism and ad-hominem attacks.  Accordingly, I will decline to quote *any* of Bob Briscoe's latest post, in favour of summarising the technical content.

Firstly, we have apparently agreed that fragmentation has no deleterious effects on ECN codepoints applied prior to fragmentation.  I do not understand why I am being attacked for expressing that agreement.

On the subject of IPv6 fragmentation, Bob previously asserted that RFC-3168 has nothing to say about IPv6.  This is demonstrably false by a simple text search over RFC-3168, in which the IPv6 Traffic Class Octet is repeatedly equated with the IPv4 TOS Byte.  In §5.3 dealing with fragmentation, neither IPv4 nor IPv6 are explicitly mentioned, implying that the language therein applies equally to both.  I have not personally verified that any given IPv6 receiver correctly implements these rules, but I would expect that they generally do.

So the only material difference is that IPv6 behaves as if the DF (Don't Fragment) bit is always set, which limits the scope of the problem at hand (for most transports) to the case of an on-path tunnel which performs outer fragmentation.  The only exception is when a jumbo datagram needs to be sent, but most such protocols are not ECTs (ECN Capable Transports) and are therefore out of scope for this discussion.  We can of course still discuss any specific counterexamples that might be identified.

Most ECTs are expected to set DF (on IPv4) and implement PMTUD.  Indeed, most do.  If certain tunnel implementations interact badly with PMTUD, I think that is primarily a problem for the tunnel, not the transport.  It is still desirable that the transport functions reasonably well in difficult circumstances, and that is part of what I hope to discuss in this thread.

That is not true, however.  What *is* present there is language which strongly advocates propagating marking on a byte-preserving basis, both for CE and ECT(1) codepoints, and that is what I have objected to, both recently and less recently.  What I referred to above is much simpler and less prescriptive, and can be summarised in the following proposed language updating RFC-3168 §5.3:

> …if any fragment of an IP packet to be reassembled has the Not-ECT codepoint set, then one of two actions MUST be taken:
> 
>  * Set the Not-ECT codepoint on the reassembled packet.  However, this MUST NOT occur if any of the other fragments contributing to this reassembly carries any ECN codepoint other than Not-ECT.
> 
>  * The packet is dropped.
> 
> If both actions are applicable, either MAY be chosen.
> 
> Reassembly of a fragmented packet MUST NOT change the ECN codepoint when all of the fragments carry the same codepoint.
> 
> Notwithstanding the above, if any fragment of an IP packet to be reassembled has the CE codepoint set, then one of two actions MUST be taken:
> 
>  * Set the CE codepoint on the reassembled packet.
> 
>  * The packet is dropped.
> 
> If neither the Not-ECT nor the CE codepoints appear on any of the fragments contributing to this reassembly, then the ECN codepoint set on the reassembled packet SHOULD be one of the ECN codepoints present on one of the fragments.  Any contributing fragment MAY be chosen as the source.

I'm sure some wordsmithing is in order on the above, but I hope this makes my position clearer.

In connection with this, Bob claims I'm asking for requirements to support SCE in a standards-track document.  If that is what I was actually asking for, then the final sentence in the above proposed language would be markedly different.  I would be asking for the existing requirement in RFC-3168 §5.3 that congestion information MUST NOT be lost during reassembly be interpreted to treat ECT(1) codepoints as congestion information.

Mindful of SCE's current status in the WG, I am deliberately *not* asking for that at this time, even though it would improve the quality of SCE signalling over tunnelled paths.  Instead, I am asking for simple, easily-implemented language which is approximately neutral between L4S and SCE, but allows for the possibility of using ECT(1) as an output from the network at some point in the future.

Finally, I thank Bob for the link to a two-year-old slide deck containing a survey of existing tunnel protocols.  I infer from this that shimmed tunnels have typically considered RFC-6040 as out of scope, as the latter only explicitly covers tunnels without a shim layer.  But the rules in RFC-6040 for handling ECN are generally applicable, I think, and have nothing whatsoever to do with fragmentation.  This latter fact informs my support for the suggestion to separate the concerns of reassembly and decapsulation.

 - Jonathan Morton

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sun, 15 Mar 2020 10:30:28 -0700 (PDT)

> clearing the DF flag in order to allow fragmentation prior to encapsulation:

Thank you for this Bob, that clears up things a fair bit.
Describing "configurable clearing of DF" as "often ignored
by tunnels..." is what lead to my questions.

> So yes, I know this is often done, but I don't know how often.

Probably more often than desireable.
BUT this is done intentionally by an Operator, so I do not view
this as a severe a problem as if there actually was an implementation
that just flat out ignored DF.

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 18:14:31 +0000

Jonathan,

> I had hoped that a thread with this title represented an opportunity to discuss the effects of fragmentation on L4S and SCE, *without* immediately descending into tribalism and ad-hominem attacks.…

[BB] Readers on this list will see that you've used this as an excuse to 
snip (and not address) the cases where SCE doesn't work (again).

When you are indulging in attention deflection tactics and get called 
out, attacking the person who called you out is even less polite than 
the original behaviour.

> Firstly, we have apparently agreed that fragmentation has no deleterious effects on ECN codepoints applied prior to fragmentation.  I do not understand why I am being attacked for expressing that a…

[BB] I asked to retract the untruthful statement you had made that L4S 
was equally deficient in this respect. You were being called out (not 
attacked) for snipping that request.

>
> On the subject of IPv6 fragmentation, Bob previously asserted that RFC-3168 has nothing to say about IPv6.

[BB] Bob actually said RFC3168 "does not seem to recognize that IPv6 
*e2e fragmentation* exists."

> This is demonstrably false by a simple text search over RFC-3168, in which the IPv6 Traffic Class Octet is repeatedly equated with the IPv4 TOS Byte.  In §5.3 dealing with fragmentation, neither IPv4 nor IPv6 are explicitly mentioned, implying that the language therein applies equally to both.  I have not personally verified that any given IPv6 receiver correctly implements these rules, but I would expect that they generally do.

[BB] You would not be able to verify anything, because RFC3168 offers no 
rules to verify. It explicitly says that it offers no rules for how to 
reassemble a mixture of ECT(0) and ECT(1) fragments.

The remedy that RFC3168 offers for the lack of guidance on how to 
reassemble a mixture of ECT(0) and ECT(1) fragments is to set DF. No 
remedy is offered for IPv6 e2e fragmentation, which has no DF flag.

> So the only material difference is that IPv6 behaves as if the DF (Don't Fragment) bit is always set, which limits the scope of the problem at hand (for most transports) to the case of an on-path t…

[BB] The additional important exception that Bob highlighted is where 
UDP datagrams are larger that the MTU, in which case IPv6 has to do e2e 
fragmentation and reassembly.

If you snip these two points again without addressing them, I will 
assume everyone on the list will notice. So I will not bother to ask the 
questions again.

> That is not true, however.  What *is* present there is language which strongly advocates propagating marking on a byte-preserving basis, both for CE and ECT(1) codepoints, and that is what I have o…

[BB] I thought your "interim language was suggested" phrase meant this 
text was in the draft. Hence why I thought you were referring to the 
byte-preserving text. It would have been straightforward for you to 
point to the place where you suggested some text.

As I said, in the latest revision, I took out the byte-preserving 
language at your request (i.e. this text *is not* present), and have 
since argued for it on the list instead, and only included least common 
denominator text in the draft.

> I'm sure some wordsmithing is in order on the above, but I hope this makes my position clearer.

[BB] I (and David Black) have argued that a separate draft is needed to 
update fragment reassembly in RFC3168. Can you please accept that we are 
not going to update ECN fragmentation and reassembly in 
rfc6040update-shim. Then it can unblock and be published.

rfc6040update-shim is meant to be about encap and decap. It would allow 
this draft to become unstuck if it just said fragmentation and 
re-assembly is out of scope (which it is).

> Mindful of SCE's current status in the WG, I am deliberately *not* asking for that at this time, even though it would improve the quality of SCE signalling over tunnelled paths.  Instead, I am aski…

[BB] Aside from the question of whether and where we update reassembly 
text, your suggested text would be incompatible with the standards track 
PCN RFCs. Whereas the previous text in the draft was compatible with 
PCN, and happened to support SCE as well.

I am aware that you haven't got your head around why byte-preserving 
produces a correct result for all of RFC3168, SCE and L4S style 
signalling. But let's continue that discussion in the other thread.

Bob

---

From: Joseph Touch <[email-masked]>
Date: Sun, 15 Mar 2020 12:50:34 -0700

> On Mar 15, 2020, at 10:30 AM, Rodney W. Grimes <[email-masked]> wrote:
> 

See RFC6864 as to why this is risky.

Yes, it makes things look like they work. What the operators don’t see are the packets they corrupt by ignoring DF, e.g., when the ID is repeated or just constant.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 15 Mar 2020 12:53:47 -0700

> [BB] I should add that PLPMTUD and clamping the max packet size complement each other. 

Except that PLPMTUD works over authenticated or encrypted transport headers and clamping does not.

Clamping could be useful if the information were in an IP header that could be ignored by routers that don’t care (e.g, by not slowing them down, as IP options do).

Joe

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 16 Mar 2020 01:35:54 +0200

Isolating one key point:

> On 15 Mar, 2020, at 8:14 pm, Bob Briscoe <[email-masked]> wrote:
> 
> I (and David Black) have argued that a separate draft is needed to update fragment reassembly in RFC3168. Can you please accept that we are not going to update ECN fragmentation and reassembly in rfc6040update-shim. Then it can unblock and be published.
> 
> rfc6040update-shim is meant to be about encap and decap. It would allow this draft to become unstuck if it just said fragmentation and re-assembly is out of scope (which it is). 

I have said several times in the past few days that I agree with that approach, and I'm sure David Black is well aware of that.

However, the -10 revision (which appears to be current) still contains the section and some language covering fragmentation and reassembly.  To be consistent with this approach, you need to update it to simply refer to RFC-3168 for the relevant semantics.  Then I think it will be possible to make progress.

We can discuss separately what, if anything, should be done to RFC-3168 on this topic.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 16 Mar 2020 08:47:33 +0000

Joe,

On 15/03/2020 19:53, Joseph Touch wrote:
> Clamping could be useful if the information were in an IP header that 
> could be ignored by routers that don’t care (e.g, by not slowing them 
> down, as IP options do).

[BB] I think I see what the terminology problem is here.

I was using the term MSS clamping from this measurement study by Gorry's 
group:
https://iain.learmonth.me/stuff/pubs/UsableMTU2018.pdf
but it uses the same term whether networks do the clamping or 
end-systems do. In my sentence that you've quoted, I was using it to 
mean the sender limiting the size of the IP payload to a lower value 
than could have actually traversed the path. Sorry for not making that 
clear.

Also, in case it wasn't clear, when I said clamping and PLPMTUD 
complement each other, I meant the sender initially clamping the payload 
size, then subsequently using PLPMTUD and removing its clamp to use a 
larger, more efficient packet size instead.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 16 Mar 2020 08:50:43 +0000

Jonathan, [David, this one's for you too]

> However, the -10 revision (which appears to be current) still contains the section and some language covering fragmentation and reassembly.  To be consistent with this approach, you need to update…
I am waiting for David on that. After his email asking me to reaffirm 
RFC3168 behaviour in ecn-encap and rfc6040update-shim, I said I'd rather 
just say fragmentation is out of scope. So I'm waiting for David's 
response as doc shepherd before I commit any changes to the draft.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 16 Mar 2020 10:12:44 +0100

Hi Bob,

> On Mar 15, 2020, at 12:12, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> Clamping avoids triggering fragmentation, but smaller than max packet sizes are inefficient. 
> Adding PLPMTUD eventually bumps up the MTU for each path to its max, removing the inefficiency.

	[SM] Well, realistically the ONLY viable solution is to design all tunnels to allow the de facto real-world internet MTU of 1500 and just make sure the tunnel itself uses jumbo-frame capable links to make sure fragmentation is not required. But honestly, in our current context I still wonder whether an operator failing to implement a non-fragmenting tunnel will ever deploy an ECN-enabled AQM...

> [BB] Just need to point out that the /operator/ cannot use PLPMTUD to make triggering fragmentation unlikely.

	[SM] The operator can however make sure its tunnels can transport ~1500 Byte ethernet payload equivalent payloads, at which points in all likelihood the tunnel will not cause additional fragmentation and the whole issue (if an ECN AQM is operatin on the tunneled packets) becomes some one else's problem.

> That has to be done at the origin sender. That's precisely why operators don't drop this crutch - because they can't be sure the sender has their own crutch.

	[SM] The operator could just drop packets if they would require fragmentation... or functionally make a fragmentation requirement exceedingly unlikely... But I accept that that is a log standing issue that probably has no solution that is all three of simple/robust, cheap and fully backward compatible...

But I do not see this as a big problem for either L4S or SCE as none of these tunnels currently uses an ECN-AQM there is next to zero legacy and hence tunnel operators can be taught to properly deal with ECN bits at the defragmentation and decapsulation stages. Sure, L4S seems coved by the existing rules (but assumes that defragmenter/decapsulators actually follow these rules!) but the rule changes for SCE seem rather straight forward to me.

BTW, I am puzzled about your claim, that L4S was carefully designed to avoid this issue for two reasons:
a) while according to the current RFCs CE-marks from a fragmented outer tunnel should propagate correctly into the inner packets ECN field on decapsulation there is little proof that all deployed tunnel defragmenting decapsulators actually do so correctly, and

b) this seems less of a conscious design decision and more a fortunate side-effect of selecting to change the meaning of CE, as far as I can see the strongest rationale for doing that was to be able to just hoist the DCTCP work into the common internet. Or do you claim that you opted to overload the CE codepoint specifically to deal wit fragmenting tunnels with ECN-using AQMs?

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 16 Mar 2020 10:19:27 +0100

Hi Bob,

> On Mar 15, 2020, at 11:50, Bob Briscoe <[email-masked]> wrote:
> 
> Rod,
> 
> Here's a selection of implementations that allow you to configure clearing the DF flag in order to allow fragmentation prior to encapsulation:
> 	• Cisco IOS: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_dplane/configuration/15-mt/sec-ipsec-data-plane-15-mt-book/sec-df-bit-ovride.html
> 	• Juniper JunOS: https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/clear-dont-fragment-bit-edit-interfaces.html
> 	• Huawei: https://support.huawei.com/enterprise/en/doc/EDOC1100034238/a1bd518c/optional-configuring-the-df-flag-bit-for-gre-packets
> 	• Ericsson Redback: http://rbman.ito.expert/en_lzn7830011_1_r4a/3_19082-CRA1191170_1-V1Uen.G.html#CHAPTER1.59
> 	• Linux: iptables -t mangle -A PREROUTING -j DF --clear
> While searching, I also found clear-df by accident in a firewall manual (not even doing any tunnelling)

	[SM] Isn't that a gross violation of the end-to-end principle? Or is the interpretation of the DF flag here, do not deliver fragments to the end-points, but anything transparent is fair game?

> 
> And I even discovered "clear-df" in an (expired) individual Internet Draft from Ericsson for the Yang data model of an IPv4 tunnel: 

	[SM] That might be use a "cover all bases" approach without actually endorsing any such use (let's try to model actual in-the-field-reality compared to RFC-reality).

> 
> Note, these are not to be confused with suppressing propagation of DF=1 from the arriving IP header to the outer (so that a downstream router within the tunnel is allowed to fragment). In Linux, that would be: iptables -t mangle -A POSTROUTING -j DF --clear
> 

	[SM] Bigger question, should the IETF actually endorse something like that?

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 16 Mar 2020 10:54:20 +0100

Dear Bob, dear list,

I think it is time for us to take a deep breath, a step back and to not assign to malice what can be equally explained by simple misunderstanding(s). I believe this discussion while heated at times is done in good faith by all parties, and it does not encourage the quality and civility of the discourse to try to assign a bad faith effort to the respective other side. So how about we (including myself here) try harder to be nicer to each other even while discussing wildly different positions.

I will add nothing material to the discussion here, since this is about improving the tone...

Best Regards
	Sebastian

> On Mar 15, 2020, at 14:28, Bob Briscoe <[email-masked]> wrote:
> 
> tsvwg,
> 
> I would like to draw the list's attention to this email, which is a classic example of the way Jonathan Morton spins untrue FUD (fear uncertainty and doubt) over L4S, even when the failure mode under discussion is actually a downside of SCE, which L4S was deliberately designed to avoid. 
> 
> 
> It's hardly necessary to be so underhand. Even without fragmentation, SCE markings applied within a tunnel usually won't reach the receiver anyway, as I've explained elsewhere. So in the sub-case of fragmentation, if SCE markings also don't survive reassembly, SCE-in-a-tunnel is just something that was already broken, getting broken again.
> 
> 
> inline...
> 
> [BB] You rewrite my description of a downside of SCE in woffle-language that pads out the scenarios with irrelevant cases and obfuscates by removing all mention of the words SCE or L4S. Then you snip (reinstated below) where I asked you to retract your untrue assertion that L4S suffers from this as well. 
> 
>> 
>>> And please do not cast doubt over L4S by saying this affects L4S when it does not.
>> 
> [BB] You've conveniently side-stepped oversize UDP datagrams over IPv6, where the IPv6 source has to do the fragmentation, and IPv6 reassembly logic at the destination would need new logic to propagate the SCE markings.
> 
> I drew attention to this in the IPv6 section of the email below (which you didn't respond to).
> 
>> 
> QUIC: https://tools.ietf.org/html/draft-ietf-quic-transport-27#section-14.1 recommends (but does not mandate) that QUIC implementations clamp the max datagram size. Ideally it recommends using PLPMTUD as well. 
> 
> The latest draft on RTP feedback over UDP mentions MTU issues but is not as concrete as QUIC:
> It was for PCN (pre-congestion notification), which is a standards track set of IETF RFCs that were implemented but not deployed, AFAICT. I removed all that text because you objected to the same language for CE. I generalized it into a requirement to achieve an externally observable effect, rather than saying how to do it.
> 
> I would be happy to reinstate it. But you can't have it both ways: you want standards track RFCs to cater for SCE, which isn't even chartered IETF work, but not to cater for other standards track RFCs.
> 
> You do not seem to be aware of the other WGs and other standards bodies (3GPP, IEEE802.1) that have been involved in this draft, and who are waiting for it to go to RFC (including an RFC being held in the RFC Editor's queue).
> 
>> 
> [BB] We're talking about a standards track RFC-to-be here. You're asking for it to includes requirements to support SCE that isn't even chartered IETF work. If you want the IETF to give a signal to implementers of large numbers of existing layering protocols to update their code for you, you have to do a lot of work convincing people first.
> 
>> 
> [BB] If you mean MTU Discovery between the tunnel endpoints, that doesn't help when a packet arrives that is too large for the discovered MTU. So you must mean supporting MTU Discovery by the endpoints (by the tunnel ingress sending a PTB ICMP). But most networks black-hole these nowadays. 
> 
> So I can only conclude that you're masquerading as an expert again, and we should just move on. I wouldn't mind if you couched your objections with "I'm not sure but,..." or whatever.  
> 
> Anyway, this is not how IETF RFCs work. They have to specify what to implement, in order to interoperate with what exists. Not what implementers and operators should have done. The only grey area is when what exists is non-compliant. But if non-compliance is widespread, it has to be catered for.
> 
>>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 16 Mar 2020 15:03:49 +0200

> On 16 Mar, 2020, at 11:12 am, Sebastian Moeller <[email-masked]> wrote:
> 

I would assume that at least some tunnels traverse networks not under the administrative control of either endpoint.  There may be differences in competency between network administrations.

So I think it's safer to assume that the *physical* network tops out at 1500 bytes MTU, tunnels may therefore only support traffic with a smaller MTU, and that ECN AQMs may be deployed anywhere.

 - Jonathan Morton

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 16 Mar 2020 10:18:55 -0400

Chair hat on.

This is a chance to remind everyone on the list of RFC 7154 
https://tools.ietf.org/html/rfc7154. Everyone participating in TSVWG 
should treat others with respect, and keep discussions impersonal.

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 08:35:39 -0700

Hi, Bob,

> On Mar 16, 2020, at 1:47 AM, Bob Briscoe <[email-masked]> wrote:
> 
> Joe,
> 
> I was using the term MSS clamping from this measurement study by Gorry's group:
>     https://iain.learmonth.me/stuff/pubs/UsableMTU2018.pdf <https://iain.learmonth.me/stuff/pubs/UsableMTU2018.pdf> 
> but it uses the same term whether networks do the clamping or end-systems do. In my sentence that you've quoted, I was using it to mean the sender limiting the size of the IP payload to a lower value than could have actually traversed the path. Sorry for not making that clear.

I believe Gorry’s paper is talking about net devices, though:

> MSS Clamping is also not without issues: we found a number of misconfigured middleboxes  

I don’t know if there’s a term for endpoints pulling down the endpoint MTU in other ways.

> Also, in case it wasn't clear, when I said clamping and PLPMTUD complement each other, I meant the sender initially clamping the payload size, then subsequently using PLPMTUD and removing its clamp to use a larger, more efficient packet size instead.

It may be useful to explain that if you include the complement issue.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 08:38:34 -0700

> On Mar 16, 2020, at 2:19 AM, Sebastian Moeller <[email-masked]> wrote:
> 

If you’re referring to not propagating tunnel info between inner and outer, draft-ietf-intarea-tunnels explains how these headers are and are not related.

If you’re referring to ignoring DF, we already have at least one doc that explains the hazards of that (RFC6864).

Joe

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 08:41:50 -0700

> On Mar 16, 2020, at 2:12 AM, Sebastian Moeller <[email-masked]> wrote:
> 

That won’t work either. Any tunnel that has an MTU of any type (e.g., even the max MTU for IP of 64KB) and can be used recursively either directly (X over X) or indirectly (X over Y over … over X) MUST support fragmentation somewhere.

Intermediate recursion tunnels can implement fragmentation at any of the protocols. Directly recursive tunnels MUST implement fragmentation (because nobody else can in way that helps).

Bob’s doc has the (IMO) correct approach - tunnels are a pair of devices that both do work.

Joe

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 16 Mar 2020 14:30:11 -0700 (PDT)

> That won?t work either. Any tunnel that has an MTU of any type (e.g., even the max MTU for IP of 64KB) and can be used recursively either directly (X over X) or indirectly (X over Y over ? over X) MUST support fragmentation somewhere.

This is partially true, so long as the configuration of the networks that cause traffic to enter the inner tunnels use a proper MTU for that inner layer.  This is a common problem with IPSec over PPPoE.  As long as you configure the sources(sender) to use a inner tunnel MTU no issue occurs.  This is infact a very common situation with VPS's, often leading to end nodes with MTU's in the 14xx range.  In theory one could nest IPSec until the inner most MTU became the IPvX minimal MTU and so long as the inner most tunnel interface device had the properly adjusted MTU property no fragmentation would occur at any layer in the recursive tunnels.

> Intermediate recursion tunnels can implement fragmentation at any of the protocols. Directly recursive tunnels MUST implement fragmentation (because nobody else can in way that helps).

Again not always true.  One can properly configure things such that no fragmentation on the inner tunnel occurs by applying MTU's at the ends (usually the encap/decap points).

> Bob?s doc has the (IMO) correct approach - tunnels are a pair of devices that both do work.

This would be ideal, but even in todays internet Tunnels are riddled with MTU issues.

Further note my assertions above do not apply to arbitrary tunnels between unknown end systems, only to tunnels between known end systems, which I believe is the norm.

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 14:38:34 -0700

> On Mar 16, 2020, at 2:30 PM, Rodney W. Grimes <[email-masked]> wrote:
> 
> This is partially true, so long as the configuration of the networks that cause traffic to enter the inner tunnels use a proper MTU for that inner layer.  

That’s not always possible. If the net runs over minimal IPv6 links, there’s no way to ensure a traffic can support 1280B packets over 1280B tunnels.

> This is a common problem with IPSec over PPPoE.  As long as you configure the sources(sender) to use a inner tunnel MTU no issue occurs.

You don’t always know where or when these are and doing this isn’t possible in some cases (see above; you’d end up claiming you support IPv6 but not supporting the min required MTU).

>  This is infact a very common situation with VPS's, often leading to end nodes with MTU's in the 14xx range.  In theory one could nest IPSec until the inner most MTU became the IPvX minimal MTU and so long as the inner most tunnel interface device had the properly adjusted MTU property no fragmentation would occur at any layer in the recursive tunnels.
> 
>> 

See above for a counterexample.

> This would be ideal, but even in todays internet Tunnels are riddled with MTU issues.

IMO, largely because of mistakes as were quoted before - e.g., ignoring DF and trying to fragment the arriving packet rather than fragment/reassembly at the tunnel edges.

> Further note my assertions above do not apply to arbitrary tunnels between unknown end systems, only to tunnels between known end systems, which I believe is the norm.

I don’t know what this refers to; a tunnel is often between endpoints that don’t do a lot to coordinate in advance other than know they are there.

Joe

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 16 Mar 2020 23:42:11 +0200

> On 16 Mar, 2020, at 11:38 pm, Joseph Touch <[email-masked]> wrote:
> 

I think the 1280B minimum was intended to allow for the possibility of running through tunnels on conventional 1500B physical networks.

 - Jonathan Morton

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 14:44:18 -0700

> On Mar 16, 2020, at 2:42 PM, Jonathan Morton <[email-masked]> wrote:
> 

Actually, the 1280B is the path min; the receiver min is 1500B exactly to allow for this sort of frag/reassembly (where a tunnel egress *is* such a receiver). The 1500B comes from Ethernet; the 1280B is smaller deliberately.

Joe

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 16 Mar 2020 20:09:38 -0700 (PDT)

> >> That?s not always possible. If the net runs over minimal IPv6 links, there?s no way to ensure a traffic can support 1280B packets over 1280B tunnels.

True, not always possible, but very often practical and done as minimal IPv6 links generally only exist inside of tunneled tunnels :-)  ((Ie, GRE -> IPSec -> PPPoE))

> Actually, the 1280B is the path min; the receiver min is 1500B exactly to allow for this sort of frag/reassembly (where a tunnel egress *is* such a receiver). The 1500B comes from Ethernet; the 128…

receiver min?  I am not familiar with that term.

And as you state 1500B no doubt comes from the Ethernet specification.

I shall endorse Jonathans position with this:
	The 1280 IPv6 minMTU originated from a November 14, 1997 mailing
	from Steve Deering to the IPng mailing list, which stated:
	"In the ipngwg meeting in Munich, I proposed increasing the IPv6
        minimum MTU from 576 bytes to something closer to the Ethernet MTU
        of 1500 bytes, (i.e., 1500 minus room for a couple layers of
        encapsulating headers, so that min- MTU-size packets that are
        tunneled across 1500-byte-MTU paths won't be subject to
        fragmentation/reassembly on ingress/egress from the tunnels, in most
        cases).

So infact this is all meant to MINIMIZE fragmentation.

---

From: Joseph Touch <[email-masked]>
Date: Mon, 16 Mar 2020 21:03:26 -0700

> On Mar 16, 2020, at 8:09 PM, Rodney W. Grimes <[email-masked]> wrote:
> 
>> 
>> 

EMTU_R as defined in RFC 1122. See also draft-ietf-intarea-tunnels

> So infact this is all meant to MINIMIZE fragmentation.

Except that it doesn’t unless you KNOW that the path supports 1500B. But there’s no rule in IPv6 that the EMTU_S min is 1280B *AND* the links MUST support 1500B.

I.e., that’s why the EMTU_S min is 1500B - so you can reassemble when you don’t know otherwise. But even when IPv6 was created there were problems with PMTUD.

Joe

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 16 Mar 2020 22:21:41 -0700 (PDT)

> EMTU_R as defined in RFC 1122. See also draft-ietf-intarea-tunnels

EMTU_R as defined in RFC 1122 is greater than or equal to 576, not 1500 as you stated.

> Except that it doesn?t unless you KNOW that the path supports 1500B. But there?s no rule in IPv6 that the EMTU_S min is 1280B *AND* the links MUST support 1500B.
> 
> I.e., that?s why the EMTU_S min is 1500B - so you can reassemble when you don?t know otherwise. But even when IPv6 was created there were problems with PMTUD.

EMTU_S min is not 1500B, and infact 1122 says:
	It is generally desirable to avoid local fragmentation and to
	choose EMTU_S low enough to avoid fragmentation in any gateway
	along the path.  In the absence of actual knowledge of the
	minimum MTU along the path, the IP layer SHOULD use
	EMTU_S <= 576 whenever the destination address is not on a
	connected network, and otherwise use the connected network's

Again the intent here appears to be to avoid framentation along the path,
and iirc is what lead the the path MTU being cached by the BSD kernels
route table code, which is one mechansim used by people to deal with
the path MTU problem and tunnels.

---

From: Joe Touch <[email-masked]>
Date: Mon, 16 Mar 2020 23:13:56 -0700

> On Mar 16, 2020, at 10:22 PM, Rodney W. Grimes <[email-masked]> wrote:
> 
> 
>> 

The term is defined originally in RFC1122. The 576 value applies to IPv4. It is 1500B in RFC8200 for IPv6.

> the path MTU problem and tunnels.

Nobody is arguing that fragmentation should be avoided where possible. However, there’s no way to avoid it at a tunnel ingress over the tunnel path unless somehow the ingress KNOWS that EMTU_S can be 1500b. It’s normally 1280 for IPv6. It’s only larger when the tunnel path is engineered that way over known links or when the ingress runs PLPMTUD. 

The choice of 1280 allows for the POTENTIAL to avoid fragmentation *IF* the path is known engineered for 1500B or PLPMTUD runs between ingress and egress. But that choice doesn’t ENSURE fragmentation is avoided.

Joe

---

From: John Leslie <[email-masked]>
Date: Tue, 17 Mar 2020 02:10:25 -0400

On Mon, Mar 16, 2020 at 08:41:50AM -0700, Joseph Touch wrote:
>> [SM] Well, realistically the ONLY viable solution is to design all
>> tunnels to allow the de facto real-world internet MTU of 1500 and
>> just make sure the tunnel itself uses jumbo-frame capable links
>> to make sure fragmentation is not required...

   Never say "never!" (isn't that what "ONLY" means?)

   Yes, jumbo-frame capable links really _are_ essential...

>> But honestly, in our current context I still wonder whether an operator
>> failing to implement a non-fragmenting tunnel will ever deploy an
>> ECN-enabled AQM...

   Again, never say "never".

   Implementing non-fragmenting tunnels really isn't that hard! It's
just that so few people act as if they beleieve it necessary. :^(

> That won???t work either. Any tunnel that has an MTU of any type
> (e.g., even the max MTU for IP of 64KB) and can be used recursively
> either directly (X over X) or indirectly (X over Y over ??? over X)
> MUST support fragmentation somewhere.

   Yes, even 64KB MTU can be exhausted!

   We need better tunneling algorithms...

> Intermediate recursion tunnels can implement fragmentation at any
> of the protocols. Directly recursive tunnels MUST implement
> fragmentation (because nobody else can in way that helps).

   I'm not sure this is the best way to say it.

   The word "fragmentation" carries an awful lot of baggage. :^(

   I will own up to having done some _evil_ things in a past life,
such as running slow links with MTU less than 1500 to simplify life.

   We need descriptions of problems that come from running tunnels
without a tunnel-path MTU greater than the incoming MTU.

   (Of course there are things you can do as an alternative; but
they tend to be harder.)

   Where we get in trouble is pretending the problem can be ignored.
Alas, _many_ pointy-haired-bosses believe they can define the problem
out of existence. :^( :^(

> Bob???s doc has the (IMO) correct approach - tunnels are a pair of
> devices that both do work.
> 
> Joe

--
John Leslie <[email-masked]>

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 17 Mar 2020 07:56:23 +0000

Hi Bob,

> http://bobbriscoe.net/projects/netsvc_i-f/consig/encap/ecn-reassembly.pdf

Thanks for clarifying, I think your message confirms the speculation Jonathan
and I landed on as the source of the misunderstanding.

As I said there, it turns out this was just about semantics, not a disagreement
about what's happening on the wire, and so I don't think the point needs more
discussion, but I just wanted to confirm I believe I now understand what you
meant, and your message reinforced that belief.

And thanks for the schema diagram, I was able to see it and it was helpful.

As I tried to say, I think the reason I got confused about your meaning was
that I thought separating the number and proportion of CE-marked packets into
2 values would only make sense for 2 values that were able to vary differently
from each other, and so thought you were saying by "number" (as a meaningfully
different value from "proportion") that the reassembly process was increasing
the report to the receiver of the number of CE marks actually emitted by the
network.

I agree that when fragmented and getting a fixed proportion of marked packets
with an even random distribution of markings, it obviously increases the number
of marked packets relative to an equal unfragmented stream.  Although I wouldn't
have thought to phrase it as "both number and proportion" since it's so nearly
the same thing as just "proportion", I acknowledge that your phrasing is not an
invalid construction, and that for some reasonable AQMs, you're accurately
describing a sense in which the number of marks can be increased due to
fragmentation.

I hope you likewise agree that the logical-OR reassembly doesn't ever increase
the count of CE marks seen by the receiver to a value above the count of CE
marks actually emitted by the network (and for threshold-based systems often
reduces that number, since fragments will often be marked together), and that
the concern you were addressing comes because fragmentation provides more
opportunities to the network to emit CE marks, not because logical-OR reassembly
can increase the raw count of CE marks actually delivered.  And also that you
can see this as a reasonable interpretation of "number of CE-marked packets"
that can reasonably result in confusion.

Best,
Jake

PS: I also think I understand the proposal outlined in section 5 of update-shim-10,
but I think allowing generalized behaviors like that text tried to do, in pursuit
of preserving the proportion of marking instead of the fact of marks on a packet,
requires a much more precise and careful treatment than the one outlined there, and
would be better done as a separate proposal if it's to be done at all (I agree with
Jonathan that it's not clear it's worthwhile).

The sample algorithm permitted by that text clearly would require a 3168 update,
but also makes some very specific assumptions about the AQM's behavior that may
not be justified without specific knowledge of the network.

It's not hard to imagine a situation where the later fragment(s) of a packet are
more likely to receive a mark (for instance if the later fragments are back to back
more often than the head, so that later fragments arrive with a systematically
fuller queue).

If that's the case, examining only the head of the fragment would systematically
suppress delivery of CE marks in fragmented packets, because of the violated
assumption about an even probability distribution for marks.

This further seems like it could give perverse throughput incentives to degenerate
behaviors like fragmenting packets at the sender, to get the benefits of reduced loss
from ECN but receive fewer marks, even though the network might emit more, and thus
pushing out unfragmented competing flows.  So the specific example algorithm given
seems like an unsafe or at least counterproductive thing to encourage reassemblers
to do.

But that's a discussion for another thread, really.

---

From: "Black, David" <[email-masked]>
Date: Thu, 19 Mar 2020 02:03:16 +0000

Internal Use - Confidential

> > However, the -10 revision (which appears to be current) still contains the
> > section and some language covering fragmentation and reassembly.  To be
> > consistent with this approach, you need to update it to simply refer to RFC-3168
> > for the relevant semantics.  Then I think it will be possible to make progress.

> response as doc shepherd before I commit any changes to the draft.

I've sent that response in a separate email thread entitled "Fragmentation & ECN encapsulation drafts, one more try" - please refer to that thread.  I will summarize an important reminder from that thread here, just in case it's been overlooked:

Note to the wise: A wise WG participant will review draft-ietf-intarea-frag-fragile before responding to this note to the wise (hint) - see  https://datatracker.ietf.org/doc/draft-ietf-intarea-frag-fragile/ .  That draft is IESG-approved to become a BCP (Best Current Practice), and is currently in the RFC Editor Queue.

Thanks, --David

> -----Original Message-----
> Sent: Monday, March 16, 2020 4:51 AM
> To: Jonathan Morton; Black, David
> Cc: [email-masked]
> Subject: Re: SCE / L4S and fragmentation
> 
> 
> [EXTERNAL EMAIL]
> 
> >> I (and David Black) have argued that a separate draft is needed to update
> fragment reassembly in RFC3168. Can you please accept that we are not going
> to update ECN fragmentation and reassembly in rfc6040update-shim. Then it
> can unblock and be published.
> >>
> this draft to become unstuck if it just said fragmentation and re-assembly is out
> of scope (which it is).
> > I have said several times in the past few days that I agree with that approach,
> and I'm sure David Black is well aware of that.
> >
> > However, the -10 revision (which appears to be current) still contains the
> section and some language covering fragmentation and reassembly.  To be
> consistent with this approach, you need to update it to simply refer to RFC-3168
> for the relevant semantics.  Then I think it will be possible to make progress.