Subject: Re: [tcpm] Possible error in accurate-ecn

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 1 Mar 2021 13:07:40 -0800

Hi Richard,

Thanks for the comments. I also think both a) and b) can mitigate the
concerns I mentioned.
Although I'm not sure if this is the best solution or not yet, it looks
certainly better to me than the previous ones.
--
Yoshi

On Wed, Feb 24, 2021 at 10:33 AM Scheffenegger, Richard <[email-masked]>
wrote:

>
> Hi Yoshi,
>
> A valid concern. Two ways to remediate these at least to some extent:
>
> a) used a "n" higher than 2 (e.g. 6)
> b) refrain from sending the ACK for n CE immediately, but hold off for
> either the delayed ack timeout, or the n+1'th receipt of CE, or a valid
> data segment by that time.
>
> While this would not fully rule out any possibilty of an inadvertend
> DupAck, it significantly reduces the time window when this may happen.
>
> Also, if an implementation uses other clues (e.g. TS opt, SACK
> negotiated but not present in the ACKs incrementing the CE.P ),
> ambiguity can be further reduced.
>
> However, I feel that these optimizations may overburden the definition
> of the mechanism.
>
> Best regards,
>     Richard
>
>
>
> Am 24.02.2021 um 11:03 schrieb Yoshifumi Nishida:
> > Hi Bob,
> >
> >
> > On Mon, Feb 22, 2021 at 4:12 PM Bob Briscoe <[email-masked]
> > <mailto:[email-masked]>> wrote:
> >
> >     Mirja, Richard, Yoshi,
> >
> >     I just posted a new rev of AccECN for a number of other reasons, and
> >     realized we hadn't converged on an answer to the tricky problem in
> >     this old thread from Nov'20. For now, the text says the following
> >     (which I've justified below, but I don't think it's the solution
> >     yet), and I've asked Richard & Mirja if we (as co-authors) can act
> >     as a little design team to thrash out this problem in the next few
> days.
> >
> >     3.2.2.5.1.  Data Receiver Safety Procedures
> >
> >         An AccECN Data Receiver:
> >
> >         o  SHOULD immediately send an ACK whenever a data packet marked
> CE
> >            arrives after the previous packet was not CE.
> >
> >         o  MUST immediately send an ACK once 'n' CE marks have arrived
> since
> >            the previous ACK, where 'n' SHOULD be 2 and MUST be in the
> range 2
> >            to 6 inclusive.
> >
> >
> >     Justification for first bullet: The aim is to trigger an ACK on a
> >     transition from non-CE to CE, but only when the latest packet is a
> >     data packet. Cases where the previous packet was a CE-marked pure
> >     ACK and therefore not ACK'd ought to be picked up by the 2nd bullet,
> >     not the first. Similarly, if there's CE-marked packets (data or pure
> >     ACKs) that haven't been ACKd yet, and the next data packet is not
> >     CE-marked, I think the first bullet is still correct, and the second
> >     rule will pick up cases with enough CE marks.
> >
> >     The second bullet is more tricky. I've added Richard's suggesting of
> >     flooring 'n' at 2. But made no other changes. That's deliberate,
> >     'cos I think it's sufficient to allow ACK CC to be added, but not to
> >     preclude it. And I think the danger of being interpreted as DupACKs
> >     shouldn't apply if there's no outstanding data. I know I'm wrong in
> >     those odd cases I mentioned when new data might cross the ACKs in
> >     the other direction. But I don't think that's going to cause too
> >     much harm.
> >
> > This might be too pathological, but I'm thinking about a certain extreme
> > example..
> > In my understanding, let's say a node sends 50 ACKs for some reasons and
> > all ACKs have CE marks, then its peer sends back 25 ACKs.
> > If these 25 ACKs have CE marks as well, it can generate another 12 ACKs,
> > and so on.
> > If this understanding is correct, I'm not sure if this is a good
> behavior.
> > I am thinking we had better do something here such as don't send an ACK
> > if the correspond ACKs carries accecn signals.
> >
> > Also, I think it's hard to tell there's no outstanding data from the
> > receiver side.
> > Let's say a sender send some data packets after sending 10 acks, and the
> > receiver sends back 5 acks as the 10 ACKs have CE marks.
> > In this case, they look like dup acks from the sender's point of view.
> > I know this is a rare case, though.
> >
> > --
> > Yoshi
> >
> >
> >
> >     On 17/11/2020 08:48, Scheffenegger, Richard wrote:
> >>
> >>     See my suggestion; The critical requirement for the 2nd bullet
> >>     point is,
> >>     to have n > 1 for pure, CE-marked ACKs. The exponential decay,
> >>     after how
> >>     many ACKs triggered by CE-marked ACKs these ACK-for-ACK stop,
> >>     strongly
> >>     depends on "n", so the recommendation should be to use a higher
> >>     value of
> >>     n in the case where you only observe a stream of pure ACKs (with or
> >>     without marks; only those marked would account against "n" though).
> >>
> >>     In the case of data packets mixed in, you don't want to delay
> >>     excessively, thus "n" may be choosen lower...
> >>
> >>     Richard
> >>
> >>
> >>
> >>     Am 10.11.2020 um 17:38 schrieb Bob Briscoe:
> >>>
> >>>     Moving on to the second bullet...
> >>>>>     The second bullet doesn't consider the possibility that the
> >>>>>     'n'th CE
> >>>>>     mark might arrive on a pure ACK. Then, the wording as it stands
> >>>>>     says
> >>>>>     the Data Receiver MUST immediately ACK a pure ACK. I know TCP
> >>>>>     never
> >>>>>     ACKs a pure ACK, but I'm not actually sure it does any harm to
> >>>>>     do so
> >>>>>     in this case (it cannot cause an infinite loop of ACKs). However,
> >>>>>     given it would be unorthodox, we maybe ought to rule it out by
> >>>>>     rewording anyway?
> >>>>     [MK] I think we also can leave this as it is because if that ’n’
> >>>>     packet is a pure ACK that still means that you have unacked data
> >>>>     packets with CE marks and you should trigger an ACK for those
> >>>>     packet
> >>>>     now (rather than waiting for the delayed ACK timer to expire).
> >>>>     However
> >>>>     this case is less important. Also we should probably make sure
> that
> >>>>     this doesn’t apply if there are only pure ACKs with CE marks,
> >>>>     maybe by
> >>>>     add “if unacknowledged data are outstanding” or something.
> >>>
> >>>     [BB] By the end of your last para you start to realize that, if
> >>>     the n'th
> >>>     CE mark arrives on pure ACK, it doesn't say anything about
> >>>     whether the
> >>>     previous (n-1) CE marks were on data or pure ACKs.
> >>>
> >>>     Consider another common scenario; a server delivering chunks of
> >>>     video
> >>>     over a high BDP path (e.g. 500 packets per RTT), and going idle
> >>>     between
> >>>     times (perhaps in response to an "exceeded high watermark" data
> >>>     packet
> >>>     from the server). Now consider the possibility that the reverse
> path
> >>>     bottleneck builds a queue during each chunk, and starts CE marking
> >>>     towards the end of each chunk. Then, after the server stops
> >>>     sending a
> >>>     chunk, it continues to see a couple of hundred CE-marked pure ACKs
> >>>     arriving.
> >>>
> >>>     However, by your proposed rule, it doesn't send any feedback,
> >>>     'cos it
> >>>     has no data to acknowledge. So the client cannot regulate its ACK
> >>>     stream. That's just tough, 'cos the client will stop sending more
> >>>     pure
> >>>     ACKs after it has received the last data, which is before any later
> >>>     feedback from the server can reach it.
> >>>
> >>>     However, if the client later sends a data packet that the server
> can
> >>>     acknowledge (e.g. "a low watermark" message), the server's ACE
> >>>     counter
> >>>     will have wrapped dozens of times, and the client won't be able
> >>>     to work
> >>>     out how many. That doesn't matter if a few round trips have
> >>>     elapsed -
> >>>     the congestion info will have become stale. But it does matter if
> >>>     it's
> >>>     fairly soon.
> >>>
> >>>     Instead, if we keep the text of the second bullet... if the
> >>>     server's 'n'
> >>>     was 2, it would send a pure ACK for every other pure ACK it
> >>>     received.
> >>>     But, what if these pure ACKs from the server were also CE-marked?
> >>>     Then
> >>>     the client would feed back pure ACKs using its 'n'. This regression
> >>>     would eventually die out, but I don't like it.
> >>>
> >>>     This conversation about the second bullet presumes AccECN is
> >>>     being used
> >>>     for ACK congestion control. I'm not comfortable with writing
> >>>     anything
> >>>     specific about ACK congestion control into a draft on the standards
> >>>     track. But I don't want to preclude AccECN being used for ACK CC.
> >>>
> >>>
> >>>     I believe the second bullet needs something changed to limit the
> >>>     ACKing
> >>>     of pure ACKs. However, before we put effort into wordsmithing,
> >>>     I'd like
> >>>     to hear whether you agree with the general idea of not defining
> >>>     what to
> >>>     do for ACK CC, but not precluding it.
> >>>
> >>>     Cheers
> >>>
> >>>
> >>>
> >>>
> >>>     Bob
> >>>
> >>>>
> >>>>     Mirja
> >>>>
> >>>>
> >>>>
> >>>>>     Thoughts anyone?
> >>>>>
> >>>>>
> >>>>>     Bob
> >>>>>
> >>>>>     --
> >>>>>     ________________________________________________________________
> >>>>>     Bob Briscoe
> >>>>>     http://bobbriscoe.net/ <http://bobbriscoe.net/>
> >>>>     _______________________________________________
> >>>>     tcpm mailing list
> >>>>     [email-masked] <mailto:[email-masked]>
> >>>>     https://www.ietf.org/mailman/listinfo/tcpm
> >>>>     <https://www.ietf.org/mailman/listinfo/tcpm>
> >>>
> >>
> >>     _______________________________________________
> >>     tcpm mailing list
> >>     [email-masked] <mailto:[email-masked]>
> >>     https://www.ietf.org/mailman/listinfo/tcpm
> >>     <https://www.ietf.org/mailman/listinfo/tcpm>
> >
> >     --
> >     ________________________________________________________________
> >     Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>
> >
> >     _______________________________________________
> >     tcpm mailing list
> >     [email-masked] <mailto:[email-masked]>
> >     https://www.ietf.org/mailman/listinfo/tcpm
> >     <https://www.ietf.org/mailman/listinfo/tcpm>
> >
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 2 Mar 2021 16:36:45 +0000

Yoshi, Richard,

 > In this case, they look like dup acks from the sender's point of view.

Let's test whether Richard's suggestion of using SACK is practical. I 
think it is.
As I understand it, the idea would involve:

If SACK has been negotiated.
If an ACK is a candidate for a DupACK (i.e. the ackno has not advanced)
Do not count it as a DupACK if there is no SACK on it.

The question then arises: how do we recommend this extra test in the 
draft? Because it's nothing to do with AccECN itself.
I guess after the bullet about 'n' CE marks, we say something like:

"This raises the possibility that an AccECN implementation will 
sometimes ACK pure ACKs, which in turn might be mistaken for duplicate 
ACKs (in scenarios where TCP peers take turns to send groups of data 
packets). To prevent spurious transmissions in such circumstances, if 
SACK has been negotiated, an implementation could assume that an ACK is 
not a Duplicate ACK if it has no SACK option, which would indicate it 
was an ACK of an  ACK."

That doesn't give a solution if SACK has not been negotiated, but:
     * The AccECN draft already recommends AccECN is implemented 
alongside SACK
     * Not implementing SACK is already unusual;
     * Yoshi's scenario is unusual in itself
     * The impact of Yoshi's scenario would be a spurious retransmission,
So, we have possible spurious retransmissions if an usual traffic 
pattern occurs in an extremely unlikely and not recommended combination 
of implementations. Can't we live with that?

Bob

On 01/03/2021 21:07, Yoshifumi Nishida wrote:
> Although I'm not sure if this is the best solution or not yet, it 
> looks certainly better to me than the previous ones.
> --
> Yoshi
>
> On Wed, Feb 24, 2021 at 10:33 AM Scheffenegger, Richard 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>     Hi Yoshi,
>
>     > On Mon, Feb 22, 2021 at 4:12 PM Bob Briscoe
>     <[email-masked] <mailto:[email-masked]>
>     > <mailto:[email-masked]
>     <mailto:[email-masked]>>> wrote:
>     >
>     >     Mirja, Richard, Yoshi,
>     >
>     >     as a little design team to thrash out this problem in the
>     next few days.
>     >
>     >         o  SHOULD immediately send an ACK whenever a data packet
>     marked CE
>     >         o  MUST immediately send an ACK once 'n' CE marks have
>     arrived since
>     >            the previous ACK, where 'n' SHOULD be 2 and MUST be
>     in the range 2
>     >            to 6 inclusive.
>     >
>     >
>     >>>>     this case is less important. Also we should probably make
>     sure that
>     >>>     from the server). Now consider the possibility that the
>     reverse path
>     >>>     However, if the client later sends a data packet that the
>     server can
>     >>>> [email-masked] <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     >> [email-masked] <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     <http://bobbriscoe.net/>
>     > [email-masked] <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 5 Mar 2021 03:14:03 -0800

Hi Bob,

I thought about it and I personally think it can be acceptable.
Another idea might be adding a simple mark into acc ecn option when the ACK
is triggered by ACKs with CE marks.
The receiver of the ACK with the mark will not generate another ACK even if
it has CE marks.
But, I am not sure adding extra info to the option for this purpose is
worthwhile. So, I am fine with using SACK.
--
Yoshi

On Tue, Mar 2, 2021 at 8:36 AM Bob Briscoe <[email-masked]> wrote:

> "This raises the possibility that an AccECN implementation will sometimes
> ACK pure ACKs, which in turn might be mistaken for duplicate ACKs (in
> scenarios where TCP peers take turns to send groups of data packets). To
> prevent spurious transmissions in such circumstances, if SACK has been
> negotiated, an implementation could assume that an ACK is not a Duplicate
> ACK if it has no SACK option, which would indicate it was an ACK of an
> ACK."
>
>     * The AccECN draft already recommends AccECN is implemented alongside
> SACK
> So, we have possible spurious retransmissions if an usual traffic pattern
> occurs in an extremely unlikely and not recommended combination of
> implementations. Can't we live with that?
>
>
> Bob
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 12 Mar 2021 10:54:01 +0000

Yoshi, tcpm list,

As promised we set up a small design team on the single issue of 
occasionally ACKing pure ACKs if they carry new info (ECN marking at the 
IP layer). The design team has two solutions and everyone would be 
prepared to accept either, but preferences differ. So we're seeking 
wider opinions (more opinions obviously won't narrow the choices, but at 
least the WG can then make a decision informed by those who care).

To understand the question, you can either read the email below. Or the 
3 slides for the tcpm meeting are briefer, and include pictures:
https://datatracker.ietf.org/meeting/110/materials/slides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00#page=6

Here's the current draft text in question (see here for context 
https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-14#section-3.2.2.5 
):

            3.2.2.5.1 Data Receiver Safety Procedures

    An AccECN Data Receiver:

    o  SHOULD immediately send an ACK whenever a data packet marked CE
       arrives after the previous packet was not CE.

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' SHOULD be 2 and MUST be in the range 2
       to 6 inclusive.

Only the second bullet is in question. Here is the proposed diff for 
each alternative, then we explain:

Alternative A

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
-     the previous ACK,
+     the previous ACK and there is outstanding data to acknowledge,
       where 'n' SHOULD be 2 and MUST be in the range 2 to 6 inclusive.

Alternative B

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
-     the previous ACK, where 'n' SHOULD be 2 and MUST be in the range 2
+     the previous ACK, where 'n' SHOULD be 3 and MUST be in the range 3
       to 6 inclusive.

Extra guidance text would be required in each case too (see the end).

Background:
AccECN is a change to the TCP wire protocol that requires the packet 
count of congestion feedback to include any congestion experienced (CE) 
arriving on Pure ACKs (amongst other things). AccECN doesn't require 
Pure ACKs to be ECN-capable, but allows for them to be. Similarly, 
AccECN doesn't require any congestion response to CE on pure ACKs, but 
having the feedback information there allows a response to be added with 
a one-ended update, if desired/necessary. Basically the data receiver is 
a 'dumb reflector'.

The above two bullets were designed to ensure that an ACK is triggered 
a) on the first sign of congestion, and b) frequently enough for the 
count of CE markings to be fed back using the 3-bit ACE field before it 
wraps, even if an occasional pure ACK is lost.

We then realized that the wording could require an ACK to be triggered 
in response to a CE-marked pure ACK. The circumstance when this could 
occur would be when peer X sends a volley of data to Y then stops, and 
the path back from Y to X is congested (probably by other flow(s) so 
that many of the ACKs are CE-marked. The second bullet above under 
alternative (B) would require X to ACK every 'n-th' CE-marked pure ACK. 
However, if Y immediately started sending a volley of data to X, Y could 
misinterpret those ACKs (of ACKs) from X as DupACKs.

There are two ways to deal with this:
A) Some of us prefer to completely prevent ACKs on pure ACKs, on the 
basis that they do not want to risk sometimes generating more ACKs today
B) Others want to ensure that these rules will cause pure ACKs to be 
ACKed when the amount of CE on the ACKs merits it. But sparingly and 
strongly damping any ACK ping-pong.

There are complexity arguments on both sides.

B more complex:
     extra (non-mandatory) 'if' condition for lack of SACK options on a 
pure ACK (to decide it's not a DupACK).
B less complex:
     consistent handling of CE marking whether on pure ACKs or data 
(which would probably remove an 'if' condition).

A more complex:
     CE markings on a string of pure ACKs can build up without feeding 
them back, until released by a data packet (if ever).
     More code at the other end to deal with the resulting risk of many 
wraps of the ACE field (or ignore?).
A less complex:
     less different from current TCP.

Extra guidance text would be necessary in either case.

* Alt A) would need text on handling the risk of many ACE wraps
     (to be written).

* Alt B) would need something like the following changes:

     For the avoidance of doubt, the above change-triggered ACK mechanism
     is deliberately worded to solely apply to data packets, and to ignore
     the arrival of a control packet with no payload, because it is
-  important that TCP does not acknowledge pure ACKs.  The change-
+  important that TCP does not acknowledge pure ACKs which convey no new
+  state information to the sender. The change-
     triggered ACK approach can lead to some additional ACKs but it feeds
     back the timing and the order in which ECN marks are received with
     minimal additional complexity.  If only CE marks are infrequent, or
     there are multiple marks in a row, the additional load will be low.
     Other marking patterns could increase the load significantly.
+
+  Providing feedback on the congestion state of the return channel
+  after a sender has ceased transmitting more data helps inform the
+  clients TCP congestion controller about the state of the return path.
+  Should the role of data sender and receiver subsequently change, the
+  new sender has more up to date knowledge of the network state,
+  preventing transmissions of inappropriate size at that moment.

     Even though the first bullet is stated as a "SHOULD", it is important
     for a transition to immediately trigger an ACK if at all possible, so
     that the Data Sender can rely on change-triggered ACKs to detect
     queue growth as soon as possible, e.g. at the start of a flow. This
     requirement can only be relaxed if certain offload hardware needed
     for high performance cannot support change-triggered ACKs (although
     high performance protocols such as DCTCP already successfully use
     change-triggered ACKs).  One possible compromise would be for the
     receiver to heuristically detect whether the sender is in slow-start,
     then to implement change-triggered ACKs while the sender is in slow-
     start, and offload otherwise.

+   The second bullet creates a possible case where an AccECN 
implementation
+   could sometimes ACK pure ACKs, which in turn might be mistaken for
+   duplicate ACKs (in scenarios where TCP peers take turns to send
+   sets of data packets). To prevent spurious transmissions in such
+   circumstances, if SACK has been negotiated, an implementation could
+   optionally assume that an ACK is not a Duplicate ACK if it has no 
SACK option,
+   which would indicate it was an ACK of an  ACK. Alternatively it 
could use
+   timestamp options to rule out DupACKs.

Bob

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Martin Duke <[email-masked]>
Date: Fri, 12 Mar 2021 13:37:10 -0800

Hi Bob,

Although I don't think the difference between these alternatives is all
that large, I believe I would go with (B) -- allow acks of acks at a rate
less than 100%. I suppose that in some corner cases it will prevent drops
out of a shallow queue, which isn't a big deal, but why not do it? To me,
the real complexity here is marking ACKs in the first place, which forces
you to do a bit of accounting about the number of acks you've sent.

It would be good to also add language to extend the RFC5681 definition of
"Duplicate ACK"

"

 DUPLICATE ACKNOWLEDGMENT: An acknowledgment is considered a
      "duplicate" in the following algorithms when (a) the receiver of
      the ACK has outstanding data, (b) the incoming acknowledgment
      carries no data, (c) the SYN and FIN bits are both off, (d) the
      acknowledgment number is equal to the greatest acknowledgment
      received on the given connection (TCP.UNA from [RFC793
<https://www.rfc-editor.org/rfc/rfc793>]) and (e)
      the advertised window in the incoming acknowledgment equals the
      advertised window in the last incoming acknowledgment.

      Alternatively, a TCP that utilizes selective acknowledgments
      (SACKs) [RFC2018, RFC2883
<https://www.rfc-editor.org/rfc/rfc2883>] can leverage the SACK
information to
      determine when an incoming ACK is a "duplicate" (e.g., if the ACK
      contains previously unknown SACK information).

"

to include "does not increase the ACE counter" as another criterion. Thus
TCP treats this as it would a pure window update, which is functionally
similar. This should solve any spurious retransmission issues.

On Fri, Mar 12, 2021 at 2:54 AM Bob Briscoe <[email-masked]> wrote:

> occasionally ACKing pure ACKs if they carry new info (ECN marking at the IP
> layer). The design team has two solutions and everyone would be prepared to
> accept either, but preferences differ. So we're seeking wider opinions
> (more opinions obviously won't narrow the choices, but at least the WG can
> then make a decision informed by those who care).
>
> To understand the question, you can either read the email below. Or the 3
> slides for the tcpm meeting are briefer, and include pictures:
>
> 3.2.2.5.1 Data Receiver Safety Procedures   An AccECN Data Receiver:
>
> Only the second bullet is in question. Here is the proposed diff for each
> alternative, then we explain:
>
> Alternative A
>
> AccECN is a change to the TCP wire protocol that requires the packet count
> of congestion feedback to include any congestion experienced (CE) arriving
> on Pure ACKs (amongst other things). AccECN doesn't require Pure ACKs to be
> ECN-capable, but allows for them to be. Similarly, AccECN doesn't require
> any congestion response to CE on pure ACKs, but having the feedback
> information there allows a response to be added with a one-ended update, if
> desired/necessary. Basically the data receiver is a 'dumb reflector'.
>
> The above two bullets were designed to ensure that an ACK is triggered a)
> on the first sign of congestion, and b) frequently enough for the count of
> CE markings to be fed back using the 3-bit ACE field before it wraps, even
> if an occasional pure ACK is lost.
>
> We then realized that the wording could require an ACK to be triggered in
> response to a CE-marked pure ACK. The circumstance when this could occur
> would be when peer X sends a volley of data to Y then stops, and the path
> back from Y to X is congested (probably by other flow(s) so that many of
> the ACKs are CE-marked. The second bullet above under alternative (B) would
> require X to ACK every 'n-th' CE-marked pure ACK. However, if Y immediately
> started sending a volley of data to X, Y could misinterpret those ACKs (of
> ACKs) from X as DupACKs.
>
> A) Some of us prefer to completely prevent ACKs on pure ACKs, on the basis
> that they do not want to risk sometimes generating more ACKs today
> B) Others want to ensure that these rules will cause pure ACKs to be ACKed
> when the amount of CE on the ACKs merits it. But sparingly and strongly
> damping any ACK ping-pong.
>
>     consistent handling of CE marking whether on pure ACKs or data (which
> would probably remove an 'if' condition).
>
> A more complex:
>     CE markings on a string of pure ACKs can build up without feeding them
> back, until released by a data packet (if ever).
> +   optionally assume that an ACK is not a Duplicate ACK if it has no SACK
> option,
> +   which would indicate it was an ACK of an  ACK. Alternatively it could
> use
> [email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 13 Mar 2021 11:29:18 +0100

Hi Martin,

 > To me, the real complexity here is marking ACKs in the first place,
 > which forces you to do a bit of accounting about the number of acks
 > you've sent.

I don't quite follow; Given that both AccECN endpoints are also using
ECN++, all control packets (ACKs without any data) can be sent as ECT.
In order for these packet to use the same Q as data packets, it is
beneficial to the sender of those, set ECT on them even.

The network is then free to set a CE mark - should the AQM deem that
necessary - on these control packets.

On the reflector, the CE.packet counter is simply incremented for each
incoming packet with the CE codepoint - no special treatment is
necessary for that aspect to work.

Thus the same rules apply to CE-marked data segments, as to any control
packets received.

 > to include "does not increase the ACE counter" as another criterion.
 > Thus TCP treats this as it would a pure window update, which is
 > functionally similar. This should solve any spurious retransmission
issues.

Unfortnuately, a true duplicate ack can just as well become CE-marked -
and IMHO this is a more probably scenario, than the misinterpretation of
a flight of CE-carrying ACKs to trigger a spurious retransmission.

As Bob's slide shows, the risk only exists under certain pathological
conditions:

a) the window from A->B must have been at least 3*n*(1/mark probability
for pure ACKs) in size
b) right afterward, the data direction has to change, with B becoming
the sender
c) B must have no other means (like SACK, Timestamp) to differentiate,
if the incoming ACKs (carrying new CE infomation, but not advancing
snd.una) have been triggered by a gap/reordering in the data segements
delivered to A

So, we effectively talk about a non-SACK, non-Timestamp, but AccECN and
ECN++ enabled flow, possibly encountering a pathological network
situation (sudden high CE marking probabilty), swapping the data
directions at that specific moment also.

What would AccECN loose by NOT sending ACKs carrying new CE-state back:

o) no timely update of the (currently quiescent) endpoint about changed
network congestion state - likely to influence the cwnd before the next
transmission and NOT bursting into a congested network (with all the
dire effects - loss, RTT/RTO delay for loss recovery, ...)
o) more complex implementation (special case for CE-marks on ACKs
without data)
o) Sender can not assume to get full information about the networks
congestion state unless data is transmitted
o) Delayed (untimely) CC reaction once data is being transmitted, if the
ACE counter wrap detection triggers (or complete removal of this
conservative CC functionality) [the ACK for new data will have the new
CE.packet counter state, indicating that some time between the prior
transmit from B, and now, there was network congestion. But no
infomation when it exactly happend.

Besides: As pure ACKs may be lost at any moment, the above effects can
impact an AccECN sender anyway, even if the receiver is properly
reflecting, in a timely manner, most up-to-date network congestion
state. But by removing the (simple) Ack-for-new-CE-marks capability from
the receiver, we forego any chance to improve the status quo in the future.

If an implementer does want to make the choice of deliberately
surpressing these ACKs, an AccECN sender has to deal with it anyway.

However, at the "cost" of a typically very minor increase in pure ACKs
sent 1/(n*1/(ACK-CE-mark probability)), AccECN can allow new CC to
remain up-to-date while no data transmissions are currently happening.

Richard

Am 12.03.2021 um 22:37 schrieb Martin Duke:
> that large, I believe I would go with (B) -- allow acks of acks at a
> rate less than 100%. I suppose that in some corner cases it will prevent
> drops out of a shallow queue, which isn't a big deal, but why not do it?
> To me, the real complexity here is marking ACKs in the first place,
> which forces you to do a bit of accounting about the number of acks
> you've sent.
>
> It would be good to also add language to extend the RFC5681 definition
> of "Duplicate ACK"
>
> "
>
>        received on the given connection (TCP.UNA from [RFC793  <https://www.rfc-editor.org/rfc/rfc793>]) and (e)
>        (SACKs) [RFC2018,RFC2883  <https://www.rfc-editor.org/rfc/rfc2883>] can leverage the SACK information to
> to include "does not increase the ACE counter" as another criterion.
> Thus TCP treats this as it would a pure window update, which is
> functionally similar. This should solve any spurious retransmission issues.
>
> On Fri, Mar 12, 2021 at 2:54 AM Bob Briscoe <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Yoshi, tcpm list,
>
>     occasionally ACKing pure ACKs if they carry new info (ECN marking at
>     the IP layer). The design team has two solutions and everyone would
>     be prepared to accept either, but preferences differ. So we're
>     seeking wider opinions (more opinions obviously won't narrow the
>     choices, but at least the WG can then make a decision informed by
>     those who care).
>
>     To understand the question, you can either read the email below. Or
>     the 3 slides for the tcpm meeting are briefer, and include pictures:
>     <https://datatracker.ietf.org/meeting/110/materials/slides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00#page=6>
>
>     <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-14#section-3.2.2.5>
>     ):
>
>
>     count of congestion feedback to include any congestion experienced
>     (CE) arriving on Pure ACKs (amongst other things). AccECN doesn't
>     require Pure ACKs to be ECN-capable, but allows for them to be.
>     Similarly, AccECN doesn't require any congestion response to CE on
>     pure ACKs, but having the feedback information there allows a
>     response to be added with a one-ended update, if desired/necessary.
>     Basically the data receiver is a 'dumb reflector'.
>
>     The above two bullets were designed to ensure that an ACK is
>     triggered a) on the first sign of congestion, and b) frequently
>     enough for the count of CE markings to be fed back using the 3-bit
>     ACE field before it wraps, even if an occasional pure ACK is lost.
>
>     We then realized that the wording could require an ACK to be
>     triggered in response to a CE-marked pure ACK. The circumstance when
>     this could occur would be when peer X sends a volley of data to Y
>     then stops, and the path back from Y to X is congested (probably by
>     other flow(s) so that many of the ACKs are CE-marked. The second
>     bullet above under alternative (B) would require X to ACK every
>     'n-th' CE-marked pure ACK. However, if Y immediately started sending
>     a volley of data to X, Y could misinterpret those ACKs (of ACKs)
>     from X as DupACKs.
>
>          extra (non-mandatory) 'if' condition for lack of SACK options
>     on a pure ACK (to decide it's not a DupACK).
>     B less complex:
>          CE markings on a string of pure ACKs can build up without
>     feeding them back, until released by a data packet (if ever).
>          More code at the other end to deal with the resulting risk of
>     many wraps of the ACE field (or ignore?).
>     A less complex:
>     +   optionally assume that an ACK is not a Duplicate ACK if it has
>     no SACK option,

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 15 Mar 2021 22:56:58 +0000

RIchard, Martin, 2 brief responses inline tagged [BB]...

On 13/03/2021 10:29, Scheffenegger, Richard wrote:
> I don't quite follow; 

[BB] I think Martin is not disagreeing with your assessment. He was just 
pointing out that the knock-on effects of ACKing pure ACKs originally 
stems from allowing pure ACKs to be ECN capable in the first place. He's 
not saying that's bad. Just it's the root cause of the new phenomenon 
we're seeing here.

> Given that both AccECN endpoints are also using

[BB] I was going to say roughly the same as Richard in response to 
Martin here.

Just because some apparent DupACKs with an increased CE count are not 
DupACKs, does not imply that a true DupACK cannot have increase CE count.

That's why we need other tests, like lack of SACK when negotiated, or 
timestamp evidence.

Bob

>>        (SACKs) [RFC2018,RFC2883

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 16 Mar 2021 17:50:26 +0100

Hi all,

Actually I think there is no problem if a ACK on an ACK is misinterpreted as dup ACK. What happens if a dupACK is detected? The cwnd is decreased and lost data is retransmitted. However, the cwnd was either already decreased with the first ACK and would not be decreased again for the next RTT (in case of Reno-like cc), or should be decreased anyway because the ACK carries a new CE feedback. I guess we could note that one should not decrease twice because if the information in the same ACK (being dup and CE counter). However, there will be no retransmission because there is no outstanding non-acked data given the received ack'ed only an ACK and no new data. So in summary, there is no problem here with dupACK generated in response to a pure CE-marked ACK that needs to be addressed in any way.

Mirja

> On 15. Mar 2021, at 23:56, Bob Briscoe <[email-masked]> wrote:
> 
> [BB] I think Martin is not disagreeing with your assessment. He was just pointing out that the knock-on effects of ACKing pure ACKs originally stems from allowing pure ACKs to be ECN capable in the first place. He's not saying that's bad. Just it's the root cause of the new phenomenon we're seeing here.
> 
> [BB] I was going to say roughly the same as Richard in response to Martin here.
> 
> Just because some apparent DupACKs with an increased CE count are not DupACKs, does not imply that a true DupACK cannot have increase CE count.
> 
> That's why we need other tests, like lack of SACK when negotiated, or timestamp evidence.
> 
> 
> Bob
> 
>>

---

From: Martin Duke <[email-masked]>
Date: Tue, 16 Mar 2021 12:04:00 -0700

Mirja,

I agree there's little harm with classic ECN -- other than the spurious
retransmission, any CE signal in an RTT or a Fast Retransmit will have a
similar effect on cwnd.

But for L4S, 3 CE marks are likely to have a quite different impact than a
packet loss.

This case can occur when data is outstanding, which is why the 5681
language is useful in mitigating it.

As I said in my first email, these differences are all quite small IMO.

On Tue, Mar 16, 2021 at 9:50 AM Mirja Kuehlewind <[email-masked]>
wrote:

> Hi all,
>
> Actually I think there is no problem if a ACK on an ACK is misinterpreted
> as dup ACK. What happens if a dupACK is detected? The cwnd is decreased and
> lost data is retransmitted. However, the cwnd was either already decreased
> with the first ACK and would not be decreased again for the next RTT (in
> case of Reno-like cc), or should be decreased anyway because the ACK
> carries a new CE feedback. I guess we could note that one should not
> decrease twice because if the information in the same ACK (being dup and CE
> counter). However, there will be no retransmission because there is no
> outstanding non-acked data given the received ack'ed only an ACK and no new
> data. So in summary, there is no problem here with dupACK generated in
> response to a pure CE-marked ACK that needs to be addressed in any way.
>
> Mirja
>
>
>
> pointing out that the knock-on effects of ACKing pure ACKs originally stems
> from allowing pure ACKs to be ECN capable in the first place. He's not
> saying that's bad. Just it's the root cause of the new phenomenon we're
> seeing here.
> >
> >>>        received on the given connection (TCP.UNA from [RFC793 <
> https://www.rfc-editor.org/rfc/rfc793>]) and (e)
> >>>        (SACKs) [RFC2018,RFC2883 <
> https://www.rfc-editor.org/rfc/rfc2883>] can leverage the SACK
> information to
> >>> <
> >>> <
> >>>     +  important that TCP does not acknowledge pure ACKs which convey
> no
> >>>     new

---

From: Martin Duke <[email-masked]>
Date: Tue, 16 Mar 2021 12:11:10 -0700

Hi Bob,

> not saying that's bad. Just it's the root cause of the new phenomenon

Yep

> That's why we need other tests, like lack of SACK when negotiated, or

Absolutely: by the same token, a window update could also be a genuine
duplicate ACK, but TCP uses a heuristic.

If people are uncomfortable with the heuristic, we could eliminate it by
saying.
- pure counter updates are not duplicate acks, as an extension to the 5681
definition, and
- A receiver that gets both an OOO packet (which might be CE marked) and
one or more CE-marked ACKs must send *two* acks; one that updates the
counter and one that does not. The one that does not is unambiguously a
duplicate ack.

I don't think this is necessary, but it would resolve the problem.

---

From: Vidhi Goel <[email-masked]>
Date: Tue, 16 Mar 2021 20:49:35 -0700

> should not decrease twice because if the information in the same ACK (being dup and CE counter). However, there will be no retransmission because there is no outstanding non-acked data given the received ack'ed only an ACK and no new data. So in summary, there is no problem here with dupACK generated in response to a pure CE-marked ACK that needs to be addressed in any way.

I think there will be retransmission if the dup_ACK criteria (3 dup_ACKs or whatever the stack is using) is met as this condition, th_ack  <= snd_una, will be true, if the new sender has already sent out data before the dup_ACK is received.

-
Vidhi

> On Mar 16, 2021, at 9:50 AM, Mirja Kuehlewind <[email-masked]> wrote:
> 
> Hi all,
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 16 Mar 2021 22:47:41 -0700

Hi Mirja,

> response to a pure CE-marked ACK that needs to be addressed in any way.

I personally am not very fine with dup acks is being misinterpreted
This might be another corner case, but here's an example I've thought.

When a sender sent one data packet D1 and bunch of acks (say 30), and D1 is
lost, all acks have CE marks.
In my understanding, this will generate 10 dup acks that request D1 with
(B)
In this case, sender retransmits D1 after 3rd dupacks and it will still
receive 7 dupacks.
These 7 dup acks might trigger another transmission of packets on some
implementations.
This is because RFC5681 states:

.  For each additional duplicate ACK received (after the third),
       cwnd MUST be incremented by SMSS.  This artificially inflates the
       congestion window in order to reflect the additional segment that
       has left the network.

RFC5681 also states:

   since the receiver can only generate a
   duplicate ACK when a segment has arrived, that segment has left the
   network and is in the receiver's buffer, so we know it is no longer
   consuming network resources.

But, I think this doesn't fit the logic in the doc very well. We might
want to add some texts for this if we choose (B)

--
Yoshi

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 17 Mar 2021 00:22:56 -0700

Hi Bob,

> damping any ACK ping-pong.

Hmm. This is not easy..
My personal preference is if SACK is negotiated and it 's utilized to
distinguish dupacks, then (B) is fine for me. Otherwise, I would prefer
(A).

BTW, I am wondering if this could leave it to implementations since both
methods have pros and cons.
I think It's hard to tell one is much better than the order.

Thanks,
--
Yoshi

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 17 Mar 2021 09:02:10 +0100

First RFC5681 states (for completeness):

       received on the given connection (TCP.UNA from [RFC793]) and (e)

       Alternatively, a TCP that utilizes selective acknowledgments
       (SACKs) [RFC2018, RFC2883] can leverage the SACK information to
       determine when an incoming ACK is a "duplicate" (e.g., if the ACK
       contains previously unknown SACK information).

So, there is already the strong advice, that existing SACK information
shall be considered for DupACK detection.

Also, I still believe, that we give this particular corner case more
stage time than warranted.

 > When a sender sent one data packet D1 and bunch of acks (say 30),
 > and D1 is lost, all acks have CE marks.
 > In my understanding, this will generate 10 dup acks that request
 > D1 with (B)

Lets work this example:

A used to be the sender of 60 segments.
B is  ACKing all the above segments with 30 segments.
The network is suddenly having a pathologic state, where each and every
of these ACKs is CE marked.
B is now transmitting D1.

A ACKs every 3rd CE-marked ACK with another ACK,no SACK, TSecr reflects
the TSval of these ACKs
A ACKs D1 (no SACK, TSecr of D1)

B observes ACKs, with a TSecr indicating they are sent before D1 could
arrive. And they don't contain SACK. However, ACE increments, thus a CC
reaction is needed *in this window*.

Or, B ignores the indications from TSopt/SACK - reduces cwnd (either due
to ACE incrementing, or the dupack trigger and retransmits D1.

If B does ignore TSopt - it likely has no notion of the Window during
non-data flights of segments. Thus a cwnd reduction due to DupAck is
probably the ONLY trigger for the required CC reduction.

Or B does use auxilliary information like TSopt, flight in
segements/expected ACKs, to drive the AccECN-based CC-reduction - and
would not mistake these ACE-incrementing ACKs as dupacks, as they belong
to a flight before D1 could have arrived.

However, from a CC perspective, the outcome (actual cwnd reduction) is
correct. Only from a efficiency perspective, an undue retransmission
could have been saved.

(And we are still talking about a pathological situation, where suddenly
a significant fraction of small ACK packets all get CE marked. In the
environments analysed for step-marking, the CE-marking during steady
state is 2 marks per window, which is lower than the threshold of 3 in
(B). Using a higher n in (B) would further reduce the risk of spurious
retransmission, as the higher risk of desynchonization of the CE.p
counters between the endpoints

Am 17.03.2021 um 06:47 schrieb Yoshifumi Nishida:
> Hi Mirja,
>
> On Tue, Mar 16, 2021 at 9:50 AM Mirja Kuehlewind <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>     Actually I think there is no problem if a ACK on an ACK is
>     misinterpreted as dup ACK. What happens if a dupACK is detected? The
>     cwnd is decreased and lost data is retransmitted. However, the cwnd
>     was either already decreased with the first ACK and would not be
>     decreased again for the next RTT (in case of Reno-like cc), or
>     should be decreased anyway because the ACK carries a new CE
>     feedback. I guess we could note that one should not decrease twice
>     because if the information in the same ACK (being dup and CE
>     counter). However, there will be no retransmission because there is
>     no outstanding non-acked data given the received ack'ed only an ACK
>     and no new data. So in summary, there is no problem here with dupACK
>     generated in response to a pure CE-marked ACK that needs to be
>     addressed in any way.
>
>
> When a sender sent one data packet D1 and bunch of acks (say 30), and D1
> is lost, all acks have CE marks.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 17 Mar 2021 18:59:36 +0100

See below

> On 17. Mar 2021, at 04:49, Vidhi Goel <[email-masked]> wrote:
> 

However these are dupACK of pure ACKs, which means the sender did not send data but only ACKs and therefore there should likely be no out-standing, not ACK'ed data when the dupACK is received.

Mirja

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 17 Mar 2021 19:01:18 +0100

This is a corner concern case; and you shouldn’t retransmit a second time if more dupACK are received within the next RTT.

> On 17. Mar 2021, at 06:47, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Mirja,
> 
> On Tue, Mar 16, 2021 at 9:50 AM Mirja Kuehlewind <[email-masked]> wrote:
> Hi all,
> 
> When a sender sent one data packet D1 and bunch of acks (say 30), and D1 is lost, all acks have CE marks.
> In my understanding, this will generate 10 dup acks that request D1 with (B) 
> In this case, sender retransmits D1 after 3rd dupacks and it will still receive 7 dupacks.
> These 7 dup acks might trigger another transmission of packets on some implementations. 
> But, I think this doesn't fit the logic in the doc very well. We might want to add some texts for this if we choose (B)
> --
> Yoshi
> 
> 
>

---

From: Vidhi Goel <[email-masked]>
Date: Wed, 17 Mar 2021 14:14:00 -0700

> However these are dupACK of pure ACKs, which means the sender did not send data but only ACKs and therefore there should likely be no out-standing, not ACK'ed data when the dupACK is received.

Sure. But if the sender (who was receiver before) has already sent out data right after the pure ACKs, there will be outstanding data before the ACK for pure ACK (or DupACK as there is outstanding data) is received from the peer, right?

Vidhi

> On Mar 17, 2021, at 10:59 AM, Mirja Kuehlewind <[email-masked]> wrote:
> 
> See below
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 18 Mar 2021 13:49:33 +0100

See inline.

> On 17. Mar 2021, at 22:14, Vidhi Goel <[email-masked]> wrote:
> 
> 

I see. That means you send a whole bunch of Pure ACKs and then some data in the same RTT. In this case it would actually be nicer to wait for the data packet instead of sending a pure ACKs. Maybe should should rather use always some thing like a delayed ACK timer for pure ACK rather then saying we need to ACK each nth pure ACH?

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 18 Mar 2021 21:50:27 +0100

Hi Mirjy, Vidhi,

> I see. That means you send a whole bunch of Pure ACKs and then some data in the same RTT. In this case it would actually be nicer to wait for the data packet instead of sending a pure ACKs. Maybe s…

Introducting an additional delay is further reducing the risk. (And it
would be relatively easy to implement, reusing existing functions.) But
remember we started out with the (unlikely, and rather pathological
corner case, where a re-routing event coincides with when the ACKs get
sent out, redirecting them over a path which marks near 100% of these
very small pure ACKs with CE - while the former sender has send a window
of packets of size 3*n (at least).

And yes, slightly holding off for sending the ACKs-for-CE-marked-ACKs
(e.g. activating the delayed ACK timer, or when the n+1'th CE-marked ACK
is received) further reduces the probability to trigger this.

As will the use of SACK and TSopt (I don't know recent numbers, but
suspect that a vast majority of TCP flows is currently used one or the
other (or both) of these TCP options.

All that, to avoid a spurious Retransmission (spurious retransmissions
do happen currently much more often for much more begning reasons, like
RTT step changes) - while the congestion control reaction is actually
the intended, and correct behavior. So that non-SACK (non-TS) sender is
only "behind" by one segment, which may have been a new data segement
instead...

I suspect this discussion is really about the mite-hole at the end of an
ant-hole, at the end of a rat-hole, at the end of a rabbit-hole ;)

At least I can not think of a realistic scenario, which is expected to
be encountered in a future L4S/SCE (shallow, instanteous marking)
environment, that would run into such a pathological setup on a regular
basis. But perhaps there are some...

(Extremely high RTT, very large BDP links, with cwnd measured in 100s of
segments, and extreme congestion of a shared-use return path, perhaps?
Still not convinced that here the marking will hit dominantely pure ACKs
(and not data segments of significantly larger size), and in
sufficiently high number to cross the 3*n (or, with delay, 4*n) threshold.

Yoshi, do you have a scenario in mind, where this effect may be relevant
or even dominant?

Best regards,
    Richard

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 19 Mar 2021 06:27:57 +0200

> On 18 Mar, 2021, at 10:50 pm, Scheffenegger, Richard <[email-masked]> wrote:
> 

Shallow marking thresholds increase the probability of packets being marked, especially under congested conditions.  I consider it quite likely that AQMs will sometimes end up marking every packet that passes through them, even if they are only small packets such as pure acks, and it is even possible for this to occur in both directions simultaneously (on slow links, for example - or faster links so congested that the fair share of each flow is slow).

During the design of Cake, I became aware of cases where an ack stream on the reverse path could become the limiting factor for goodput on the forward path, even when many (Not-ECT) acks were being dropped by the AQM because the stream had persistently exceeded the threshold at 5ms.  That is why Cake has an ack coalescing filter.  These conditions arise even with conventional AQM and relatively high RTT; they are mainly associated with the highly asymmetric links that are commonly provisioned at the last mile.

So these conditions are not so rare as to be beneath examination, at the very least.  The conditions giving rise to the provision of Cake's ack filter would also result in high marking rates of ECT acks.  The question is whether the resulting effects are mild enough to tolerate.

Summarising the situation so far:

- It must be assumed that AQMs are capable of CE-marking 100% of ECN-Capable packets, if the AQM considers that conditions warrant it.

- ECN-Capable pure acks give rise to the need to signal the receipt of CE marks on pure acks back to their sender.

- When the receiver of a CE-marked pure ack has data to send, it can piggyback that notification on a future data segment.  This is not a problem.

- When the receiver of a CE-marked pure ack *does not* have data to send, it must either generate a pure ack to carry the notification, or store the notification for some future opportunity.

- The former choice can be described as "acking an ack", and can lead to an infinite series of ack-acking exchanges if all such acks are CE marked in the network, and no mechanism for ending the exchange is specified.

- Additionally, if the sender of an ECN-Capable pure ack subsequently has data available to send, and does so in more than one segment, a subsequent notification of a CE mark on the ack could be misinterpreted as a DupAck indicating possible loss of the data segment, combined with a CE mark on the second segment or later.

The latter two points are what we need to evaluate, and guard against to the extent deemed necessary.

One very simple method of breaking the infinite chain of notifications would be to specify that acks of acks should themselves be Not-ECT.  These will never be CE marked (modulo broken middleboxes).  Another is to require at least two outstanding "pure ack" CE marks before a pure ack can be generated as a notification; this does not preclude immediate notification of CE marks either upon or via data segments.  The latter method would guarantee an exponential decay of the volume of acks exchanged per RTT, and eventually halt them entirely.

I would recommend language specifying that one of the above methods, or some equivalently effective protection, SHOULD be implemented.

There are a number of commonly-used mechanisms to disambiguate true DupAcks from spurious ones:

- If TCP Timestamps are used, the TSecr field could be compared with the TSval attached to the previously generated pure acks and/or each of the outstanding data segments.  This would indicate that the candidate DupAck was generated prior to receipt of any of the data segments, and must therefore be an "ack of an ack".

- If SACK has been negotiated, it is reasonable to assume that a lost segment followed by a received one would be notified by SACK, and not by a SACK-less DupAck.  The latter is therefore probably an "ack of an ack".

I would recommend language stating that ECN-Capable pure acks SHOULD NOT be generated unless either TCP Timestamps or TCP SACK has been successfully negotiated on the connection, or some equivalent disambiguating information is available, AND that disambiguating information is actually used to prevent spurious retransmissions.  Since both Timestamps and SACK are in common use, that shouldn't be onerous.

However, it is also true that a spurious retransmission on occasion is probably not very harmful.

 - Jonathan Morton

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 19 Mar 2021 01:24:47 -0700

Hi Richard,

> counters between the endpoints

Yes, I think the logic works.
My point was that we had better have something to mitigate dupack effects
caused by acking pure acks.
Since the logic above does the job, I am fine with the logic if it's
mentioned in the draft.

Also, I think we might want to mention that this logic overwrites the
presumptions in RFC5681 in the draft.
Because RFC5681 suggests receiving a dupack packet can be considered as an
indication of arrival of data
segment at the other side.
--
Yoshi

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 19 Mar 2021 10:17:15 +0100

Thank you Jonathan!

Your argument that this effect has to be considered to be a valid
(hopefully) temporary one is understood.

> One very simple method of breaking the infinite chain of notifications would be to specify that acks of acks should themselves be Not-ECT.  These will never be CE marked (modulo broken middleboxes)…

This is already in the text (one ACK at most as often as every other
CE-mark received, as a MUST).

One proposal (maintaining the CE-mark feedback in these situations) was
to change the exponential decay factor, by sending out one
ACK-for-CE-ACKs only after a cumulative change of at least 3 counts (not
just two).

An additional (implementation specific) heuristic would be to send out
the ACK for n with the receipt of the n+1'th marked ACK, while priming
the delayed ACK timeout for sending a ACE counter change - improving the
chances that this information can be sent along with new data somewhat.

> I would recommend language stating that ECN-Capable pure acks SHOULD NOT be generated unless either TCP Timestamps or TCP SACK has been successfully negotiated on the connection, or some equivalent…

Not ECT-marking these ACKs-for-CE-marked ACKs is actually an alternative
we (authors) haven't thought about, actually. For flows not doing either
SACK or Timestamps, that may be another SHOULD recommendation to include.

Thanks,
   Richard

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Mar 2021 17:07:38 +0000

Yoshi & Mirja,

>> But, I think this doesn't fit the logic in the doc very well. We might want to add some texts for this if we choose (B)

[BB] To summarize, the picture is something like slide 6 here except one 
of the early CE-marked ACKs from Y to X is a data packet:
https://datatracker.ietf.org/meeting/110/materials/slides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00#page=6
We'll call the left and right hosts X and Y.

Unlike in the previous case that you described, in this case Y receives 
more DupACKs than the number of packets it sent since its last ACK'd 
data packet. I started thinking of ways to check for this, but I quickly 
came to the conclusion that we should not try to add logic to check for 
that.

Before we say this is a corner-case, I'd like to try to generalize it, 
to understand how broad it might be.

* It might involve one or more packets from Y to X being lost (or delayed).
* And there might have been some data packets from Y to X delivered 
after the gap, which would each have triggered a (genuine) DupACK of the 
seqno before the gap.

Also, all these cases generalize to loss detected in time (like RACK) 
not just counting DupACKs. When data from Y to X ends, ACKs from X to Y 
continue for a round trip. If many are CE-marked, rule (B) triggers ACKs 
of the ACKs later in the round. For instance, if RACK tolerates 
reordering for RTT/8 (say) then ACKs of ACKs in the last 7/8 of the 
round could be considered DupACKs, if the conditions of these corner 
cases are in place.

I still think we can categorize all these as corner cases. but we are 
finding quite a few corners.

Nonetheless, I think it is still simpler to use the wording of alt (B) 
as it is, and leave these corner cases as potential cases where DupACKs 
might occasionally be wrongly detected. But just live with that.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Mar 2021 17:37:10 +0000

Yoshi,

On 17/03/2021 07:22, Yoshifumi Nishida wrote:
>     occasionally ACKing pure ACKs if they carry new info (ECN marking
>     at the IP layer). The design team has two solutions and everyone
>     would be prepared to accept either, but preferences differ. So
>     we're seeking wider opinions (more opinions obviously won't narrow
>     the choices, but at least the WG can then make a decision informed
>     by those who care).
>
>     To understand the question, you can either read the email below.
>     Or the 3 slides for the tcpm meeting are briefer, and include
>     pictures:
>     Only the second bullet is in question. Here is the proposed diff
>     for each alternative, then we explain:
>
>     Alternative A
>
>     AccECN is a change to the TCP wire protocol that requires the
>     packet count of congestion feedback to include any congestion
>     experienced (CE) arriving on Pure ACKs (amongst other things).
>     AccECN doesn't require Pure ACKs to be ECN-capable, but allows for
>     them to be. Similarly, AccECN doesn't require any congestion
>     response to CE on pure ACKs, but having the feedback information
>     there allows a response to be added with a one-ended update, if
>     triggered in response to a CE-marked pure ACK. The circumstance
>     when this could occur would be when peer X sends a volley of data
>     to Y then stops, and the path back from Y to X is congested
>     (probably by other flow(s) so that many of the ACKs are CE-marked.
>     The second bullet above under alternative (B) would require X to
>     ACK every 'n-th' CE-marked pure ACK. However, if Y immediately
>     started sending a volley of data to X, Y could misinterpret those
>     ACKs (of ACKs) from X as DupACKs.
>
>     A) Some of us prefer to completely prevent ACKs on pure ACKs, on
>     the basis that they do not want to risk sometimes generating more
>     ACKs today
>     B) Others want to ensure that these rules will cause pure ACKs to
>     be ACKed when the amount of CE on the ACKs merits it. But
>     sparingly and strongly damping any ACK ping-pong.
>
>
> Hmm. This is not easy..
> distinguish dupacks, then (B) is fine for me. Otherwise, I would 
> prefer (A).
>
> BTW, I am wondering if this could leave it to implementations since 
> both methods have pros and cons.

[BB] The problem is that the decision impacts the complexity of the 
other end, not just the implementer's own code.

* If an implementer chooses (A) (doesn't ACK pure ACKs at all), when it 
finally does send some data, it's CE counter could have built up an 
(unlimited) store of unreported CE marks, which it will report all at 
once. So implementations will have to include code that copes with 
receiving potentially multiple wraps of the ACE field (or just ignore 
the possibility).

* If an implementer chooses (B), the code at the other end should just 
work without any specific logic to recognize the other end is doing 
behaviour (B). It could add the checks on DupACKs using SACK or 
timestamps, but I think that could be a MAY or SHOULD.

Put another way, if the draft left the choice between A & B to the 
implementer, the benefit of B in simplifying everyone's code would have 
been lost because, even if an implementer chose B herself, everyone 
would still have to write extra code in case the other end had chosen A.

This is why I would recommend that the draft says B and doesn't leave A 
as a choice.

Bob

>     no new
>     +   which would indicate it was an ACK of an  ACK. Alternatively
>     it could use

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 22 Mar 2021 11:08:31 +0100

See inline.

> On 19. Mar 2021, at 18:37, Bob Briscoe <[email-masked]> wrote:
> 
> Yoshi,
> 
>> As promised we set up a small design team on the single issue of occasionally ACKing pure ACKs if they carry new info (ECN marking at the IP layer). The design team has two solutions and everyone would be prepared to accept either, but preferences differ. So we're seeking wider opinions (more opinions obviously won't narrow the choices, but at least the WG can then make a decision informed by those who care).
>> 
>> To understand the question, you can either read the email below. Or the 3 slides for the tcpm meeting are briefer, and include pictures:
>> Here's the current draft text in question (see here for context https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-14#section-3.2.2.5 ):
>> Only the second bullet is in question. Here is the proposed diff for each alternative, then we explain:
>> 
>> Alternative A
>> AccECN is a change to the TCP wire protocol that requires the packet count of congestion feedback to include any congestion experienced (CE) arriving on Pure ACKs (amongst other things). AccECN doesn't require Pure ACKs to be ECN-capable, but allows for them to be. Similarly, AccECN doesn't require any congestion response to CE on pure ACKs, but having the feedback information there allows a response to be added with a one-ended update, if desired/necessary. Basically the data receiver is a 'dumb reflector'.
>> 
>> The above two bullets were designed to ensure that an ACK is triggered a) on the first sign of congestion, and b) frequently enough for the count of CE markings to be fed back using the 3-bit ACE field before it wraps, even if an               occasional pure ACK is lost.
>> 
>> We then realized that the wording could require an ACK to be triggered in response to a CE-marked pure ACK. The circumstance when this could occur would be when peer X sends a volley of data to Y then stops, and the path back from Y to X is congested (probably by other flow(s) so that many of the ACKs are CE-marked. The second bullet above under alternative (B) would require X to ACK every 'n-th' CE-marked pure ACK. However, if Y immediately started sending a volley of data to X, Y could misinterpret those ACKs (of ACKs) from X as DupACKs.
>> 
>> A) Some of us prefer to completely prevent ACKs on pure ACKs, on the basis that they do not want to risk sometimes generating more ACKs today
>> B) Others want to ensure that these rules will cause pure ACKs to be ACKed when the amount of CE on the ACKs merits it. But sparingly and strongly damping any ACK ping-pong.
>> 
>> Hmm. This is not easy.. 
>> My personal preference is if SACK is negotiated and it 's utilized to distinguish dupacks, then (B) is fine for me. Otherwise, I would prefer (A). 
>> 
>> BTW, I am wondering if this could leave it to implementations since both methods have pros and cons. 
> [BB] The problem is that the decision impacts the complexity of the other end, not just the implementer's own code.
> 
> * If an implementer chooses (A) (doesn't ACK pure ACKs at all), when it finally does send some data, it's CE counter could have built up an (unlimited) store of unreported CE marks, which it will report all at once. So implementations will have to include code that copes with receiving potentially multiple wraps of the ACE field (or just ignore the possibility).

You always need to cover this case because it is always possible to lose ACKs  and then counter may wrap. However, I don’t think you actually need to implement any logic to detect this. It only means the counters on both ends are off by N x 8 but congestion information is only valid that the point when it happens and resyncing on outdated information doesn’t help anybody.

> 
> * If an implementer chooses (B), the code at the other end should just work without any specific logic to recognize the other end is doing behaviour (B). It could add the checks on DupACKs using SACK or timestamps, but I think that could be a MAY or SHOULD.
> 
> Put another way, if the draft left the choice between A & B to the implementer, the benefit of B in simplifying everyone's code would have been lost because, even if an implementer chose B herself, everyone would still have to write extra code in case the other end had chosen A.
> 
> This is why I would recommend that the draft says B and doesn't leave A as a choice.
> 
> 
> Bob
> 
>> 
>> Thanks,
>> --
>> Yoshi
>>  
>>  
>>     extra (non-mandatory) 'if' condition for lack of SACK options on a pure ACK (to decide it's not a DupACK).
>> B less complex: 
>>     consistent handling of CE marking whether on pure ACKs or data (which would probably remove an 'if' condition).
>> 
>> A more complex: 
>>     CE markings on a string of pure ACKs can build up without feeding them back, until released by a data packet (if ever). 
>>     More code at the other end to deal with the resulting risk of many wraps of the ACE field (or ignore?).
>> A less complex: 
>> +   The second bullet creates a possible case where an AccECN implementation 
>> +   optionally assume that an ACK is not a Duplicate ACK if it has no SACK option,
>> +   which would indicate it was an ACK of an  ACK. Alternatively it could use 
>> http://bobbriscoe.net/
> http://bobbriscoe.net/

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 24 Mar 2021 01:44:22 -0700

Hi Bob, Mirja

On Mon, Mar 22, 2021 at 3:08 AM Mirja Kuehlewind <[email-masked]>
wrote:

> arriving on Pure ACKs (amongst other things). AccECN doesn't require Pure
> ACKs to be ECN-capable, but allows for them to be. Similarly, AccECN
> doesn't require any congestion response to CE on pure ACKs, but having the
> feedback information there allows a response to be added with a one-ended
> update, if desired/necessary. Basically the data receiver is a 'dumb
> reflector'.
> >>
> a) on the first sign of congestion, and b) frequently enough for the count
> of CE markings to be fed back using the 3-bit ACE field before it wraps,
> even if an               occasional pure ACK is lost.
> >>
> in response to a CE-marked pure ACK. The circumstance when this could occur
> (unlimited) store of unreported CE marks, which it will report all at once.
> So implementations will have to include code that copes with receiving
> potentially multiple wraps of the ACE field (or just ignore the
> possibility).
>
> You always need to cover this case because it is always possible to lose
> ACKs  and then counter may wrap. However, I don’t think you actually need
> to implement any logic to detect this. It only means the counters on both
> ends are off by N x 8 but congestion information is only valid that the
> point when it happens and resyncing on outdated information doesn’t help
> anybody.
>
>
So, in my understanding, there are two types of choices for implementations
on the other end.
1) need to decide whether ignore CE counter wraps or prepare for it  (for
both A and B)
2) need to decide whether identify ACKs on pure ACKs by accecn or accept
possible ack ping-pong  (for only B)

In this case,
For 1)
   Choosing (A) or (B) doesn't matter because both (A) and (B) have this
potential risk.
For 2)
   if we pick A, we don't need this.
   if we pick B, we need this
   if we allow both, we need this

If I think in this way, choosing (B) is not much different from allowing
both?
--
Yoshi

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Mar 2021 19:03:49 +0000

Yoshi,

On 24/03/2021 08:44, Yoshifumi Nishida wrote:
> Hi Bob, Mirja
>
> On Mon, Mar 22, 2021 at 3:08 AM Mirja Kuehlewind <[email-masked] 
>     > On 19. Mar 2021, at 18:37, Bob Briscoe <[email-masked]
>     >> On Fri, Mar 12, 2021 at 2:54 AM Bob Briscoe
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     >> Yoshi, tcpm list,
>     >>
>     >> As promised we set up a small design team on the single issue
>     of occasionally ACKing pure ACKs if they carry new info (ECN
>     marking at the IP layer). The design team has two solutions and
>     everyone would be prepared to accept either, but preferences
>     differ. So we're seeking wider opinions (more opinions obviously
>     won't narrow the choices, but at least the WG can then make a
>     decision informed by those who care).
>     >>
>     >> To understand the question, you can either read the email
>     below. Or the 3 slides for the tcpm meeting are briefer, and
>     include pictures:
>     >>
>     >> Only the second bullet is in question. Here is the proposed
>     diff for each alternative, then we explain:
>     >>
>     >> Alternative A
>     >> A) Some of us prefer to completely prevent ACKs on pure ACKs,
>     on the basis that they do not want to risk sometimes generating
>     more ACKs today
>     >> B) Others want to ensure that these rules will cause pure ACKs
>     to be ACKed when the amount of CE on the ACKs merits it. But
>     >> My personal preference is if SACK is negotiated and it 's
>     utilized to distinguish dupacks, then (B) is fine for me.
>     Otherwise, I would prefer (A).
>     >>
>     > [BB] The problem is that the decision impacts the complexity of
>     the other end, not just the implementer's own code.
>     >
>     > * If an implementer chooses (A) (doesn't ACK pure ACKs at all),
>     when it finally does send some data, it's CE counter could have
>     built up an (unlimited) store of unreported CE marks, which it
>     will report all at once. So implementations will have to include
>     code that copes with receiving potentially multiple wraps of the
>     ACE field (or just ignore the possibility).
>
>     You always need to cover this case because it is always possible
>     to lose ACKs  and then counter may wrap. However, I don’t think
>     you actually need to implement any logic to detect this. It only
>     means the counters on both ends are off by N x 8 but congestion
>     information is only valid that the point when it happens and
>     resyncing on outdated information doesn’t help anybody.
>
>
> So, in my understanding, there are two types of choices for 
> implementations on the other end.
> 1) need to decide whether ignore CE counter wraps or prepare for it  
> (for both A and B)
> 2) need to decide whether identify ACKs on pure ACKs by accecn or 
> accept possible ack ping-pong  (for only B)
>
> In this case,
> For 1)
>    Choosing (A) or (B) doesn't matter because both (A) and (B) have 
> this potential risk.
> For 2)
> If I think in this way, choosing (B) is not much different from 
> allowing both?

[BB] Yes,...
...but I should make it clear that these pros and cons solely describe 
the implementation complexity of each choice. They don't say anything 
about the benefit of the feedback itself being available during that 
switch-over round trip...

In scenarios where the direction of data switches, option A is simpler 
for one peer, but it denies its peer the benefit of continuing to 
maintain cwnd up to the point it starts sending. So:

If we pick A, no-one gets the benefit but there's not the complexity of 
the extra check for B
If we pick B, everyone gets the benefit and everyone has the complexity 
of the extra check for B
If we allow both, one peer only gets the benefit if the other peer 
chooses the complexity of the extra check for B.

If we allow both, one peer also has the uncertainty of not knowing 
whether the other peer implements B. An implementater might be able to 
develop a  better cwnd validation algorithm if it knows for certain 
whether absence of feedback from the other end is purely because it 
hasn't implemented option B.

Cheers

Bob

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 30 Mar 2021 02:35:18 -0700

Hi Bob,

On Thu, Mar 25, 2021 at 12:03 PM Bob Briscoe <[email-masked]> wrote:

> ...but I should make it clear that these pros and cons solely describe the
> implementation complexity of each choice. They don't say anything about the
> benefit of the feedback itself being available during that switch-over
> round trip...
>
> In scenarios where the direction of data switches, option A is simpler for
> one peer, but it denies its peer the benefit of continuing to maintain cwnd
> up to the point it starts sending. So:
>
> If we pick B, everyone gets the benefit and everyone has the complexity of
> the extra check for B
> If we allow both, one peer only gets the benefit if the other peer chooses
> the complexity of the extra check for B.
>
> If we allow both, one peer also has the uncertainty of not knowing whether
> the other peer implements B. An implementater might be able to develop a
> better cwnd validation algorithm if it knows for certain whether absence of
> feedback from the other end is purely because it hasn't implemented option
> B.
>

Yes, It seems that they all have pros and cons.
It would be good if we can get feedback from implementers here..
If there is no strong opinion on this point in the community, I think the
author will need to pick one at some point.
--
Yoshi

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 30 Mar 2021 13:12:45 +0200

As a co-author and implementor, I would prefer (B).

While I understand the concerns raised about additional ACKs, formally
these ACKs-for-CE-ACks do convey new information.

Also, the risk of rare spurious retransmissions (because of a
misclassification, or not considering present SACK / TS information) by
the NEW sender is IMHO lower, than the benefit of maintaining good
congetion state in the OLD sender for subsequent transmissions.

As mentioned earlier, additional heuristics (activating delayed timeout,
eliciting the response for 'n' immediately on receipt of 'n+1' (and
retaining the one pending delta) can further help diminish the
ping-pong, while an 'n' of 3 dramatically dampens the number of
ACKs-for-CE-ACKs.

However, I suspect that the above mentioned changes would not be
necessary in non-pathological circumstances at all.

Best regards,
   Richard

Am 30.03.2021 um 11:35 schrieb Yoshifumi Nishida:
> Hi Bob,
>
> On Thu, Mar 25, 2021 at 12:03 PM Bob Briscoe <[email-masked]
>>     On Mon, Mar 22, 2021 at 3:08 AM Mirja Kuehlewind
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         See inline.
>>
>>         >> As promised we set up a small design team on the single
>>         issue of occasionally ACKing pure ACKs if they carry new info
>>         (ECN marking at the IP layer). The design team has two
>>         solutions and everyone would be prepared to accept either, but
>>         preferences differ. So we're seeking wider opinions (more
>>         opinions obviously won't narrow the choices, but at least the
>>         WG can then make a decision informed by those who care).
>>         >>
>>         >> AccECN is a change to the TCP wire protocol that requires
>>         the packet count of congestion feedback to include any
>>         congestion experienced (CE) arriving on Pure ACKs (amongst
>>         other things). AccECN doesn't require Pure ACKs to be
>>         ECN-capable, but allows for them to be. Similarly, AccECN
>>         doesn't require any congestion response to CE on pure ACKs,
>>         but having the feedback information there allows a response to
>>         be added with a one-ended update, if desired/necessary.
>>         >> The above two bullets were designed to ensure that an ACK
>>         is triggered a) on the first sign of congestion, and b)
>>         frequently enough for the count of CE markings to be fed back
>>         using the 3-bit ACE field before it wraps, even if an
>>              occasional pure ACK is lost.
>>         >>
>>         >> We then realized that the wording could require an ACK to
>>         be triggered in response to a CE-marked pure ACK. The
>>         circumstance when this could occur would be when peer X sends
>>         a volley of data to Y then stops, and the path back from Y to
>>         X is congested (probably by other flow(s) so that many of the
>>         ACKs are CE-marked. The second bullet above under alternative
>>         (B) would require X to ACK every 'n-th' CE-marked pure ACK.
>>         However, if Y immediately started sending a volley of data to
>>         X, Y could misinterpret those ACKs (of ACKs) from X as DupACKs.
>>         >>
>>         >> A) Some of us prefer to completely prevent ACKs on pure
>>         ACKs, on the basis that they do not want to risk sometimes
>>         generating more ACKs today
>>         >> B) Others want to ensure that these rules will cause pure
>>         ACKs to be ACKed when the amount of CE on the ACKs merits it.
>>         But sparingly and strongly damping any ACK ping-pong.
>>         >>
>>         >> Hmm. This is not easy..
>>         >> BTW, I am wondering if this could leave it to
>>         implementations since both methods have pros and cons.
>>         > [BB] The problem is that the decision impacts the complexity
>>         of the other end, not just the implementer's own code.
>>         >
>>         > * If an implementer chooses (A) (doesn't ACK pure ACKs at
>>         all), when it finally does send some data, it's CE counter
>>         could have built up an (unlimited) store of unreported CE
>>         marks, which it will report all at once. So implementations
>>         will have to include code that copes with receiving
>>         potentially multiple wraps of the ACE field (or just ignore
>>         the possibility).
>>
>>         You always need to cover this case because it is always
>>         possible to lose ACKs  and then counter may wrap. However, I
>>         don’t think you actually need to implement any logic to detect
>>         this. It only means the counters on both ends are off by N x 8
>>         but congestion information is only valid that the point when
>>         it happens and resyncing on outdated information doesn’t help
>>         anybody.
>>
>>
>>     1) need to decide whether ignore CE counter wraps or prepare for
>>     it  (for both A and B)
>>        Choosing (A) or (B) doesn't matter because both (A) and (B)
>>     have this potential risk.
>>     For 2)
>     ...but I should make it clear that these pros and cons solely
>     describe the implementation complexity of each choice. They don't
>     say anything about the benefit of the feedback itself being
>     available during that switch-over round trip...
>
>     In scenarios where the direction of data switches, option A is
>     simpler for one peer, but it denies its peer the benefit of
>     continuing to maintain cwnd up to the point it starts sending. So:
>
>     If we pick A, no-one gets the benefit but there's not the complexity
>     of the extra check for B
>     If we pick B, everyone gets the benefit and everyone has the
>     complexity of the extra check for B
>     whether the other peer implements B. An implementater might be able
>     to develop a  better cwnd validation algorithm if it knows for
>     certain whether absence of feedback from the other end is purely
>     because it hasn't implemented option B.
>
>
> If there is no strong opinion on this point in the community, I think
> the author will need to pick one at some point.
> --
> Yoshi
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 30 Mar 2021 16:30:26 +0000

How is the sender of the pure ACKs supposed to react upon receiving the feedback that ACKs themselves are causing congestion? Option B conveys this additional information but how is it supposed to be used? Can a pure receiver get into such a state and does it need to start throttling ACKs?

ECN++ draft section 3.3.3 leaves the reaction to the AccECN draft but I am not able to locate the text in the latest AccECN draft. 

-----Original Message-----
From: tcpm <[email-masked]> On Behalf Of Scheffenegger, Richard
Sent: Tuesday, March 30, 2021 4:13 AM
To: Yoshifumi Nishida <[email-masked]>; Bob Briscoe <[email-masked]>
Cc: Yoshifumi Nishada <[email-masked]>; tcpm IETF list <[email-masked]>; Mirja Kuehlewind <[email-masked]>
Subject: [EXTERNAL] Re: [tcpm] Seeking WG opinions on ACKing ACKs with good cause (was: Possible error in accurate-ecn)

While I understand the concerns raised about additional ACKs, formally these ACKs-for-CE-ACks do convey new information.

Also, the risk of rare spurious retransmissions (because of a misclassification, or not considering present SACK / TS information) by the NEW sender is IMHO lower, than the benefit of maintaining good congetion state in the OLD sender for subsequent transmissions.

As mentioned earlier, additional heuristics (activating delayed timeout, eliciting the response for 'n' immediately on receipt of 'n+1' (and retaining the one pending delta) can further help diminish the ping-pong, while an 'n' of 3 dramatically dampens the number of ACKs-for-CE-ACKs.

However, I suspect that the above mentioned changes would not be necessary in non-pathological circumstances at all.

Best regards,
   Richard

Am 30.03.2021 um 11:35 schrieb Yoshifumi Nishida:
>>         https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0
>>         <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0>
>>         >>
>>         https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0
>>         <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0>
>>         ):
>>         don't think you actually need to implement any logic to detect
>>         it happens and resyncing on outdated information doesn't help
>>         anybody.
>>
>>

_______________________________________________
tcpm mailing list
[email-masked]
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=zEm1fZiCGkXdJIK3RUzItPrKvuRm9WypbyWzRX0D8wA%3D&amp;reserved=0

---

From: Vidhi Goel <[email-masked]>
Date: Tue, 30 Mar 2021 11:04:04 -0700

> How is the sender of the pure ACKs supposed to react upon receiving the feedback that ACKs themselves are causing congestion? Option B conveys this additional information but how is it supposed to…

This was discussed briefly during the WG. The receiver will act by reducing the ssthresh and cwnd (if it is not already minimum). This wouldn’t impact the receiver’s ACKing, but it would have an impact when the receiver starts sending data.

Vidhi

> On Mar 30, 2021, at 9:30 AM, Praveen Balasubramanian <pravb=[email-masked]> wrote:
> 
> From: tcpm <[email-masked] <mailto:[email-masked]>> On Behalf Of Scheffenegger, Richard
> To: Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>>; Bob Briscoe <[email-masked] <mailto:[email-masked]>>
> Cc: Yoshifumi Nishada <[email-masked] <mailto:[email-masked]>>; tcpm IETF list <[email-masked] <mailto:[email-masked]>>; Mirja Kuehlewind <[email-masked] <mailto:[email-masked]>>
>>>        https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0>
>>>        <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F110%2Fmaterials%2Fslides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00%23page%3D6&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=t5T6laDBNruH4VDgCEjhHYI7IpbVAOrktd9iCw1MpKg%3D&amp;reserved=0>>
>>>>> 
>>>        https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0>
>>>        <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-14%23section-3.2.2.5&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FJrrumB1WBtsjRgiDPO5hWlnbveUaUWb8kAF2Qhk2xA%3D&amp;reserved=0>>
>>>        ):
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=zEm1fZiCGkXdJIK3RUzItPrKvuRm9WypbyWzRX0D8wA%3D&amp;reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Cpravb%40microsoft.com%7C12697bf5f0b94211a1a508d8f36cf000%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637526997015192764%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=zEm1fZiCGkXdJIK3RUzItPrKvuRm9WypbyWzRX0D8wA%3D&amp;reserved=0>
> 
> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 30 Mar 2021 21:10:09 +0300

> On 30 Mar, 2021, at 9:04 pm, Vidhi Goel <vidhi_goel=[email-masked]> wrote:
> 

The information could also be used to tune the Delayed Ack ratio, via (for example) Gomez' Ack Rate Request option.  That *would* have a congestion control effect on ack traffic.

 - Jonathan Morton

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 30 Mar 2021 18:51:19 +0000

This is uncharted territory for congestion control. To date ACK packets have not been considered part of bytes in flight in TCP and QUIC. Congestion control itself impeding the feedback loop has consequences.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Tuesday, March 30, 2021 11:10 AM
To: Vidhi Goel <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause (was: Possible error in accurate-ecn)

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 30 Mar 2021 22:16:49 +0300

> On 30 Mar, 2021, at 9:51 pm, Praveen Balasubramanian <[email-masked]> wrote:
> 

I'm aware of the tradeoffs.  I'm just saying that Ack Rate Request can make use of this information, among other sources, and choose an appropriate level of feedback that suits the sender's needs and path characteristics.

 - Jonathan Morton

---

From: Vidhi Goel <[email-masked]>
Date: Tue, 30 Mar 2021 12:18:13 -0700

> The information could also be used to tune the Delayed Ack ratio, via (for example) Gomez' Ack Rate Request option.  That *would* have a congestion control effect on ack traffic.

I don’t think we should change the ACK ratio, at the very least, we shouldn’t reduce it. ACKs are used to convey crucial information to the sender esp. when there is a congestion.

> This is uncharted territory for congestion control. To date ACK packets have not been considered part of bytes in flight in TCP and QUIC. Congestion control itself impeding the feedback loop has co…

Agreed. And that’s why I think the receiver should not reduce / throttle ACKs based on AccECN feedback on its pure ACKs.

-
Vidhi

> On Mar 30, 2021, at 11:51 AM, Praveen Balasubramanian <[email-masked]> wrote:
>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 30 Mar 2021 22:52:41 +0300

> On 30 Mar, 2021, at 10:18 pm, Vidhi Goel <[email-masked]> wrote:
> 

That really depends which direction the congestion is in.  No doubt plenty of acks are helpful when the forward path is congested.

But we are talking here about congestion being reported, by the network, to be on the path carrying the ack stream.  This means that either the network is on the point of *dropping* acks (and would already have done so, if they weren't marked ECT), which effectively changes the ack ratio without the endpoints' consent - OR the ack stream is now the limiting factor, via the ack clock mechanism, for the forward-path data throughput, so reducing the number of acks being generated would allow an increase in goodput.  In both cases, a mechanism for the sender to adjust the ack ratio of the receiver makes sense.

We can defer detailed discussion of the advantages or otherwise to threads on Ack Rate Request.

 - Jonathan Morton