Subject: [tcpm] Possible error in accurate-ecn

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Nov 2020 00:31:13 +0000

Mirja, Richard, Ilpo, tcpm list,

I've just been reading through the accurate-ecn draft to double-check. I 
think there's a problem with the following text...

3.2.2.5.1.  Data Receiver Safety Procedures

    An AccECN Data Receiver:

    o  SHOULD immediately send an ACK whenever a data packet marked CE
       arrives after the previous [data] packet was not CE.

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
       than 6.

...
    For the avoidance of doubt, the change-triggered ACK mechanism is
    deliberately worded to solely apply to data packets, and to ignore
    the arrival of a control packet with no payload, because it is
    important that TCP does not acknowledge pure ACKs.

In the first bullet, I think it doesn't matter whether the previous 
packet marked CE was a data packet or a pure ACK (i.e we should remove 
the second occurrence of 'data' that I have put in [square brackets].

The second bullet doesn't consider the possibility that the 'n'th CE 
mark might arrive on a pure ACK. Then, the wording as it stands says the 
Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a 
pure ACK, but I'm not actually sure it does any harm to do so in this 
case (it cannot cause an infinite loop of ACKs). However, given it would 
be unorthodox, we maybe ought to rule it out by rewording anyway?

Thoughts anyone?

Bob

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 10 Nov 2020 11:08:03 +0100

Hi Bob,

Please see below.

> On 10. Nov 2020, at 01:31, Bob Briscoe <[email-masked]> wrote:
> 
> I've just been reading through the accurate-ecn draft to double-check. I think there's a problem with the following text...
> 
> In the first bullet, I think it doesn't matter whether the previous packet marked CE was a data packet or a pure ACK (i.e we should remove the second occurrence of 'data' that I have put in [square brackets].

I believe it does matter. If the previous packet was a pure ACK and was CE marked, you didn’t send an ACK and so you should immediately send one now. If the previous packet was a data packet and CE marked, you don’t need to send an immediate ACK because you already did this with the previous packet. However, it text might be a but ambiguous because what is meant is “if the previous packet was a data packet and CE marked”. So this does not apply if we e.g. have a CE-marked data packets, then a pure ACK that is not CE marked, and then another CE-marked data packet.  

> 
> The second bullet doesn't consider the possibility that the 'n'th  CE mark might arrive on a pure ACK. Then, the wording as it stands says the Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a pure ACK, but I'm not actually sure it does any harm to do so in this case (it cannot cause an infinite loop of ACKs). However, given it would be unorthodox, we maybe ought to rule it out by rewording anyway?

I think we also can leave this as it is because if that ’n’ packet is a pure ACK that still means that you have unacked data packets with CE marks and you should trigger an ACK for those packet now (rather than waiting for the delayed ACK timer to expire). However this case is less important. Also we should probably make sure that this doesn’t apply if there are only pure ACKs with CE marks, maybe by add “if unacknowledged data are outstanding” or something.

Mirja

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Nov 2020 16:38:57 +0000

Mirja,

On 10/11/2020 10:08, Mirja Kuehlewind wrote:
> Hi Bob, Please see below.
>> On 10. Nov 2020, at 01:31, Bob Briscoe <[email-masked]> 
>> wrote: Mirja, Richard, Ilpo, tcpm list, I've just been reading 
>> through the accurate-ecn draft to double-check. I think there's a 
>> problem with the following text... 
>> packet marked CE was a data packet or a pure ACK (i.e we should 
>> remove the second occurrence of 'data' that I have put in [square 
>> brackets]. 
> [MK] I believe it does matter. If the previous packet was a pure ACK 
> and was CE marked, you didn’t send an ACK and so you should 
> immediately send one now. 

[BB] Neither the current text, nor my proposed edit, nor the text you 
propose below, says this. And I don't think it needs to. Surely this 
falls into the case of the second bullet ('n' CE marks in a row before 
needing to feed back anything).

> [MK] If the previous packet was a data packet and CE marked, you don’t 
> need to send an immediate ACK because you already did this with the 
> previous packet. However, it text might be a but ambiguous because 
> what is meant is “if the previous packet was a data packet and CE 
> marked”. So this does not apply if we e.g. have a CE-marked data 
> packets, then a pure ACK that is not CE marked, and then another 
> CE-marked data packet. 

[BB] An AQM will typically mark packets irrespective of their type or 
size (that's a feature not a bug). So there's no reason for the receiver 
to treat some marks differently to others, whatever type or size of 
packets they're on.

Let's revisit your last sentence above with a more general scenario - 
with more than one pure ACK in between the data - actually quite a 
common pattern. I.e. a server receives a small request, sends its larger 
response, then when that response has completed, the client sends 
another small request. So the server's TCP will see one data packet, a 
load of pure ACKs then one data packet again.

Now consider the case where both arriving data packets (the requests) 
are CE-marked. There are three relevant cases for the marking of the 
pure ACKs in between:
1) no CE-marked pure ACKs
2) some non-CE-marked pure ACKs.
3) all CE-marked pure ACKs

In cases 1 & 2, there has been a transition from none to some CE marks, 
but there hasn't been an opportunity to report it yet, so use the first 
opportunity to ACK a data packet.
In case 3, surely the server doesn't need to send feedback unless there 
are 'n' CE marks to report.

So how about this text to replace the first bullet:

    An AccECN Data Receiver:

    o  SHOULD send an ACK at the first opportunity (in response to the
       arrival of a data packet) whenever a mix of non-CE marked
       and CE marked packets have arrived since any previous ACK.

Moving on to the second bullet...
>> mark might arrive on a pure ACK. Then, the wording as it stands says 
>> the Data Receiver MUST immediately ACK a pure ACK. I know TCP never 
>> ACKs a pure ACK, but I'm not actually sure it does any harm to do so 
>> in this case (it cannot cause an infinite loop of ACKs). However, 
>> given it would be unorthodox, we maybe ought to rule it out by 
>> rewording anyway? 
> [MK] I think we also can leave this as it is because if that ’n’ 
> packet is a pure ACK that still means that you have unacked data 
> packets with CE marks and you should trigger an ACK for those packet 
> now (rather than waiting for the delayed ACK timer to expire). However 
> this case is less important. Also we should probably make sure that 
> this doesn’t apply if there are only pure ACKs with CE marks, maybe by 
> add “if unacknowledged data are outstanding” or something.

[BB] By the end of your last para you start to realize that, if the n'th 
CE mark arrives on pure ACK, it doesn't say anything about whether the 
previous (n-1) CE marks were on data or pure ACKs.

Consider another common scenario; a server delivering chunks of video 
over a high BDP path (e.g. 500 packets per RTT), and going idle between 
times (perhaps in response to an "exceeded high watermark" data packet 
from the server). Now consider the possibility that the reverse path 
bottleneck builds a queue during each chunk, and starts CE marking 
towards the end of each chunk. Then, after the server stops sending a 
chunk, it continues to see a couple of hundred CE-marked pure ACKs 
arriving.

However, by your proposed rule, it doesn't send any feedback, 'cos it 
has no data to acknowledge. So the client cannot regulate its ACK 
stream. That's just tough, 'cos the client will stop sending more pure 
ACKs after it has received the last data, which is before any later 
feedback from the server can reach it.

However, if the client later sends a data packet that the server can 
acknowledge (e.g. "a low watermark" message), the server's ACE counter 
will have wrapped dozens of times, and the client won't be able to work 
out how many. That doesn't matter if a few round trips have elapsed - 
the congestion info will have become stale. But it does matter if it's 
fairly soon.

Instead, if we keep the text of the second bullet... if the server's 'n' 
was 2, it would send a pure ACK for every other pure ACK it received. 
But, what if these pure ACKs from the server were also CE-marked? Then 
the client would feed back pure ACKs using its 'n'. This regression 
would eventually die out, but I don't like it.

This conversation about the second bullet presumes AccECN is being used 
for ACK congestion control. I'm not comfortable with writing anything 
specific about ACK congestion control into a draft on the standards 
track. But I don't want to preclude AccECN being used for ACK CC.

I believe the second bullet needs something changed to limit the ACKing 
of pure ACKs. However, before we put effort into wordsmithing, I'd like 
to hear whether you agree with the general idea of not defining what to 
do for ACK CC, but not precluding it.

Cheers

Bob

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 11 Nov 2020 01:37:27 -0800

On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind <[email-masked]>
wrote:

> packet marked CE was a data packet or a pure ACK (i.e we should remove the
> second occurrence of 'data' that I have put in [square brackets].
>
> I believe it does matter. If the previous packet was a pure ACK and was CE
> marked, you didn’t send an ACK and so you should immediately send one now.
> If the previous packet was a data packet and CE marked, you don’t need to
> send an immediate ACK because you already did this with the previous
> packet. However, it text might be a but ambiguous because what is meant is
> “if the previous packet was a data packet and CE marked”. So this does not
> apply if we e.g. have a CE-marked data packets, then a pure ACK that is not
> CE marked, and then another CE-marked data packet.
>
> >
> Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a pure
> ACK, but I'm not actually sure it does any harm to do so in this case (it
> cannot cause an infinite loop of ACKs). However, given it would be
> unorthodox, we maybe ought to rule it out by rewording anyway?
>

even though it won't cause an infinite loop, can they be dup acks? If so,
it doesn't look good to me especially when early retransmit is activated.

> I think we also can leave this as it is because if that ’n’ packet is a
> pure ACK that still means that you have unacked data packets with CE marks
> and you should trigger an ACK for those packet now (rather than waiting for
> the delayed ACK timer to expire). However this case is less important. Also
> we should probably make sure that this doesn’t apply if there are only pure
> ACKs with CE marks, maybe by add “if unacknowledged data are outstanding”
> or something.
>
>
"can send a pure ack on a pure ack when the ack value advances"?
--
Yoshi

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Nov 2020 23:18:05 +0000

Yoshi,

On 11/11/2020 09:37, Yoshifumi Nishida wrote:
>
>
> On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Bob,
>
>     Please see below.
>
>     > On 10. Nov 2020, at 01:31, Bob Briscoe <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     > I've just been reading through the accurate-ecn draft to
>     double-check. I think there's a problem with the following text...
>     >
>     > In the first bullet, I think it doesn't matter whether the
>     previous packet marked CE was a data packet or a pure ACK (i.e we
>     should remove the second occurrence of 'data' that I have put in
>     [square brackets].
>
>     I believe it does matter. If the previous packet was a pure ACK
>     immediately send one now. If the previous packet was a data packet
>     and CE marked, you don’t need to send an immediate ACK because you
>     already did this with the previous packet. However, it text might
>     be a but ambiguous because what is meant is “if the previous
>     packet was a data packet and CE marked”. So this does not apply if
>     we e.g. have a CE-marked data packets, then a pure ACK that is not
>     > The second bullet doesn't consider the possibility that the
>     'n'th  CE mark might arrive on a pure ACK. Then, the wording as it
>     stands says the Data Receiver MUST immediately ACK a pure ACK. I
>     know TCP never ACKs a pure ACK, but I'm not actually sure it does
>     any harm to do so in this case (it cannot cause an infinite loop
>     of ACKs). However, given it would be unorthodox, we maybe ought to
>     rule it out by rewording anyway?
>
>
> even though it won't cause an infinite loop, can they be dup acks? If 
> so, it doesn't look good to me especially when early retransmit is 
> activated.

[BB] I'm not expert in TCP loss recovery but, surely if TCP A has 
nothing outstanding, it cannot consider receipt of dup acknowledgements 
from TCP B as a sign of loss? Nonetheless, there would be a chance that 
such a dup acknowledgement might be in flight when TCP A happened to 
have just sent some more data, and they crossed in flight.

To avoid this case, it would be possible for TCP B to send pure ACK(s) 
with SEG.SEQ = SND.NXT-1 (using the same trick as keepalive probes). 
Being out of window, each of these pure ACKs would elicit a further pure 
ACK from TCP A. However, that would acknowledge SND-NXT, so the loop 
would stop there. Nonetheless, if TCP B started sending data again, and 
if TCP B's data crossed the pure ACKs that it had recently elicited from 
TCP A, once these pure ACKs arrived at TCP B, I think it would treat 
them as DupACKs.

It's seems possible for TCP B to recognize that it elicited these 
DupACKs from TCP A, and to therefore work out that it could ignore them.

I've kept on digging into this particular obscure corner of ACK 
congestion control, only 'cos I wanted to see if there was a useful end 
to this hole (so it would be worth the AccECN RFC allowing others to dig 
there later, rather than ruling it out).

Nonethless, the AccECN draft itself doesn't have to go anywhere near 
this subject - it can leave all that for a potential future draft about 
using AccECN for ACK congestion control.

>     I think we also can leave this as it is because if that ’n’ packet
>     is a pure ACK that still means that you have unacked data packets
>     with CE marks and you should trigger an ACK for those packet now
>     (rather than waiting for the delayed ACK timer to expire). However
>     this case is less important. Also we should probably make sure
>     that this doesn’t apply if there are only pure ACKs with CE marks,
>     maybe by add “if unacknowledged data are outstanding” or something.
>

[BB] Yes. That's a fairly safe thing to require.
I think the best strategy here is to require an immediate ACK in 
response to a data packet if there are at least 'n' CE marks to be 
reported. And to extend that requirement to immediately ACKing a pure 
ACK if the ACK value has advanced since the last ACK. But just to say 
nothing about ACKing pure ACKs in other cases, so we don't preclude a 
case that can be made safe in future (e.g. with something like the 
keepalive approach above).

If you (Yoshi), Mirja and others agree, let's try to work out the 
wording of this second bullet.

Bob

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 12 Nov 2020 14:41:49 -0800

Hi Bob,

On Wed, Nov 11, 2020 at 3:18 PM Bob Briscoe <[email-masked]> wrote:

>> > I've just been reading through the accurate-ecn draft to double-check.
>> I think there's a problem with the following text...
>> >
>> I believe it does matter. If the previous packet was a pure ACK and was
>> CE marked, you didn’t send an ACK and so you should immediately send one
>> now. If the previous packet was a data packet and CE marked, you don’t need
>> to send an immediate ACK because you already did this with the previous
> [BB] I'm not expert in TCP loss recovery but, surely if TCP A has nothing
> outstanding, it cannot consider receipt of dup acknowledgements from TCP B
> as a sign of loss? Nonetheless, there would be a chance that such a dup
> acknowledgement might be in flight when TCP A happened to have just sent
> some more data, and they crossed in flight.
>
> with SEG.SEQ = SND.NXT-1 (using the same trick as keepalive probes). Being
> out of window, each of these pure ACKs would elicit a further pure ACK from
> TCP A. However, that would acknowledge SND-NXT, so the loop would stop
> there. Nonetheless, if TCP B started sending data again, and if TCP B's
> data crossed the pure ACKs that it had recently elicited from TCP A, once
> these pure ACKs arrived at TCP B, I think it would treat them as DupACKs.
>
> It's seems possible for TCP B to recognize that it elicited these DupACKs
> from TCP A, and to therefore work out that it could ignore them.
>
> I've kept on digging into this particular obscure corner of ACK congestion
> control, only 'cos I wanted to see if there was a useful end to this hole
> (so it would be worth the AccECN RFC allowing others to dig there later,
> rather than ruling it out).
>
> Nonethless, the AccECN draft itself doesn't have to go anywhere near this
> subject - it can leave all that for a potential future draft about using
> AccECN for ACK congestion control.
>

I personally prefer not to specify very detailed logic for ACK CC here
unless it is crucial for the doc.
Since the doc is aiming to be a PS, I think it would be better to avoid
risks for nasty corner cases caused by some tricky techniques.

> I think the best strategy here is to require an immediate ACK in response
> to a data packet if there are at least 'n' CE marks to be reported. And to
> extend that requirement to immediately ACKing a pure ACK if the ACK value
> has advanced since the last ACK. But just to say nothing about ACKing pure
> ACKs in other cases, so we don't preclude a case that can be made safe in
> future (e.g. with something like the keepalive approach above).
>
> If you (Yoshi), Mirja and others agree, let's try to work out the wording
> of this second bullet.
>

Works for me. If you could have proposed texts, I will take a look.
--
Yoshi

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 17 Nov 2020 09:37:26 +0100

Without having read any more recent responses,

If you want to use AccECN to eventually control the ACK flow, I think
the 2nd bullet point is useful.

In order to not continously ACK an ACK (if they all get CE marked), the
sentence may need to be revised to

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' MUST NOT be lower than 2 and MUST be
       no greater than 6.

That should ensure that any ACK-for-ACK will die down rather rapidely,
if the stream of ACKs on the return path suddenly all get CE marked
(e.g. path change across a device, which erroneously overrides TOS and
happens to paint both IP ECN bits too).

Richard

Am 10.11.2020 um 01:31 schrieb Bob Briscoe:
> Bob Briscoehttp://bobbriscoe.net/
>
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 17 Nov 2020 09:48:36 +0100

See my suggestion; The critical requirement for the 2nd bullet point is,
to have n > 1 for pure, CE-marked ACKs. The exponential decay, after how
many ACKs triggered by CE-marked ACKs these ACK-for-ACK stop, strongly
depends on "n", so the recommendation should be to use a higher value of
n in the case where you only observe a stream of pure ACKs (with or
without marks; only those marked would account against "n" though).

In the case of data packets mixed in, you don't want to delay
excessively, thus "n" may be choosen lower...

Richard

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 17 Nov 2020 09:53:38 +0100

This would preclude the use of AccECN for AckCC purposes; effectively
allowing ACKs only to be sent for data packets.. (or a stream of
CE-marked ACKs after an unacked "uneven" data packet was received before).

If the last prior data packet was already ACKs, and then a stream of
CE-marked pure ACKs is observed, we loose synchronizity of the AccECN
counters...

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 23 Nov 2020 12:15:06 +0100

Hi Bob, hi all,

Sorry for my late reply. See inline.

> On 12. Nov 2020, at 23:41, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Bob,
> 
>> On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind <[email-masked]> wrote:
>> Hi Bob,
>> 
>> Please see below.
>> 
>> even though it won't cause an infinite loop, can they be dup acks? If so, it doesn't look good to me especially when early retransmit is activated.
> 
> [BB] I'm not expert in TCP loss recovery but, surely if TCP A has nothing outstanding, it cannot consider receipt of dup acknowledgements from TCP B as a sign of loss? Nonetheless, there would be a chance that such a dup acknowledgement might be in flight when TCP A happened to have just sent some more data, and they crossed in flight. 
> 
> To avoid this case, it would be possible for TCP B to send pure ACK(s) with SEG.SEQ = SND.NXT-1 (using the same trick as keepalive probes). Being out of window, each of these pure ACKs would elicit a further pure ACK from TCP A. However, that would acknowledge SND-NXT, so the loop would stop there. Nonetheless, if TCP B started sending data again, and if TCP B's data crossed the pure ACKs that it had recently elicited from TCP A, once these pure ACKs arrived at TCP B, I think it would treat them as DupACKs.
> 
> It's seems possible for TCP B to recognize that it elicited these DupACKs from TCP A, and to therefore work out that it could ignore them. 
> 
> I've kept on digging into this particular obscure corner of ACK congestion control, only 'cos I wanted to see if there was a useful end to this hole (so it would be worth the AccECN RFC allowing others to dig there later, rather than ruling it out).
> 
> Nonethless, the AccECN draft itself doesn't have to go anywhere near this subject - it can leave all that for a potential future draft about using AccECN for ACK congestion control. 
> 
> I personally prefer not to specify very detailed logic for ACK CC here unless it is crucial for the doc. 
> Since the doc is aiming to be a PS, I think it would be better to avoid risks for nasty corner cases caused by some tricky techniques. 

I would leave anything on ACK CC also for a separate draft.

> I think the best strategy here is to require an immediate ACK in response to a data packet if there are at least 'n' CE marks to be reported. And to extend that requirement to immediately ACKing a pure ACK if the ACK value has advanced since the last ACK. But just to say nothing about ACKing pure ACKs in other cases, so we don't preclude a case that can be made safe in future (e.g. with something like the keepalive approach above).
> 
> If you (Yoshi), Mirja and others agree, let's try to work out the wording of this second bullet. 
> 

Works for me too.

On Richard’s point that this could lead to synchronicity lost of the AccECN
counters: Only if you don’t use the option. And as long as ACK CC is not implemented, it’s probably also we can live with.

Mirja