Subject: RE: ECN in QUIC

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 20 Sep 2017 08:39:15 +0000

Hi

And sorry for the sloth-ish response (you should see me type  a text message..).
To summarize the two main questions

  1.  ECN in ACK frame or in a separate frame : Given the recent discussion on the list and fear that the ACK frame is already enough complex, it seems unwise to try to cram in additional info in the ACK frames. So my personal feeling is that we should use a separate frame type for ECN , but this frame type must be a MUST to implement and support ! Comments are very welcome.
  2.  As regards to the level of detail : There are pros and cons with both alternatives (marked bytes or detailed indication). I would like to hear the opinions from others as well on this matter .

/Ingemar

From: Colin Perkins [mailto:[email-masked]]
Sent: den 16 augusti 2017 12:53
To: Ingemar Johansson S <[email-masked]>
Cc: QUIC IETF mailing list <[email-masked]>; Magnus Westerlund <[email-masked]>; Bob Briscoe ([email-masked]) <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Eggert, Lars ([email-masked]) <[email-masked]>; marcelo bagnulo braun <[email-masked]>; Mirja Kühlewind <[email-masked]>; Piers O'Hanlon <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Subject: Re: ECN in QUIC

On 16 Aug 2017, at 10:33, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:

Hi

Finally back from vacation, and very grateful for the support to continue the work to add ECN in QUIC.
Just to recap.. there were two main topics raised at the meeting

1) ECN info in ACK frame or in dedicated frame : There were concerns about adding extra complexity in an already potentially complex ACK frame, one can have differing opinions about the complexity but can understand the concerns. As far as I am concerned, a separate frame type for ECN is possible, possibly one need to add information about the amount of not-ECT marked packets as well to keep the signaling robust, this needs further investigation though. One concern with a separate ECN frame is that it becomes a not-implemented or optional feature, is there any reason to be worried about this ?

2) More detailed ECN information : Earlier versions of the ECN in QUIC draft (seehttps://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt ) provided with examples. We (Myself, Koen, Mirja and Praveen) discussed this and we could not come up with any use case where it is beneficial to know exactly how each packet is ECN marked. I know that this kind of detailed ECN information is suggested for the generic feedback for RMCAT and I personally have a problem to see the gain with the detailed ECN information also here. Input from others is very welcome here.

For the RMCAT format, we wanted per-packet loss and timing information, and it was as easy to feedback per-packet ECN information along with it as to design something different.

A benefit of per-packet ECN marking could be to allow a congestion controller that reacted differently to bursts of consecutive ECN marks than it did to isolated ECN marks, given the same fraction of marked packets (i.e., that reacted to ECN marking events rather than ECN marking rate, like how TCP responds to loss events). I don’t think we have such a thing, but certainly in the context of RMCAT where we’re experimenting with novel congestion control schemes for traffic that has very different characteristics to traditional bulk flows, it might be plausible. Per-packet marking information is also useful for troubleshooting.

Certainly we need to know number of NotECT, ECT(0), ECT(1), and ECN-CE marks since the last report, but I guess that’s already possible.

Cheers,
Colin

There are a consequences with detailed ECN marking information.
a) necessary to correlate with the list of transmitted packets, this increases amount of code on sender side, not sure of that is a large concern as lookup is anyway needed to process incoming ACKs
b) necessary to embed ECN information in ACK frame ?, at least this was my conclusion when I devised the detailed ECN marking info in the 01 version of the draft.

Comments are welcome
/Ingemar

==================================
Ingemar Johansson  M.Sc.
Master Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com<x-msg://103/www.ericsson.com>

A mistake is to commit a misunderstanding
                     Bob Dylan
==================================

--
Colin Perkins
https://csperkins.org/

---

From: Ian Swett <[email-masked]>
Date: Wed, 20 Sep 2017 08:41:04 -0400

On Wed, Sep 20, 2017 at 4:39 AM, Ingemar Johansson S <
[email-masked]> wrote:

>    1. ECN in ACK frame or in a separate frame : Given the recent
>    discussion on the list and fear that the ACK frame is already enough
>    complex, it seems unwise to try to cram in additional info in the ACK
>    frames. So my personal feeling is that we should use a separate frame type
>    for ECN , but this frame type must be a MUST to implement and support !
>    Comments are very welcome.
>
> I was neutral on this before, but I started thinking about implementing
this, and separating the frames makes it more complex, and potentially
really, really difficult to get the implementation correct.  As an example,
if the frames were accidentally sent in two separate packets, I would count
the packet as acked, and potentially increase the congestion window and
other state variables.  I might even send an extra packet based on this
information.  Then in the next packet I'd realize that packet had an ECN
marking, and actually should have been treated as lost from a congestion
controller perspective.  In order to do this correctly, I'd have to undo a
bunch of things.  My experience with undo is that it's terribly error
prone, and I would never recommend it.

Alternately, the ack could get lost, and the next packet would convey ECN
marking information for packets that hadn't been acknowledged yet.  Which
is just weird.

If we require support for ECN, then the implementation has a certain extra
level of complexity.  As illustrated above, putting it in one frame is
actually simpler overall if the congestion controller needs to use all the
information at once to be implemented correctly.  This same logic applies
to timestamps as well, in my opinion.

>
>    1.
>    2. As regards to the level of detail : There are pros and cons with
>    both alternatives (marked bytes or detailed indication). I would like to
>    hear the opinions from others as well on this matter .
>
>
>
> /Ingemar
>
>
>
> *From:* Colin Perkins [mailto:[email-masked]]
> *Sent:* den 16 augusti 2017 12:53
> *To:* Ingemar Johansson S <[email-masked]>
> *Cc:* QUIC IETF mailing list <[email-masked]>; Magnus Westerlund <
> [email-masked]>; Bob Briscoe ([email-masked]) <
> [email-masked]>; Praveen Balasubramanian <[email-masked]>;
> Eggert, Lars ([email-masked]) <[email-masked]>; marcelo bagnulo braun <
> [email-masked]>; Mirja Kühlewind <[email-masked]>;
> Piers O'Hanlon <[email-masked]>; De Schepper, Koen (Nokia -
> BE/Antwerp) <[email-masked]>
> *Subject:* Re: ECN in QUIC
>
>
>
>
>
> On 16 Aug 2017, at 10:33, Ingemar Johansson S <
> [email-masked]> wrote:
>
>
>
> Hi
>
>
>
> 1) ECN info in ACK frame or in dedicated frame : There were concerns about
> adding extra complexity in an already potentially complex ACK frame, one
> can have differing opinions about the complexity but can understand the
> concerns. As far as I am concerned, a separate frame type for ECN is
> possible, possibly one need to add information about the amount of not-ECT
> marked packets as well to keep the signaling robust, this needs further
> investigation though. One concern with a separate ECN frame is that it
> becomes a not-implemented or optional feature, is there any reason to be
> worried about this ?
>
>
>
> 2) More detailed ECN information : Earlier versions of the ECN in QUIC
> draft (seehttps://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt )
> provided with examples. We (Myself, Koen, Mirja and Praveen) discussed this
> and we could not come up with any use case where it is beneficial to know
> exactly how each packet is ECN marked. I know that this kind of detailed
> ECN information is suggested for the generic feedback for RMCAT and I
> personally have a problem to see the gain with the detailed ECN information
> also here. Input from others is very welcome here.
>
>
>
> For the RMCAT format, we wanted per-packet loss and timing information,
> and it was as easy to feedback per-packet ECN information along with it as
> to design something different.
>
>
>
> A benefit of per-packet ECN marking could be to allow a congestion
> controller that reacted differently to bursts of consecutive ECN marks than
> it did to isolated ECN marks, given the same fraction of marked packets
> (i.e., that reacted to ECN marking events rather than ECN marking rate,
> like how TCP responds to loss events). I don’t think we have such a thing,
> but certainly in the context of RMCAT where we’re experimenting with novel
> congestion control schemes for traffic that has very different
> characteristics to traditional bulk flows, it might be plausible.
> Per-packet marking information is also useful for troubleshooting.
>
>
>
> Certainly we need to know number of NotECT, ECT(0), ECT(1), and ECN-CE
> marks since the last report, but I guess that’s already possible.
>
>
>
> Cheers,
>
> Colin
>
>
>
>
>
>
>
>
>
> a) necessary to correlate with the list of transmitted packets, this
> increases amount of code on sender side, not sure of that is a large
> concern as lookup is anyway needed to process incoming ACKs
>
> b) necessary to embed ECN information in ACK frame ?, at least this was my
> conclusion when I devised the detailed ECN marking info in the 01 version
> of the draft.
>
>
>
> Comments are welcome
>
> /Ingemar
>
>
>
>
>
>
>
> SMS/MMS +46-73 078 3289 <+46%2073%20078%2032%2089>
>
> [email-masked]
>
> www.ericsson.com
>
>
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 20 Sep 2017 13:27:58 +0000

Thanks Ian for the prompt response.
I must admit that I have perhaps overlooked possible combos of loss and ECN, the price you pay for working too much with simulators. Your arguments are very strong indeed.

In the SCReAM congestion control (RMCAT work) the ECN and loss info comes in the same feedback and the handling of loss and ECN is straightforward there. Personally I would prefer ECN in the ACK frames and the earlier drafts suggested formats for that, I however got the impression that there is a certain resistance against additional things in the ACK frame ?.

/Ingemar

From: Ian Swett [mailto:[email-masked]]
Sent: den 20 september 2017 14:41
To: Ingemar Johansson S <[email-masked]>
Cc: Colin Perkins <[email-masked]>; Magnus Westerlund <[email-masked]>; Bob Briscoe ([email-masked]) <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Eggert, Lars ([email-masked]) <[email-masked]>; marcelo bagnulo braun <[email-masked]>; Mirja Kühlewind <[email-masked]>; Piers O'Hanlon <[email-masked]>; QUIC IETF mailing list <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Subject: Re: ECN in QUIC

On Wed, Sep 20, 2017 at 4:39 AM, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:
Hi

  1.  ECN in ACK frame or in a separate frame : Given the recent discussion on the list and fear that the ACK frame is already enough complex, it seems unwise to try to cram in additional info in the ACK frames. So my personal feeling is that we should use a separate frame type for ECN , but this frame type must be a MUST to implement and support ! Comments are very welcome.
I was neutral on this before, but I started thinking about implementing this, and separating the frames makes it more complex, and potentially really, really difficult to get the implementation correct.  As an example, if the frames were accidentally sent in two separate packets, I would count the packet as acked, and potentially increase the congestion window and other state variables.  I might even send an extra packet based on this information.  Then in the next packet I'd realize that packet had an ECN marking, and actually should have been treated as lost from a congestion controller perspective.  In order to do this correctly, I'd have to undo a bunch of things.  My experience with undo is that it's terribly error prone, and I would never recommend it.

Alternately, the ack could get lost, and the next packet would convey ECN marking information for packets that hadn't been acknowledged yet.  Which is just weird.

If we require support for ECN, then the implementation has a certain extra level of complexity.  As illustrated above, putting it in one frame is actually simpler overall if the congestion controller needs to use all the information at once to be implemented correctly.  This same logic applies to timestamps as well, in my opinion.

  1.

/Ingemar

From: Colin Perkins [mailto:[email-masked]<mailto:[email-masked]>]
Sent: den 16 augusti 2017 12:53
To: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Cc: QUIC IETF mailing list <[email-masked]<mailto:[email-masked]>>; Magnus Westerlund <[email-masked]<mailto:[email-masked]>>; Bob Briscoe ([email-masked]<mailto:[email-masked]>) <[email-masked]<mailto:[email-masked]>>; Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; Eggert, Lars ([email-masked]<mailto:[email-masked]>) <[email-masked]<mailto:[email-masked]>>; marcelo bagnulo braun <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Piers O'Hanlon <[email-masked]<mailto:[email-masked]>>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<mailto:[email-masked]>>
Subject: Re: ECN in QUIC

Hi

Cheers,
Colin

Comments are welcome
/Ingemar

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289<tel:+46%2073%20078%2032%2089>
[email-masked]<mailto:[email-masked]>
www.ericsson.com<http://www.ericsson.com>

--
Colin Perkins
https://csperkins.org/

---

From: Ian Swett <[email-masked]>
Date: Wed, 20 Sep 2017 10:08:36 -0400

I think there are good reasons to keep frames smaller and simpler in
general.  But in this particular case, I think there are strong arguments
for putting ECN in the ACK frames.  I suspect this would have come up as
soon as someone implemented ECN in QUIC.

On Wed, Sep 20, 2017 at 9:27 AM, Ingemar Johansson S <
[email-masked]> wrote:

> I must admit that I have perhaps overlooked possible combos of loss and
> ECN, the price you pay for working too much with simulators. Your arguments
> are very strong indeed.
>
>
>
> In the SCReAM congestion control (RMCAT work) the ECN and loss info comes
> in the same feedback and the handling of loss and ECN is straightforward
> there. Personally I would prefer ECN in the ACK frames and the earlier
> drafts suggested formats for that, I however got the impression that there
> is a certain resistance against additional things in the ACK frame ?.
>
>
>
> /Ingemar
>
>
>
> *From:* Ian Swett [mailto:[email-masked]]
> *Sent:* den 20 september 2017 14:41
> *Cc:* Colin Perkins <[email-masked]>; Magnus Westerlund <
> Piers O'Hanlon <[email-masked]>; QUIC IETF mailing list <
> [email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <
> [email-masked]>
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 20 Sep 2017 16:14:54 +0200

hi Ian, all,

delurking momentarily to say...

> On 20 Sep 2017, at 14:41, Ian Swett <[email-masked]> wrote:
> 
> 
> On Wed, Sep 20, 2017 at 4:39 AM, Ingemar Johansson S <[email-masked]> wrote:
>> Hi
>> 
>> 
>> 
>> 	• ECN in ACK frame or in a separate frame : Given the recent discussion on the list and fear that the ACK frame is already enough complex, it seems unwise to try to cram in additional info in the ACK frames. So my personal feeling is that we should use a separate frame type for ECN , but this frame type must be a MUST to implement and support ! Comments are very welcome.
> 

What Ian said. This is a tradeoff between complexity in frame structure (and therefore frame parsing) and complexity in handling congestion control dynamics in an environment in which we'll be receiving both loss and explicit congestion control signaling. Careful design of the frame structure and decent testing can mitigate the risk that the former will cause problems. The latter is a scary swamp filled with nightmares no matter what you do.

Cheers,

Brian

> 	• As regards to the level of detail : There are pros and cons with both alternatives (marked bytes or detailed indication). I would like to hear the opinions from others as well on this matter .
> 
> 
> /Ingemar
> 
> 
> 
> 
> On 16 Aug 2017, at 10:33, Ingemar Johansson S <[email-masked]> wrote:
> 
> 
> 
> Hi
> 
> 
>

---

From: Frank Kastenholz <[email-masked]>
Date: Wed, 20 Sep 2017 10:21:42 -0400

Hi

I wonder if the ACK can be kept simple by including only the ECN
information from the most recently received packet, or maybe N (where N is
fairly small) most recently received packets? This would reduce the amount
of information to be carried as well as the complexity of including it.

I confess that I am not familiar with the various congestion control
algorithms ... but I would imagine that more recent ECN information would
be more useful than older ECN info. No?

Frank Kastenholz

-- 
Thanks
Frank Kastenholz

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 20 Sep 2017 15:30:25 +0100

On 20/09/2017, 15:21, Frank Kastenholz wrote:
> information from the most recently received packet, or maybe N (where 
> N is fairly small) most recently received packets? This would reduce 
> the amount of information to be carried as well as the complexity of 
> including it.
>
> algorithms ... but I would imagine that more recent ECN information 
> would be more useful than older ECN info. No?
No - there are two styles of ECN CC. The more recent requires accurate 
feedback of the ECN marks - think of this more as understanding the 
"rate of marking from the bottleneck" and in some case "was this (test) 
packet marked". This requires per-packet ECN info - at least one bit per 
received packet, and preferably 2 bits.

Gorry

>
> Frank Kastenholz
>
> On Wed, Sep 20, 2017 at 9:27 AM, Ingemar Johansson S 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I must admit that I have perhaps overlooked possible combos of
>     loss and ECN, the price you pay for working too much with
>     simulators. Your arguments are very strong indeed.
>
>     In the SCReAM congestion control (RMCAT work) the ECN and loss
>     info comes in the same feedback and the handling of loss and ECN
>     is straightforward there. Personally I would prefer ECN in the ACK
>     frames and the earlier drafts suggested formats for that, I
>     however got the impression that there is a certain resistance
>     against additional things in the ACK frame ?.
>
>     /Ingemar
>
>     *From:* Ian Swett [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     *To:* Ingemar Johansson S <[email-masked]
>     <mailto:[email-masked]>>
>     *Cc:* Colin Perkins <[email-masked]
>     <mailto:[email-masked]>>; Magnus Westerlund
>     <[email-masked]
>     <mailto:[email-masked]>>; Bob Briscoe
>     ([email-masked] <mailto:[email-masked]>)
>     <[email-masked] <mailto:[email-masked]>>;
>     Praveen Balasubramanian <[email-masked]
>     <mailto:[email-masked]>>; Eggert, Lars ([email-masked]
>     <mailto:[email-masked]>) <[email-masked]
>     <mailto:[email-masked]>>; marcelo bagnulo braun
>     <[email-masked] <mailto:[email-masked]>>; Mirja Kühlewind
>     <[email-masked]
>     <mailto:[email-masked]>>; Piers O'Hanlon
>     <[email-masked] <mailto:[email-masked]>>;
>     QUIC IETF mailing list <[email-masked] <mailto:[email-masked]>>; De
>     Schepper, Koen (Nokia - BE/Antwerp)
>     <[email-masked]
>     <mailto:[email-masked]>>
>
>
>     On Wed, Sep 20, 2017 at 4:39 AM, Ingemar Johansson S
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         Hi
>
>             discussion on the list and fear that the ACK frame is
>             already enough complex, it seems unwise to try to cram in
>             additional info in the ACK frames. So my personal feeling
>             is that we should use a separate frame type for ECN , but
>             this frame type must be a MUST to implement and support !
>     I was neutral on this before, but I started thinking about
>     implementing this, and separating the frames makes it more
>     complex, and potentially really, really difficult to get the
>     implementation correct.  As an example, if the frames were
>     accidentally sent in two separate packets, I would count the
>     packet as acked, and potentially increase the congestion window
>     and other state variables.  I might even send an extra packet
>     based on this information.  Then in the next packet I'd realize
>     that packet had an ECN marking, and actually should have been
>     treated as lost from a congestion controller perspective.  In
>     order to do this correctly, I'd have to undo a bunch of things. 
>     My experience with undo is that it's terribly error prone, and I
>     would never recommend it.
>
>     Alternately, the ack could get lost, and the next packet would
>     convey ECN marking information for packets that hadn't been
>     acknowledged yet.  Which is just weird.
>
>     If we require support for ECN, then the implementation has a
>     certain extra level of complexity.  As illustrated above, putting
>     it in one frame is actually simpler overall if the congestion
>     controller needs to use all the information at once to be
>     implemented correctly.  This same logic applies to timestamps as
>     well, in my opinion.
>
>         1.
>          2. As regards to the level of detail : There are pros and
>             cons with both alternatives (marked bytes or detailed
>             indication). I would like to hear the opinions from others
>             as well on this matter .
>
>         /Ingemar
>
>         *From:* Colin Perkins [mailto:[email-masked]
>         <mailto:[email-masked]>]
>         *To:* Ingemar Johansson S <[email-masked]
>         <mailto:[email-masked]>>
>         *Cc:* QUIC IETF mailing list <[email-masked]
>         <mailto:[email-masked]>>; Magnus Westerlund
>         <[email-masked]
>         <mailto:[email-masked]>>; Bob Briscoe
>         ([email-masked] <mailto:[email-masked]>)
>         <[email-masked] <mailto:[email-masked]>>;
>         Praveen Balasubramanian <[email-masked]
>         <mailto:[email-masked]>>; Eggert, Lars ([email-masked]
>         <mailto:[email-masked]>) <[email-masked]
>         <mailto:[email-masked]>>; marcelo bagnulo braun
>         <[email-masked] <mailto:[email-masked]>>; Mirja
>         Kühlewind <[email-masked]
>         <mailto:[email-masked]>>; Piers O'Hanlon
>         <[email-masked]
>         <mailto:[email-masked]>>; De Schepper, Koen (Nokia
>         - BE/Antwerp) <[email-masked]
>         <mailto:[email-masked]>>
>             On 16 Aug 2017, at 10:33, Ingemar Johansson S
>             <[email-masked]
>             <mailto:[email-masked]>> wrote:
>
>             Hi
>
>             Finally back from vacation, and very grateful for the
>             support to continue the work to add ECN in QUIC.
>
>             1) ECN info in ACK frame or in dedicated frame : There
>             were concerns about adding extra complexity in an already
>             potentially complex ACK frame, one can have differing
>             opinions about the complexity but can understand the
>             concerns. As far as I am concerned, a separate frame type
>             for ECN is possible, possibly one need to add information
>             about the amount of not-ECT marked packets as well to keep
>             the signaling robust, this needs further investigation
>             though. One concern with a separate ECN frame is that it
>             becomes a not-implemented or optional feature, is there
>             any reason to be worried about this ?
>
>             2) More detailed ECN information : Earlier versions of the
>             ECN in QUIC draft
>             (seehttps://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt
>             <https://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt>)
>             provided with examples. We (Myself, Koen, Mirja and
>             Praveen) discussed this and we could not come up with any
>             use case where it is beneficial to know exactly how each
>             packet is ECN marked. I know that this kind of detailed
>             ECN information is suggested for the generic feedback for
>             RMCAT and I personally have a problem to see the gain with
>             the detailed ECN information also here. Input from others
>             is very welcome here.
>
>         For the RMCAT format, we wanted per-packet loss and timing
>         information, and it was as easy to feedback per-packet ECN
>         information along with it as to design something different.
>
>         A benefit of per-packet ECN marking could be to allow a
>         congestion controller that reacted differently to bursts of
>         consecutive ECN marks than it did to isolated ECN marks, given
>         the same fraction of marked packets (i.e., that reacted to ECN
>         marking events rather than ECN marking rate, like how TCP
>         responds to loss events). I don’t think we have such a thing,
>         but certainly in the context of RMCAT where we’re
>         experimenting with novel congestion control schemes for
>         traffic that has very different characteristics to traditional
>         bulk flows, it might be plausible. Per-packet marking
>         information is also useful for troubleshooting.
>
>         Certainly we need to know number of NotECT, ECT(0), ECT(1),
>         and ECN-CE marks since the last report, but I guess that’s
>         already possible.
>
>         Cheers,
>
>         Colin
>
>             a) necessary to correlate with the list of transmitted
>             packets, this increases amount of code on sender side, not
>             sure of that is a large concern as lookup is anyway needed
>             to process incoming ACKs
>
>             b) necessary to embed ECN information in ACK frame ?, at
>             least this was my conclusion when I devised the detailed
>             ECN marking info in the 01 version of the draft.
>
>             Comments are welcome
>
>             /Ingemar
>
>             SMS/MMS +46-73 078 3289 <tel:+46%2073%20078%2032%2089>
>
>             <mailto:[email-masked]>
>
>             www.ericsson.com <http://www.ericsson.com>
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 20 Sep 2017 16:57:44 +0200

> On 20 Sep 2017, at 16:30, Gorry Fairhurst <[email-masked]> wrote:
> 
>> I wonder if the ACK can be kept simple by including only the ECN information from the most recently received packet, or maybe N (where N is fairly small) most recently received packets? This would reduce the amount of information to be carried as well as the complexity of including it.
>> 
>> I confess that I am not familiar with the various congestion control algorithms ... but I would imagine that more recent ECN information would be more useful than older ECN info. No?
> No - there are two styles of ECN CC. The more recent requires accurate feedback of the ECN marks - think of this more as understanding the "rate of marking from the bottleneck" and in some case "was this (test) packet marked". This requires per-packet ECN info - at least one bit per received packet, and preferably 2 bits.

I'll also note that AccECN (draft-ietf-tcpm-accurate-ecn) would seem to be far easier to add to QUIC than to TCP. In the TCP case, it needs to reallocate a flag bit and add an option, both of which need to be defensively specified against path interference, *all* of which simply go away when the information is carried in encrypted QUIC ACK frames.

AccECN also shows the way to handle the recency of information issues raised above: the counters echoed are all the least significant N bits of a cumulative counter, which is both recent and robust against ACK loss.

Cheers,

Brian

> Gorry
> 
>> 
>> Frank Kastenholz
>> 
>> On Wed, Sep 20, 2017 at 9:27 AM, Ingemar Johansson S <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>>    order to do this correctly, I'd have to undo a bunch of things.     My experience with undo is that it's terribly error prone, and I
>>        --         Colin Perkins

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 20 Sep 2017 16:51:14 +0000

Hi
Detailed info about which packets are ECN marked can be useful but are not stricktly necessary. One alternative, that is presented in the ECN-QUIC draft is to count number of marked bytes. 
That said, some like the detailed packet-level information type better. I have not yet been convinced that that this level of detail is necessary though. 

/Ingemar

> -----Original Message-----
> From: Gorry Fairhurst [mailto:[email-masked]]
> Sent: den 20 september 2017 16:30
> To: Frank Kastenholz <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>; Magnus
> Westerlund <[email-masked]>; Bob Briscoe
> ([email-masked]) <[email-masked]>; Praveen
> Balasubramanian <[email-masked]>; Ian Swett
> <[email-masked]>; Eggert, Lars ([email-masked])
> <[email-masked]>; marcelo bagnulo braun <[email-masked]>; Mirja
> Kühlewind <[email-masked]>; Piers O'Hanlon
> <[email-masked]>; QUIC IETF mailing list <[email-masked]>; Colin
> Perkins <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp)
> <[email-masked]>
> Subject: Re: ECN in QUIC
> 
> feedback of the ECN marks - think of this more as understanding the "rate of
> marking from the bottleneck" and in some case "was this (test) packet
> marked". This requires per-packet ECN info - at least one bit per received
> packet, and preferably 2 bits.
> 
> Gorry
> 
> >
> > Frank Kastenholz
> >

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 Sep 2017 12:02:44 -0700

> That said, some like the detailed packet-level information type better. I have not yet been convinced that that this level of detail is necessary though. 

The problem is whether you can define a one size fit all ECN processing,
that if possible should be compatible with ACK compression. There are at
least two:

* RFC 3168 says, echo ECN=1 if at least one of the acked packets had ECN
set.

* draft-ietf-tcpm-dctcp-10 suggests setting ECN so that the fraction of
packet acked with ECN in 1RTT is the same as the fraction of packets
with ECN set.

So, which one?

--Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 20 Sep 2017 12:53:24 -0700

If you have an ECN bit per acked packet, like with timestamps, you don't
have to choose. That can be a sender-side decision of what to do with the
signal.

On Wed, Sep 20, 2017 at 12:02 PM, Christian Huitema <[email-masked]>
wrote:

> > Detailed info about which packets are ECN marked can be useful but are
> not stricktly necessary. One alternative, that is presented in the ECN-QUIC
> draft is to count number of marked bytes.
> > That said, some like the detailed packet-level information type better.
> I have not yet been convinced that that this level of detail is necessary
> though.
>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 Sep 2017 21:43:45 -0700

On 9/20/2017 12:53 PM, Jana Iyengar wrote:

> If you have an ECN bit per acked packet, like with timestamps, you
> don't have to choose. That can be a sender-side decision of what to do
> with the signal.
>

Yes, but if you want to do that you need the ACK to convey the list of
each acknowledged packet, with information like time stamp and ECN bit.
This is different from carrying information about the ranges of
acknowledged packet, which in the simplest form is "received up to
largest, size of range". You also have to worry about ACK compression,
ACK losses, and out of order delivery of ACK. That will be a big
departure from the current spec.

-- Christian Huitema

---

From: Colin Perkins <[email-masked]>
Date: Thu, 21 Sep 2017 14:53:42 +0100

Yes, but note that you need to echo two ECN bits per acked packet.
Colin

> On 20 Sep 2017, at 20:53, Jana Iyengar <[email-masked]> wrote:
> 
> If you have an ECN bit per acked packet, like with timestamps, you don't have to choose. That can be a sender-side decision of what to do with the signal.
> 
> On Wed, Sep 20, 2017 at 12:02 PM, Christian Huitema <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 22 Sep 2017 07:20:25 +0000

Hi
Yes, I agree with Colin and Jana.
If you have one (or two) bits per ACKed packet then there should not be any problem.
See page 5 and 6 in https://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt   for possible encodings
One can discuss if it is necessary with 2 bits, one benefit with 2 bits is that it is more versatile and gives a possibility to track possible "unlawful" remarking of the ECN bits in the network.

With that said, as mentioned earlier, one can echo the number of marked bytes as shown in section 2.3 in https://tools.ietf.org/id/draft-johansson-quic-ecn-03.txt
Of course, one lose the ability to see which packets are marked how, the benefit is that the format becomes more compact.

/Ingemar

From: Colin Perkins [mailto:[email-masked]]
Sent: den 21 september 2017 15:54
To: Jana Iyengar <[email-masked]>
Cc: Christian Huitema <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: ECN in QUIC

On 20 Sep 2017, at 20:53, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Sep 20, 2017 at 12:02 PM, Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

So, which one?

--Christian Huitema

--
Colin Perkins
https://csperkins.org/

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 25 Sep 2017 13:26:32 +0200

There is Accurate ECN (draft-ietf-tcpm-accurate-ecn) in tcpm that changes the feedback defined in RFC3168 because we believe more accurate information in needed in future.

Mirja

> Am 20.09.2017 um 21:02 schrieb Christian Huitema <[email-masked]>:
> 
> 
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 Sep 2017 12:31:12 +0100

+1

DCTP is not IETF standard - it's documenting a solution that is deployed 
in data centres - as it says - my hope is QUIC receives wider deployment.

I suggest you start from AccECN, or examine the feedback discussed in 
RMCAT - which I think strove for equivalence to this.

Gorry

On 25/09/2017, 12:26, Mirja Kühlewind wrote:
>> Am 20.09.2017 um 21:02 schrieb Christian Huitema<[email-masked]>:
>>
>>
>>

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 27 Sep 2017 09:38:36 +0000

Hi

General question. Is this Ok to discuss on this list or should we have the discussion on the tracker ?

To summarize.. I can make the following observations :

1) ECN in ACK frame or dedicated ECN frame : Proponents of a dedicated ECN frame argue that adding extra ECN info in the ACK frame can increase the complexity of the ACK frame encoding and parsing. Ian Swett however as a very strong point in  his remark that ECN in a separate frame can give additional issues in the congestion control functionality, in other words, what we save (in complexity) in one part of the chain can cause extra cost and in the worst case incomprehensible code in another part of the chain. My takeaway is that it is best to have ECN in the ACK frame.

2) Detailed ECN marking info (2 bits/packet) or ECN marked bytes : I see some preference to implement something like what is suggested for RMCAT, I am not convinced that one need to impose the limitations of AccEcn as the latter is given by the limitations by TCP. Fig 2 and 3 in  https://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt suggests two formats, my preference is the version presented in figure 3 (alt 2), it is padded at the end of the ACK frame. I copy the necessary parts below, it is not intended as a take it or leave it.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |              First Ack Block Length (8/16/32/48)            ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |  [Gap 1 (8)]  |       [Ack Block 1 Length (8/16/32/48)]     ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |  [Gap 2 (8)]  |       [Ack Block 2 Length (8/16/32/48)]     ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                  ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |  [Gap N (8)]  |       [Ack Block N Length (8/16/32/48)]     ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |ECE|ECE|... variable length, padded to full octets           ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   The ECE field requires two bits encoding per
      frame.  The additional overhead amounts to ceil(N*2/8) octets
      where the N is the sum of the ACK block lengths.

/Ingemar

> Sent: den 25 september 2017 13:31
> To: Mirja Kühlewind <[email-masked]>
> Cc: Christian Huitema <[email-masked]>; [email-masked]
> Subject: Re: ECN in QUIC
>
> +1
>
> DCTP is not IETF standard - it's documenting a solution that is deployed in
> data centres - as it says - my hope is QUIC receives wider deployment.
>
> > There is Accurate ECN (draft-ietf-tcpm-accurate-ecn) in tcpm that changes
> the feedback defined in RFC3168 because we believe more accurate
> information in needed in future.
> >
> > Mirja
> >
> >
> >> Am 20.09.2017 um 21:02 schrieb Christian
> Huitema<[email-masked]<mailto:[email-masked]>>:
> >>
> >>
> >>
> >>> That said, some like the detailed packet-level information type better. I
> have not yet been convinced that that this level of detail is necessary though.
> >> The problem is whether you can define a one size fit all ECN
> >> processing, that if possible should be compatible with ACK
> >> compression. There are at least two:
> >>
> >> * RFC 3168 says, echo ECN=1 if at least one of the acked packets had
> >> ECN set.
> >>
> >> * draft-ietf-tcpm-dctcp-10 suggests setting ECN so that the fraction
> >> of packet acked with ECN in 1RTT is the same as the fraction of
> >> packets with ECN set.
> >>
> >> So, which one?
> >>
> >> --Christian Huitema
> >>
>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 27 Sep 2017 12:18:18 -0700

> discussion on the tracker ?

We should have something on the tracker. Would you open an issue?

> 1) ECN in ACK frame or dedicated ECN frame : Proponents of a dedicated ECN
> frame argue that adding extra ECN info in the ACK frame can increase the
> complexity of the ACK frame encoding and parsing. Ian Swett however as a
> very strong point in  his remark that ECN in a separate frame can give
> additional issues in the congestion control functionality, in other words,
> what we save (in complexity) in one part of the chain can cause extra cost
> and in the worst case incomprehensible code in another part of the chain.
> My takeaway is that it is best to have ECN in the ACK frame.
>

I agree with Ian's argument -- having ECN in a separate frame would make
things harder for congestion control.

2) Detailed ECN marking info (2 bits/packet) or ECN marked bytes : I see
> some preference to implement something like what is suggested for RMCAT, I
> am not convinced that one need to impose the limitations of AccEcn as the
> latter is given by the limitations by TCP. Fig 2 and 3 in
> *https://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt*
> <https://tools.ietf.org/id/draft-johansson-quic-ecn-01.txt> suggests two
> formats, my preference is the version presented in figure 3 (alt 2), it is
> padded at the end of the ACK frame. I copy the necessary parts below, it is
> not intended as a take it or leave it.
>

 I think we'll need to think carefully about framing, but let's do that on
the issue.

- jana

> > From: Gorry Fairhurst [mailto:[email-masked]
> <[email-masked]>]
> in
> > Huitema<[email-masked]>:
> > >>
> > >>
> > >>

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 28 Sep 2017 12:36:50 +0000

The two discussion points are now posted as issue (https://github.com/quicwg/base-drafts/issues/804 ) and (https://github.com/quicwg/base-drafts/issues/805 )

/Ingemar

From: Jana Iyengar [mailto:[email-masked]]
Sent: den 27 september 2017 21:18
To: Ingemar Johansson S <[email-masked]>
Cc: [email-masked]; Mirja Kühlewind <[email-masked]>; [email-masked]; Christian Huitema <[email-masked]>
Subject: Re: ECN in QUIC

I agree with Ian's argument -- having ECN in a separate frame would make things harder for congestion control.

 I think we'll need to think carefully about framing, but let's do that on the issue.

- jana

      0                   1                   2                   3

/Ingemar

> To: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>
> Cc: Christian Huitema <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
> Subject: Re: ECN in QUIC
>
> +1
>