Subject: RE: about draft-johansson-quic-ecn-00

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 1 Feb 2017 13:24:56 +0000

Hi Gorry and Marcelo + others

Thanks for the review and comments. I understand that the negotiation phase is the more problematic aspect.

The proposed ECN frame sounds like a good idea as this makes it possible to do the ECN negotiation without delaying the connection setup (in case ECN fails). The possible drawback is that "non-ECN capable" connection setup packets may be discarded over a congested ECN capable path and that may delay the connection setup too.
So if I understand the ECN frame correctly. It would be of a given size (~MTU?) and have the ECN bits set (=CE?). 
The receiver can reply in two ways.
1) Using the ECN echo in the ACK frame, assumes that the ECN frame has a sequence number
2) Using an ECN frame in the reverse direction (the ECN frame can then indicate the ECN bit)

More comments inline below

/Ingemar

> -----Original Message-----
> From: Gorry Fairhurst [mailto:[email-masked]]
> Sent: den 31 januari 2017 19:56
> To: marcelo bagnulo braun <[email-masked]>
> Cc: [email-masked]; Ingemar Johansson S
> <[email-masked]>
> Subject: Re: about draft-johansson-quic-ecn-00
> 
> I'd also like to thank you for this consideration of where ECN fits within QUIC.
> I have a few comments, see below:
> 
> I really do see merit in requiring implementation of the receiver functionality.
> This to me is independent of a decision whether to use the ECN functionality
> at the sender. I’d support it being mandatory to implement *at least* basic
> ECN receiver support (which I think is similar to what marcelo said).
> 
> draft-ietf-tsvwg-rfc5405bis (now in RFC-ED queue), provides a checklist for
> UDP support for ECN, the current list in the draft seems similar, but is not
> quite the same. I think since rfc5405bis is scheduled for BCP, it would be good
> to cite this and align where possible.
[IJ] Totally missed this, sorry for the ignorance, should definitely be referenced.

> 
> The decision whether a sender chooses to mark using ECT(0) or ECT(1) or
> neither needs to be based on understanding of whether the combined send
> and receive stacks and network devices currently provide support (or more
> likely measurement), monitoring as described in 2.6 (i.e., this is a per-path
> run-time decision). One thing a QUIC  receiver may already know is whether
> a receive stack has support for ECN.
[IJ] Yes, it should be possible for a stack to send packets over the local interface (127.0.0.1) and check the status of the ECN bits.

> 
> I suggest this places requirements on the feedback method:
> 
> A sender may need to understand how the network forwards ECT(x) marks
> on the forward path - (Alt 3 shows some functionality for this).  this can
> confirm whether ECT(x) marks are received. I expect this will be needed to
> know that ECT and ECN-CE marks are actually being sent across the entire
> path, and are not cleared at some midpoint, erasing the congestion
> information.  I think it is also would be useful to know that
> ECT(1) is actually traversing a path when using new ECT(1) semantics as
> described in the tsvwg draft (aka like L4S).
> 
> I can see real advantages if  alt-1, alt-2 at least provides a count to indicate
> the number of packets (bytes?) received with not-ECT; ECT(0); and ECT(1).
[IJ] I would say that you will get the same information with all the three methods,  in the case of alt 1 and 2 the burden to compute the number of not-ECT, ECT(0) and ECT(1) is on the sender side, while with alt 3 the burden is on the receiver side (but the cost is admittedly quite low). 

> 
> Knowing receive code point information would also provide opportunities to
> detect and react appropriately to misbehaving receivers/paths (including
> sender control to set the mark value) - although at least while the
> path/receiver is being verified the sender will likely also be needed to know
> *which* packets carried a specific mark. (as noted alt-3 may not provide
> this). How we should do this, I am not sure - various ways exist - but
> designing a transport that can allow this should be possible.
[IJ] Yes, alt 3, does not allow to determine which packets have which ECN mark, it is unclear though to me how important this is.

> 
> Gorry
> 
> 
> 
> On 26/01/2017 10:40, marcelo bagnulo braun wrote:
> > Hi,
> >
> > I have read the draft and i think it is a good starting point for this
> > issue. thanks for writing it.
> >
> > My main comment is about the ECN negotiation. I am unconvinced that
> > the quic version negotiation is the right approach for this.
> >
> > I mean, afaiu, quic version negotiation imposes an extra RTT when the
> > new version is not supported. I think this is ok for major version
> > changes, but I dont think this is appropriate for feature negotiation
> > such as ECN.
> >
> > I mean, unless ECN is supported by default in the first quic version,
> > I think using quic version negotiation for  ECN support would be an
> > obstacle in ECN adoption in quic. I mean, if we end up in the
> > situation that the first (widely deployed) quic protocol specification
> > does not support ECN by default, then having ECN negotiation using
> > quic version will imply that if a client wants to tries to see if the
> > server supports ECN, it risks to have a extra RTT penalty. (i
> > understand the client can cache which servers support ECN, but we are
> > back into working around these difficulties that make ecn adoption
> > harder).
[IJ] Quite convincing arguments I must say.

> >
> > The other problem is related to the blocking of ECN bits in the IP
> > header. Ideally, I guess, the client should be able to learn if the
> > ECN bits are blocked during the negotiation phase i.e. the negotiation
> > can fail either because the server doesnt support it or because the
> > network doesnt support it. Because of this, I would argue that the
> > good approach is to have the ECN bits set in the IP header of the
> > packet carrying the ECN negotiation.
[IJ] OK

> >
> > So, with all this, I think it may be a better approach to make ECN
> > negotiation to be done using a new frame type that is sent in the
> > stream 1. This would be a message that the sender issues, after the
> > connection has been established, to request ECN support from the other
> > endpoint. This message would be sent in an ECN marked packet, to
> > verify that the network delivers the message. The other endpoint can
> > reply with another message accepting of not the ECN support for the
> > connection.
[IJ] Yes, another message or to simply indicate it in the ACK frame.

> >
> > I think this approach has the benefit of not imposing extra latency
> > for those who want to check if ECN is supported and at the same time
> > verifies that the network is able to deliver ECN marked packets. If
> > any of these conditions fail, the negotiation fail. Moreover, i think
> > these messages can be used to disable ECN in the middle of the
> > connection. One case where this can be useful is in a mobility
> > scenario, where one of the endpoints move and in order to check if the
> > new path still supports ECN, they can resend the option and see if the
> > packets still make it. I guess more complicated verification methods
> > could be built into this.
> >
> > thoughts?
[IJ] Sounds quite reasonable, I a bit uncertain if it should be an ECN frame that is acknowledged by an ECN frame or if the ACK frame with an ECN field is sufficient.  

> > regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 1 Feb 2017 14:58:15 +0100

inline...

> The proposed ECN frame sounds like a good idea as this makes it possible to do the ECN negotiation without delaying the connection setup (in case ECN fails). The possible drawback is that "non-ECN…
I dont think we should prevent the quic handshake packets to be ECT marked.
(funny enough, if i understand correctly stream 1 packets are not 
constrained by congestion control, so there wouldnt be needed to feed 
the CE back, since the sender is not supposed to react to this, if i 
understand this correctly)

> So if I understand the ECN frame correctly. It would be of a given size (~MTU?) and have the ECN bits set (=CE?).

i am not sure why it should carry more than one byte for the frame type 
and an extra byte for signalling
The packet carrying this one would contain the ECT codepoint, i would say

> 2) Using an ECN frame in the reverse direction (the ECN frame can then indicate the ECN bit)

but there is a difference between having received the ECN frame 
correctly and the server actually accpeting the use of ECN in the 
connection. that is why i think we need to use one of the bits in the 
ECN flag to confirm that the receiver wants to use ECN for this 
connection, no?
regards, marcelo

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 1 Feb 2017 14:07:15 +0000

Hi

Inline
/I

> From: marcelo bagnulo braun [mailto:[email-masked]]
> Sent: den 1 februari 2017 14:58
> To: Ingemar Johansson S <[email-masked]>;
> [email-masked]
> Cc: [email-masked]
> > The proposed ECN frame sounds like a good idea as this makes it possible
> to do the ECN negotiation without delaying the connection setup (in case
> ECN fails). The possible drawback is that "non-ECN capable" connection setup
> packets may be discarded over a congested ECN capable path and that may
> delay the connection setup too.
> (funny enough, if i understand correctly stream 1 packets are not constrained
> by congestion control, so there wouldnt be needed to feed the CE back,
> since the sender is not supposed to react to this, if i understand this
> correctly)
> 
> > So if I understand the ECN frame correctly. It would be of a given size
> (~MTU?) and have the ECN bits set (=CE?).
> 
> i am not sure why it should carry more than one byte for the frame type and
> an extra byte for signalling The packet carrying this one would contain the
> ECT codepoint, i would say

[IJ] Yes, it should be sufficient with a small ECN frame, a thought that struck me is that if there is some magic differentiation between small and large packets in the network, but perhaps this is a long shot...

> but there is a difference between having received the ECN frame correctly
> and the server actually accpeting the use of ECN in the connection. that is
> why i think we need to use one of the bits in the ECN flag to confirm that the
> receiver wants to use ECN for this connection, no?
> regards, marcelo
[IJ] Yes, I assume that you mean a possible ECN flag in the ACK frame ?. Alternatively an ECN frame can be used in the reverse direction .

> >> This to me is independent of a decision whether to use the ECN
> >> functionality at the sender. I’d support it being mandatory to
> >> implement *at least* basic ECN receiver support (which I think is similar to
> what marcelo said).
> >>
> >> draft-ietf-tsvwg-rfc5405bis (now in RFC-ED queue), provides a
> >> checklist for UDP support for ECN, the current list in the draft
> >> seems similar, but is not quite the same. I think since rfc5405bis is
> >> scheduled for BCP, it would be good to cite this and align where possible.
> >> The decision whether a sender chooses to mark using ECT(0) or ECT(1)
> >> or neither needs to be based on understanding of whether the combined
> >> send and receive stacks and network devices currently provide support
> >> (or more likely measurement), monitoring as described in 2.6 (i.e.,
> >> this is a per-path run-time decision). One thing a QUIC  receiver may
> >> already know is whether a receive stack has support for ECN.
> > [IJ] Yes, it should be possible for a stack to send packets over the local
> interface (127.0.0.1) and check the status of the ECN bits.
> >
> >
> >> on the forward path - (Alt 3 shows some functionality for this).
> >> this can confirm whether ECT(x) marks are received. I expect this
> >> will be needed to know that ECT and ECN-CE marks are actually being
> >> sent across the entire path, and are not cleared at some midpoint,
> >> erasing the congestion information.  I think it is also would be
> >> useful to know that
> >> ECT(1) is actually traversing a path when using new ECT(1) semantics
> >> as described in the tsvwg draft (aka like L4S).
> >>
> >> I can see real advantages if  alt-1, alt-2 at least provides a count
> >> to indicate the number of packets (bytes?) received with not-ECT; ECT(0);
> and ECT(1).
> > [IJ] I would say that you will get the same information with all the three
> methods,  in the case of alt 1 and 2 the burden to compute the number of
> not-ECT, ECT(0) and ECT(1) is on the sender side, while with alt 3 the burden
> is on the receiver side (but the cost is admittedly quite low).
> >
> >> Knowing receive code point information would also provide
> >> opportunities to detect and react appropriately to misbehaving
> >> receivers/paths (including sender control to set the mark value) -
> >> although at least while the path/receiver is being verified the
> >> sender will likely also be needed to know
> >> *which* packets carried a specific mark. (as noted alt-3 may not
> >> provide this). How we should do this, I am not sure - various ways
> >> exist - but designing a transport that can allow this should be possible.
> > [IJ] Yes, alt 3, does not allow to determine which packets have which ECN
> mark, it is unclear though to me how important this is.
> >
> >> Gorry
> >>
> >>
> >>
> >>> I have read the draft and i think it is a good starting point for
> >>> this issue. thanks for writing it.
> >>>
> >>> I mean, afaiu, quic version negotiation imposes an extra RTT when
> >>> the new version is not supported. I think this is ok for major
> >>> version changes, but I dont think this is appropriate for feature
> >>> negotiation such as ECN.
> >>>
> >>> I mean, unless ECN is supported by default in the first quic
> >>> version, I think using quic version negotiation for  ECN support
> >>> would be an obstacle in ECN adoption in quic. I mean, if we end up
> >>> in the situation that the first (widely deployed) quic protocol
> >>> specification does not support ECN by default, then having ECN
> >>> negotiation using quic version will imply that if a client wants to
> >>> tries to see if the server supports ECN, it risks to have a extra
> >>> RTT penalty. (i understand the client can cache which servers
> >>> support ECN, but we are back into working around these difficulties
> >>> that make ecn adoption harder).
> >>> ECN bits are blocked during the negotiation phase i.e. the
> >>> negotiation can fail either because the server doesnt support it or
> >>> because the network doesnt support it. Because of this, I would
> >>> argue that the good approach is to have the ECN bits set in the IP
> >>> header of the packet carrying the ECN negotiation.
> > [IJ] OK
> >
> >>> connection has been established, to request ECN support from the
> >>> other endpoint. This message would be sent in an ECN marked packet,
> >>> to verify that the network delivers the message. The other endpoint
> >>> can reply with another message accepting of not the ECN support for
> >>> the connection.
> >>> any of these conditions fail, the negotiation fail. Moreover, i
> >>> think these messages can be used to disable ECN in the middle of the
> >>> scenario, where one of the endpoints move and in order to check if
> >>> the new path still supports ECN, they can resend the option and see
> >>> if the packets still make it. I guess more complicated verification
> >>> methods could be built into this.
> >>>
> >>> thoughts?
> > [IJ] Sounds quite reasonable, I a bit uncertain if it should be an ECN frame
> that is acknowledged by an ECN frame or if the ACK frame with an ECN field is
> sufficient.
> >
> >>> regards, marcelo
> >>>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 1 Feb 2017 17:16:53 +0100

> [IJ] Yes, I assume that you mean a possible ECN flag in the ACK frame ?. Alternatively an ECN frame can be used in the reverse direction .

I would keep the ECN separated, so, i mean to also use the ECN frame in 
the reverse direction

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 2 Feb 2017 09:38:34 +0000

Hi
Yes, added a section on monitoring in the draft, currently it is more or less just a stub. Not sure if there is any QUIC work item that covers monitoring  BTW. 
A more recent ECN treatment paper is https://csperkins.org/publications/2015/10/mcquistin2015ecn-udp.pdf 

/Ingemar

> Sent: den 2 februari 2017 10:15
> agree
> Other information, I was reading Steve Bauer's paper on measuring ECN (see
> link below) and they seem to claim that they dont see much difference in
> treatment of ECN marked packets by the network whether they are tcp or
> udp, which i guess it is good, but maybe it would be worthwhile to measure
> it?
> 
> 
> (https://www.akamai.com/cn/zh/multimedia/documents/technical-
> publication/measuring-the-state-of-ecn-readiness-in-servers-clients-and-
> routers-technical-publication.pdf)
> 
> El 02/02/17 a las 09:32, Ingemar Johansson S escribió:
> > OK, thanks, it is perhaps the most clean cut option to use the ECN frame in
> both directions.
> > /I
> >
> >> Sent: den 1 februari 2017 17:17
> >>> [IJ] Yes, I assume that you mean a possible ECN flag in the ACK frame ?.
> >> Alternatively an ECN frame can be used in the reverse direction .
> >>
> >> I would keep the ECN separated, so, i mean to also use the ECN frame
> >> in the reverse direction
> >>
> >>>>>> I'd also like to thank you for this consideration of where ECN
> >>>>>> fits within
> >>>> QUIC.
> >>>>>> seems similar, but is not quite the same. I think since
> >>>>>> rfc5405bis is scheduled for BCP, it would be good to cite this
> >>>>>> and align where
> >> possible.
> >>>>> [IJ] Totally missed this, sorry for the ignorance, should
> >>>>> definitely be
> >>>> referenced.
> >>>>>> The decision whether a sender chooses to mark using ECT(0) or
> >>>>>> ECT(1) or neither needs to be based on understanding of whether
> >>>>>> the combined send and receive stacks and network devices
> >>>>>> currently provide support (or more likely measurement),
> >>>>>> monitoring as described in 2.6 (i.e., this is a per-path run-time
> >>>>>> decision). One thing a QUIC  receiver may already know is whether
> >>>>>> a receive stack has
> >> support for ECN.
> >>>>>> will be needed to know that ECT and ECN-CE marks are actually
> >>>>>> being sent across the entire path, and are not cleared at some
> >>>>>> midpoint, erasing the congestion information.  I think it is also
> >>>>>> would be useful to know that
> >>>>>> ECT(1) is actually traversing a path when using new ECT(1)
> >>>>>> semantics as described in the tsvwg draft (aka like L4S).
> >>>>>>
> >>>>>> I can see real advantages if  alt-1, alt-2 at least provides a
> >>>>>> count to indicate the number of packets (bytes?) received with
> >>>>>> not-ECT; ECT(0);
> >>>> and ECT(1).
> >>>> methods,  in the case of alt 1 and 2 the burden to compute the
> >>>> number of not-ECT, ECT(0) and ECT(1) is on the sender side, while
> >>>> with alt 3 the burden is on the receiver side (but the cost is admittedly
> quite low).
> >>>>>> receivers/paths (including sender control to set the mark value)
> >>>>>> - although at least while the path/receiver is being verified the
> >>>>>> provide this). How we should do this, I am not sure - various
> >>>>>> ways exist - but designing a transport that can allow this should be
> possible.
> >>>>>>> I have read the draft and i think it is a good starting point
> >>>>>>> for this issue. thanks for writing it.
> >>>>>>>
> >>>>>>> My main comment is about the ECN negotiation. I am unconvinced
> >>>>>>> that the quic version negotiation is the right approach for this.
> >>>>>>>
> >>>>>>> I mean, afaiu, quic version negotiation imposes an extra RTT
> >>>>>>> when the new version is not supported. I think this is ok for
> >>>>>>> major version changes, but I dont think this is appropriate for
> >>>>>>> feature negotiation such as ECN.
> >>>>>>>
> >>>>>>> would be an obstacle in ECN adoption in quic. I mean, if we end
> >>>>>>> up in the situation that the first (widely deployed) quic
> >>>>>>> protocol specification does not support ECN by default, then
> >>>>>>> having ECN negotiation using quic version will imply that if a
> >>>>>>> client wants to tries to see if the server supports ECN, it
> >>>>>>> risks to have a extra RTT penalty. (i understand the client can
> >>>>>>> cache which servers support ECN, but we are back into working
> >>>>>>> around these difficulties that make ecn adoption harder).
> >>>>>>> The other problem is related to the blocking of ECN bits in the
> >>>>>>> IP header. Ideally, I guess, the client should be able to learn
> >>>>>>> if the ECN bits are blocked during the negotiation phase i.e.
> >>>>>>> the negotiation can fail either because the server doesnt
> >>>>>>> support it or because the network doesnt support it. Because of
> >>>>>>> this, I would argue that the good approach is to have the ECN
> >>>>>>> bits set in the IP header of the packet carrying the ECN negotiation.
> >>>>> [IJ] OK
> >>>>>
> >>>>>>> So, with all this, I think it may be a better approach to make
> >>>>>>> ECN negotiation to be done using a new frame type that is sent
> >>>>>>> in the stream 1. This would be a message that the sender issues,
> >>>>>>> after the connection has been established, to request ECN
> >>>>>>> support from the other endpoint. This message would be sent in
> >>>>>>> an ECN marked packet, to verify that the network delivers the
> >>>>>>> message. The other endpoint can reply with another message
> >>>>>>> accepting of not the ECN support for the connection.
> >>>>>>> I think this approach has the benefit of not imposing extra
> >>>>>>> latency for those who want to check if ECN is supported and at
> >>>>>>> the same time verifies that the network is able to deliver ECN
> >>>>>>> marked packets. If any of these conditions fail, the negotiation fail.
> >>>>>>> Moreover, i think these messages can be used to disable ECN in
> >>>>>>> the middle of the connection. One case where this can be useful
> >>>>>>> is in a mobility scenario, where one of the endpoints move and
> >>>>>>> in order to check if the new path still supports ECN, they can
> >>>>>>> resend the option and see if the packets still make it. I guess
> >>>>>>> more complicated verification methods could be built into this.
> >>>>>>>
> >>>>>>> thoughts?
> >>>> that is acknowledged by an ECN frame or if the ACK frame with an
> >>>> ECN field is sufficient.
> >>>>>>> regards, marcelo
> >>>>>>>