Subject: [tsvwg] draft-stewart-tsvwg-sctpecn

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 2 Aug 2012 23:58:26 +0000

Hi,

While looking as to how other protocol do ECN feedback signaling, I found that this draft is missing (at least not mentioning) an interesting detail.

The existence of a CWR Chunk indicates, that the feedback signal is supposed to be reliable.

However, the ECN Echo chunk defines the Number of CE marked Packets as 32-bit unsigned int, and does not specify how a receiver should behave it never received a CWR chunk...

The sheer size of the ECN Echo chunk would lend itself to be a simple feedback signal, without any explicit signal from the sender.

Historically, in RFC4960, it appears that this chunk effectively contained only a single bit (no counter; only the presence of the chunk). In order to establish a reliable feedback for at most one ECN signal per RTT, a handshake mechanism (CWR chunk) is required. That mechanism was very much alike what TCP does with ECE / CWR.

IMHO, this draft should be more clear around the mechanism:

If a handshake is required, the 32-bit counter MUST NOT roll over at 2^32-1, but remain at that value until CWR is received.

The sheer size of the counter however makes it highly unlikely that the sender will not receive a signal (before the session collapses for lack of feedback from the receiver). So a handshake-less (CWR chunk optional on the sender?) is also feasible, but then the ECN counter MUST roll over...

As it stands now, the counter rolls over, and is reset via CWR...

Are there any specific reasons for this chimera mechanism? 

Does it allow for special use cases, which cannot be addressed by either handshake or simple counter alone?

I'm very interested in the answers to the last two questions, as I am one of the authors of an Accurate ECN feedback draft for TCP, and TCPM has a milestone to define an appropriate (accurate) feedback mechanism that allows more than 1 bit per RTT... 

Best regards,

Richard Scheffenegger

> draft-stewart-tsvwg-sctpecn-02
> Partially replaces (draft-stewart-tsvwg-sctp-nonce)
> IETF-78 suggested there was interest in this topic.
> Discussed at IETF-83, Paris.
> DUE: ECN is within the TSVWG Charter,
> will call for adoption on the list.
> Please comment on list.

---

From: Randy Stewart <[email-masked]>
Date: Fri, 3 Aug 2012 11:35:22 -0400

Richard:

Well maybe am mis-understanding your comments. Are you concerned
that the counter really should not roll over?

This would require 2Billion attempts at sending an update CWR. This is a bit
far-fetched so I never thought about it ever happening.

Now the counter does not have a defined use, sort of like the duplicate receipts of TSN's
in the SACK. I was playing with it actually in a different CC algorithm and thought it 
was a good idea to have for future work since its easily obtainable.

As far as what to do when you don't get a CWR… hmm let me re-read the document. You are
supposed to (like TCP) continue to retransmit ECN Echo chunks until a CWR is received

Maybe I missed that statement in the document.

R

>> will call for adoption on the list.

-----
Randall Stewart
[email-masked]

---

From: Randy Stewart <[email-masked]>
Date: Fri, 3 Aug 2012 11:52:38 -0400

Richard:

Hmm

Re-reading the document a bit, I don't see that it does not tell
you to retransmit the chunk in the receiver side (5.3) we state:

"
After transmission of the ECN Echo chunk, usually bundled with the
   SACK, the receiver does NOT discard the ECN Echo chunk.  Instead it
   keeps the chunk in its queue and continues to send this chunk bundled
   with at least a SACK chunk on each outgoing packet, updating it as
   described above if other CE codepoint data packets arrive.  The ECN
   Echo chunk should only be discarded when a CWR Chunk arrives holding
   a TSN value that is greater than or equal to the value inside the ECN
   Echo Chunk.
"

If this is not clear enough please let me know a wording that would make it clearer.

Note that the ECN echo continues to be received until a CWR shows up
with a TSN number at least as big as the TSN sent in the ECN Echo.

R

-----
Randall Stewart
[email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 3 Aug 2012 16:17:08 +0000

Hi Randy,

My point being, that it is not clear, architecturally, if the mechanism implements a reliable feedback, or not.

The way TCP (3168) signals ECN is, that you have a counter that counts up to a specific maximum value, and remains at that value until reset. In TCP, that counter is only 1 bit (ECE), and CWR is the reset signal.

Now, when the counter is large enough (5+ bits), the resiliency no longer requires a reset signal (statistically), as the counter is increasingly unlikely to wrap over unnoticed. So with a 32 bit counter, you can do away with the CWR signal entirely. Or is there any use to the receiver to know when the sender has reacted to congestion (by an unknown extent)? (Historically, SCTP implemented the same one-bit counter/reset scheme that TCP did, IIRC; so CWR is really only relevant for legacy compatibility, correct?).

Since the receiver resets the ECN counter only if it receives a CWR with the appropriate TSN, that means that you implicitly require at least one RTT without a CE mark, for that to happen...

Assume a pretty congested network, where you'll get one CE mark per RTT (and the source doesn't react traditionally with halving the sending rate), then the counter can increase continuously. Understanding the concept (counter with reset, pure counter) would answer what the receiver is to do when the counter hits 2^32-1...

I am bringing this up because for TCP, a single bit counter is insufficient for the requirements of Conex; and I want to understand if the signaling scheme was given thorough thought that I could leverage for a redesign in the TCP space...

I do agree, that when one excludes misbehaving implementations, this question is more of academic interest rather than a true issue of a deployment following the draft, owed to the large counter.

Richard Scheffenegger

> -----Original Message-----
> From: Randy Stewart [mailto:[email-masked]]
> Sent: Freitag, 03. August 2012 08:53
> To: Scheffenegger, Richard
> Cc: [email-masked]; [email-masked]; [email-masked];
> Matthew Mathis ([email-masked]); [email-masked]
> Subject: Re: [tsvwg] draft-stewart-tsvwg-sctpecn
> 
> Richard:
> 
> Hmm
> 
> Re-reading the document a bit, I don't see that it does not tell you to
> retransmit the chunk in the receiver side (5.3) we state:
> 
> "
> Note that the ECN echo continues to be received until a CWR shows up with
> a TSN number at least as big as the TSN sent in the ECN Echo.
> 
> R
> 
> > While looking as to how other protocol do ECN feedback signaling, I
> found that this draft is missing (at least not mentioning) an interesting
> detail.
> >
> > However, the ECN Echo chunk defines the Number of CE marked Packets as
> 32-bit unsigned int, and does not specify how a receiver should behave it
> never received a CWR chunk...
> >
> > The sheer size of the ECN Echo chunk would lend itself to be a simple
> feedback signal, without any explicit signal from the sender.
> >
> > Historically, in RFC4960, it appears that this chunk effectively
> contained only a single bit (no counter; only the presence of the chunk).
> In order to establish a reliable feedback for at most one ECN signal per
> RTT, a handshake mechanism (CWR chunk) is required. That mechanism was
> very much alike what TCP does with ECE / CWR.
> >
> >
> >
> > If a handshake is required, the 32-bit counter MUST NOT roll over at
> 2^32-1, but remain at that value until CWR is received.
> >
> > The sheer size of the counter however makes it highly unlikely that the
> sender will not receive a signal (before the session collapses for lack of
> feedback from the receiver). So a handshake-less (CWR chunk optional on
> the sender?) is also feasible, but then the ECN counter MUST roll over...
> >
> >
> > I'm very interested in the answers to the last two questions, as I am
> one of the authors of an Accurate ECN feedback draft for TCP, and TCPM has
> a milestone to define an appropriate (accurate) feedback mechanism that
> allows more than 1 bit per RTT...
> >
> > Best regards,
> >
> > Richard Scheffenegger
> >
> >> DUE: ECN is within the TSVWG Charter, will call for adoption on the
> >> list.
> >> Please comment on list.
> >
> >
> 
> -----
> Randall Stewart
> [email-masked]
> 
> 
>

---

From: Randy Stewart <[email-masked]>
Date: Fri, 3 Aug 2012 13:27:31 -0400

Richard:

Ok so this is not a case where you are not seeing the ECN/CWR are reliable. Thats good ;-)

On Aug 3, 2012, at 12:17 PM, Scheffenegger, Richard wrote:

> Since the receiver resets the ECN counter only if it receives a CWR with the appropriate TSN, that means that you implicitly require at least one RTT without a CE mark, for that to happen…

Correct.

> 
> Assume a pretty congested network, where you'll get one CE mark per RTT (and the source doesn't react traditionally with halving the sending rate),
                                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

and here is the heart of the problem. If you follow the RFC and 1/2 your rate, eventually you *would* reach a state where you could get (possibly just 1) packet
though without a mark. So first and for most I see the question phrased as "If I do things wrong and violate the spec, what happens when this counter over here wraps?".

> then the counter can increase continuously. Understanding the concept (counter with reset, pure counter) would answer what the receiver is to do when the counter hits 2^32-1…

The CE counter in the ECN-Echo was never reset, this is because when the TSN actually does
cover the value then the ECN-E it is discarded so the next set of events will start
at 1 all over again with a higher TSN. This means that the ECN-E receiver, if
the sender wants to track the actual number of packets CE marked,  then when sending
a CWR it needs to note down (in the TCB) the last counter/TSN. If it receives another
ECN-E with a larger TSN and a larger counter in the next window of data, then this is a continuing episode. If
it receives a smaller counter value (nominally 1) then it can be assured the past
CWR was processed and we have entered a new congestion event. Note that the 
last counter/TSN needs to be held for one new window of data, after which
it can be discarded as well (unless of course another ECN-E arrives.. then
it has to be updated and held another window of data aka RTT).

Of course if you are not reducing your cwnd, you could stay in this for 2billion packets, but if
so then I guess you probably don't care about how many CE marks are there anyway since you
are not doing anything to slow down … ;-)

R

-----
Randall Stewart
[email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 3 Aug 2012 17:44:32 +0000

Hi Randy,

you have elegantly avoided to answer my question around the conceptual behavior :)

So let me rephrase my question a little bit differently:

Suppose you are restricted on bits available to signal the congestion back to the sender, would you implement a continuously increasing counter (that eventually wraps), or a gauge that only increases up to a certain point before requiring an explicit reset.

Independently from this, do you see value in letting the receiver (and implicitly, the network) explicitly know that the sender (supposedly) did perform a congestion control reaction?

Think of this scenario as when the send window (# packets) is larger than the largest value that the counter could hold; With TCP, that happens with windows > 1 packet; with this draft, when the window grows beyond 2^32 packets (purely hypothetical).

Thanks for your help!

Richard Scheffenegger

> Sent: Freitag, 03. August 2012 10:28
> Cc: [email-masked]; [email-masked]; [email-masked]; Matthew
> Mathis ([email-masked]); [email-masked]
> > The way TCP (3168) signals ECN is, that you have a counter that counts
> up to a specific maximum value, and remains at that value until reset. In
> TCP, that counter is only 1 bit (ECE), and CWR is the reset signal.
> >
> > Now, when the counter is large enough (5+ bits), the resiliency no
> longer requires a reset signal (statistically), as the counter is
> increasingly unlikely to wrap over unnoticed. So with a 32 bit counter,
> you can do away with the CWR signal entirely. Or is there any use to the
> receiver to know when the sender has reacted to congestion (by an unknown
> extent)? (Historically, SCTP implemented the same one-bit counter/reset
> scheme that TCP did, IIRC; so CWR is really only relevant for legacy
> compatibility, correct?).
> >
> > Since the receiver resets the ECN counter only if it receives a CWR
> > with the appropriate TSN, that means that you implicitly require at
> > least one RTT without a CE mark, for that to happen...
> 
> Correct.
> 
> >
> > Assume a pretty congested network, where you'll get one CE mark per
> > RTT (and the source doesn't react traditionally with halving the
> > sending rate),
> 
> and here is the heart of the problem. If you follow the RFC and 1/2 your
> rate, eventually you *would* reach a state where you could get (possibly
> just 1) packet though without a mark. So first and for most I see the
> question phrased as "If I do things wrong and violate the spec, what
> happens when this counter over here wraps?".
> 
> 
> > then the counter can increase continuously. Understanding the concept
> > (counter with reset, pure counter) would answer what the receiver is
> > to do when the counter hits 2^32-1...
> 
> The CE counter in the ECN-Echo was never reset, this is because when the
> TSN actually does cover the value then the ECN-E it is discarded so the
> next set of events will start at 1 all over again with a higher TSN. This
> means that the ECN-E receiver, if the sender wants to track the actual
> number of packets CE marked,  then when sending a CWR it needs to note
> down (in the TCB) the last counter/TSN. If it receives another ECN-E with
> a larger TSN and a larger counter in the next window of data, then this is
> a continuing episode. If it receives a smaller counter value (nominally 1)
> then it can be assured the past CWR was processed and we have entered a
> new congestion event. Note that the last counter/TSN needs to be held for
> one new window of data, after which it can be discarded as well (unless of
> course another ECN-E arrives.. then it has to be updated and held another
> window of data aka RTT).
> 
> Of course if you are not reducing your cwnd, you could stay in this for
> 2billion packets, but if so then I guess you probably don't care about how
> many CE marks are there anyway since you are not doing anything to slow
> down ... ;-)
> 
> R
> 
> >
> > I am bringing this up because for TCP, a single bit counter is
> insufficient for the requirements of Conex; and I want to understand if
> the signaling scheme was given thorough thought that I could leverage for
> a redesign in the TCP space...
> >
> >
> > I do agree, that when one excludes misbehaving implementations, this
> question is more of academic interest rather than a true issue of a
> deployment following the draft, owed to the large counter.
> >
> > Richard Scheffenegger
> >
> >> Cc: [email-masked]; [email-masked];
> >> [email-masked]; Matthew Mathis ([email-masked]);
> >> [email-masked]
> >> Re-reading the document a bit, I don't see that it does not tell you
> >> to retransmit the chunk in the receiver side (5.3) we state:
> >>
> >> "
> >> found that this draft is missing (at least not mentioning) an
> >> interesting detail.
> >>>
> >> 32-bit unsigned int, and does not specify how a receiver should
> >> behave it never received a CWR chunk...
> >>>
> >> In order to establish a reliable feedback for at most one ECN signal
> >> per RTT, a handshake mechanism (CWR chunk) is required. That
> >> mechanism was very much alike what TCP does with ECE / CWR.
> >>>
> >>>
> >>>
> >> sender will not receive a signal (before the session collapses for
> >> lack of feedback from the receiver). So a handshake-less (CWR chunk
> >> optional on the sender?) is also feasible, but then the ECN counter
> MUST roll over...
> >>>
> >>>
> >>> Does it allow for special use cases, which cannot be addressed by
> >>> either
> >> handshake or simple counter alone?
> >>>
> >> one of the authors of an Accurate ECN feedback draft for TCP, and
> >> TCPM has a milestone to define an appropriate (accurate) feedback
> >> mechanism that allows more than 1 bit per RTT...
> >>>
> >>> Best regards,
> >>>
> >>> Richard Scheffenegger
> >>>

---

From: John Leslie <[email-masked]>
Date: Fri, 3 Aug 2012 21:03:03 -0400

Scheffenegger, Richard <[email-masked]> wrote:
> 
> Suppose you are restricted on bits available to signal the congestion
> back to the sender, would you implement a continuously increasing
> counter (that eventually wraps), or a gauge that only increases up to
> a certain point before requiring an explicit reset.

   [tcpm] is facing this issue for TCP: I haven't posted an opinion
there yet; but frankly my opinion is that a counter that wraps is a
better mechanism. It end-runs a lot of race conditions.

   I don't know the exact details for SCTP -- but it seems avoiding
race conditions would be best here, too.

> Independently from this, do you see value in letting the receiver
> (and implicitly, the network) explicitly know that the sender
> (supposedly) did perform a congestion control reaction?

   There is arguably some value in that; but it tends to strike me as
a MAY rather than a MUST.

> Think of this scenario as when the send window (# packets) is larger
> than the largest value that the counter could hold; With TCP, that
> happens with windows > 1 packet; with this draft, when the window
> grows beyond 2^32 packets (purely hypothetical).

   TCP, here, serves only as a bad example IMHO. The one-bit counter
was barely adequate when first specified, and can never pass more than
one event per RTT. This is already considered inadequate for things we
want TCP to do today.

   IMHO, if there is a CWR indication at all, it deserves to be the
value of the ECE counter when the window was reduced -- thus as many
bits as the ECE counter. Again, this end-runs race conditions.

   (The mechanism is analogous to the circular-buffer algorithms from
the 1960s when we dealt with shared-memory communication between
processes that could interrupt each other.)

--
John Leslie <[email-masked]>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 4 Aug 2012 03:07:35 +0000

Thanks John,

I think I understand now what you are saying - at least to my understanding, you weren't as explicit over in the TCPM WG list :)

Let me discuss these considerations with my co-author, perhaps we can come up with a proper way to encode all these signals appropriately.

Any thoughts as to the exact sequence of CE marks being seen on the receiver side? (I think that this will be too much information, better served by some kind of explicit byte-congestion counter, that has similar properties to a packet-congestion counter).

Best regards,

Richard Scheffenegger

> From: John Leslie [mailto:[email-masked]]
> Sent: Freitag, 03. August 2012 18:03
> Cc: Randy Stewart; [email-masked]