Subject: [re-ECN] Revised agenda theory

From: Leslie Daigle <[email-masked]>
Date: Fri, 23 Oct 2009 23:03:55 -0400

Hi,

We need to get an agenda posted.  Subject to suggested changes, and/or 
absent howls of dissent, we'll post the following. Note that I have 
expanded it to allow more discussion of the specific viability of the 
approach.

We'll work on framing up some discussion for that part of the agenda 
during the next week, or so.

Leslie.

Congestion Exposure (ConEx) is a proposed new IETF activity to enable
congestion to be exposed along the forwarding path of the Internet. By
revealing expected congestion in the IP header of every packet,
congestion exposure provides a generic network capability which allows
greater freedom over how capacity is shared. Such information could be
used for many purposes, including congestion policing, accountability
and inter-domain SLAs. It may also open new approaches to QoS and
traffic engineering.

The purpose of the BoF is to explore the support for and viability of
pursuing an IETF activity to define a basic protocol to expose the
expected rest-of-path congestion in the IP header. Any such protocol
should work with minimal changes to the existing network, in particular
it should work with unmodified routers. There is already one existing
proposal that builds on ECN to provide rest-of-path congestion
information in every IP header and other proposals may come forward.

More detail is available at:
http://trac.tools.ietf.org/area/tsv/trac/wiki/re-ECN

BoF Co-Chairs:

Leslie Daigle <[email-masked]>
Philip Eardley <[email-masked]>

Agenda

  5 mins  administrivia
  5 mins  introduction by chairs
40 mins  the problem
             context/motivation [Rich Woundy]
             technical problem [Mark Handley]
10 mins  constraints  [Philip Eardley]
30 mins  towards a solution
             overview of re-ECN   [Bob Briscoe]
40 mins  discussion of viability
20 mins  draft charter discussion
10 mins  questions and hums

N.B.:  This assumes our current 160min agenda space (check my math!). If 
the BoF moves, we'll have to re-think.

-- 

-------------------------------------------------------------------
"Reality:
      Yours to discover."
                                 -- ThinkingCat
Leslie Daigle
[email-masked]
-------------------------------------------------------------------

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 25 Oct 2009 00:27:38 +0100

Leslie,

This looks good, but see below about the viability item, wherein lie 
monsters. Also, a couple of thoughts before that:

1/ I've been thinking... We should add an item to the many purposes list:
- evolution path beyond TCP (running out of dynamic range)

2/ I think it would be cool to have a brief session on the community 
around this; what people are doing in this space, why, etc. This 
could be structured:
- either as one of the chairs reporting all this (requiring people to 
have told you in advance what they're doing - plus scraping the list 
archive for what people said when they introduced their interest recently),
- or as a vox pop.

It would also be a chance to report on some of the activity that has 
been going on to ensure the commercial & public policy community 
would be happy with such a change to the Internet (e.g. the GIIC 
thing you went to, and any ISOC activities you're planning).

3/ On the "discussion of viability" it occurs to me that the BoF 
can't really discuss how viable it might be to deploy re-ECN without 
also considering the viability of alternative approaches to solve 
each of the problems we claim to be able to solve (or even whether 
alternatives exist).

How otherwise are we going to
- side-step the net neutrality problem?
- simplify inter-domain e2e QoS?
- move beyond TCP which is running out of dynamic range?
- and so on?

Taking evolution beyond TCP as an example...

In the IRTF ICCRG, Matt Mathis has proposed that moving to 1/p 
congestion controls (rather than 1/sqrt(p) like TCP) would allow 
congestion control to scale indefinitely, by maintaining a high 
congestion information rate as flow rates & link rates scale. Whereas 
if we stick with TCP-friendly, the information rate per window is 
already becoming stupidly low (hours between loss signals) as rates 
increase over the next few years. To put it bluntly, as carriers 
deploy more capacity, we'll be held back by the e2e protocols, which 
aren't able to use it.

Widespread re-ECN deployment would allow us to relax the TCP-friendly 
requirement, so we can bless 1/p controls and shift onto a sounder 
evolution path.

The alternative seems to be changing every router (and every LSR and 
every ethernet switch) in the Internet to do RCP (which still aims 
for flow-rate-fairness so it still drives ISPs into net neutrality 
issues, yada yada).

____
Do you see my point about trying to talk about the viability of 
re-ECN in isolation from wider debate on the viability of the 
alternatives? But such a session would require a lot of cluefulness 
in the audience to understand it. You might recall the p2pi workshop, 
which covered just a small part of this space. It was a selected set 
of participants, but even then the cluefulness factor from each 
specialist about a each related area was painfully low.

I'm not saying we shouldn't try to talk about viability, but it will 
be a complicated session to do justice to the debate.

Bob

At 04:03 24/10/2009, Leslie Daigle wrote:

>Hi,
>
>We need to get an agenda posted.  Subject to suggested changes, 
>and/or absent howls of dissent, we'll post the following. Note that 
>I have expanded it to allow more discussion of the specific 
>viability of the approach.
>
>N.B.:  This assumes our current 160min agenda space (check my 
>math!). If the BoF moves, we'll have to re-think.
>
>
>--
>
>_______________________________________________
>re-ECN mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/re-ecn

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Leslie Daigle <[email-masked]>
Date: Sun, 25 Oct 2009 18:06:19 -0400

Hi Bob,

I do take your point about viability (of mechanisms for congestion 
exposure, btw, not re-ECN specifically) being a challenge, but I also 
don't think we'll get to discussing chartering a working group if we 
don't take a whack at it.  "Focus" will be the watch-word.

I'm happy to discuss how to make sure that discussion _is_ focused.  I 
actually think your smiley-table (mail to Fred) is a reasonable starting 
point for at least some aspect of illustrating this.

I should also say -- I don't think the BoF can determine that congestion 
exposure is The One True Way to better Internet health.  I'm not even 
particularly convinced a WG can do that.  I'm hoping the bar is set at 
demonstrating that it is a useful and important step to pursue, even as 
others are pursue elsewhere.

A few more points, in-line:

> - evolution path beyond TCP (running out of dynamic range)

Which is pretty cool, but on the bof agenda might lead to some ratholing 
on whether we're just bashing TCP, no?

> around this; what people are doing in this space, why, etc. This could 
> be structured:
> been going on to ensure the commercial & public policy community would 
> be happy with such a change to the Internet (e.g. the GIIC thing you 
> went to, and any ISOC activities you're planning).

I'm happy to have _a_ slide pointing out there's a lot of stuff going on 
in the world, because, as you note, it does communicate that this is 
getting broad airtime in the world at large.  But for the purpose of the 
BoF, I think the major emphasis should be on making sure the people in 
the room have enough information to determine whether they think there 
is an IETF activity here.

The latter is the theory behind "the problem" item on the agenda.  If 
there's a better way to handle that piece of it, happy to take sugestions.

And, I'll be sure to announce the ISOC activity here, when it's, errr, 
announced. :)

> 
> 3/ On the "discussion of viability" it occurs to me that the BoF can't 
> really discuss how viable it might be to deploy re-ECN without also 
> considering the viability of alternative approaches to solve each of the 
> problems we claim to be able to solve (or even whether alternatives exist).
> 
> congestion information rate as flow rates & link rates scale. Whereas if 
> we stick with TCP-friendly, the information rate per window is already 
> becoming stupidly low (hours between loss signals) as rates increase 
> over the next few years. To put it bluntly, as carriers deploy more 
> capacity, we'll be held back by the e2e protocols, which aren't able to 
> use it.
> 
> every ethernet switch) in the Internet to do RCP (which still aims for 
> flow-rate-fairness so it still drives ISPs into net neutrality issues, 
> yada yada).
> 
> ____
> Do you see my point about trying to talk about the viability of re-ECN 
> in isolation from wider debate on the viability of the alternatives? But 
> such a session would require a lot of cluefulness in the audience to 
> understand it. You might recall the p2pi workshop, which covered just a 
> small part of this space. It was a selected set of participants, but 
> even then the cluefulness factor from each specialist about a each 
> related area was painfully low.
> 
> I'm not saying we shouldn't try to talk about viability, but it will be 
> a complicated session to do justice to the debate.

And, the sad reality is, we will not do it justice.  What we need to do 
is hit the high points to illustrate that there is viability here, and 
it's worth chartering an IETF activity to delve into it in more depth.

One refinement to the agenda text occurs to me:

CONEX
   5 mins  administrivia
Background
Discussion of potential IETF work
  40 mins  discussion of viability of congestion exposure

Leslie.

>> N.B.:  This assumes our current 160min agenda space (check my math!). 
>> If the BoF moves, we'll have to re-think.
>>
>>
>> -- 
>>

--

---

From: <[email-masked]>
Date: Mon, 26 Oct 2009 10:30:20 -0000

Some inline comments

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Leslie Daigle
> Sent: 25 October 2009 22:06
> To: Briscoe,RJ,Bob,XVR9 BRISCORJ R
> Cc: Matt MATHIS; [email-masked]
> Subject: Re: [re-ECN] Revised agenda theory
> 
> 
> Hi Bob,
> 

It may or may not be the One True Way, but as you say that is well
beyond the scope of a BoF or WG (maybe even beyond the IAB...)

> on whether we're just bashing TCP, no?

Definitely a rat hole (and we don't want to be Terriers pursuing rats in
this BoF).

> could
> would

I suspect the key here won't be convincing people that the IETF should
be doing work in this area. The bigger problem will be convincing them
that there is a small enough chunk of this huge problem space that the
IETF can do something sensible with it before the Internet becomes
extinct! So what is needed is very careful scoping. Yes, make it clear
that this is a problem that causes lots of people to go all gooey and
become too excited, but also impose strict limits on the scope to be
considered. 

> can't
> if
> for

I'm not sure that is THE alternative. We need to be very careful not to
come across as scare-mongers, Nor to keep harping on about the evils of
flow-fairness. Those will just open up a line of ratholes to get stuck
in...

> > understand it. You might recall the p2pi workshop, which covered
just
> a
> be

I like that refinement a lot... BUT to start that discussion we need a
slide with a very limited range of topics and a warning from the chairs
that the discussion can't get ratholed. The "discussion of viability..."
item needs to cover things like  "can we do congestion exposure?", "is
the IETF the right place to do this?", "Does re-ECn seem like a good
starting point?" and "How tightly should we limit any charter in this
area (what big items should be explicitly excluded from the charter)?"

Incidentally, strictly we have a 170 minute slot I believe (1520-1810
including the afternoon NON-snack break).

> and/or
> math!).

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 26 Oct 2009 11:46:19 +0000

Leslie,

At 22:06 25/10/2009, Leslie Daigle wrote:

>exposure, btw, not re-ECN specifically) being a challenge, but I 
>also don't think we'll get to discussing chartering a working group 
>if we don't take a whack at it.  "Focus" will be the watch-word.

[BB] Currently it looks like an open discussion session. Is that the 
intention? If so, a recipe for 'lack of focus'!

>I'm happy to discuss how to make sure that discussion _is_ 
>focused.  I actually think your smiley-table (mail to Fred) is a 
>reasonable starting point for at least some aspect of illustrating this.

[BB] Yes - that would work well.

We (the MIT CFP working group on interconnection & traffic 
management) are doing a white paper that draws of roadmap of the 
space. Once it's sufficiently coherent (soon) I'll post a draft. It 
may help structure this debate a little.

>I should also say -- I don't think the BoF can determine that 
>congestion exposure is The One True Way to better Internet 
>health.  I'm not even particularly convinced a WG can do that.  I'm 
>hoping the bar is set at demonstrating that it is a useful and 
>important step to pursue, even as others are pursue elsewhere.

[BB] Exactly.
I see the question the BoF is answering as
* "Will the IETF allow these guys a space to experiment?"

I see the question a w-g would answer as
* "Has there been sufficient verification and interest in this 
approach to move from experiment to stds track?"

BTW, 'space' in the first bullet means both space in terms of agenda 
time and potentially header space, but the precise header space at 
issue will be raised late once the w-g has deliberated

>>This looks good, but see below about the viability item, wherein 
>>lie monsters. Also, a couple of thoughts before that:
>Which is pretty cool, but on the bof agenda might lead to some 
>ratholing on whether we're just bashing TCP, no?

[BB] Not at all. Everyone in the transport area knows TCP has run out 
of dynamic range. It's a well-known problem.

I suggested adding this point, because the alternatives in this space 
seem totally impractical (e.g. upgrading every router at once).

>>2/ I think it would be cool to have a brief session on the 
>>community around this; what people are doing in this space, why, 
>>etc. This could be structured:
>>- either as one of the chairs reporting all this (requiring people 
>>to have told you in advance what they're doing - plus scraping the 
>>list archive for what people said when they introduced their 
>>interest recently),
>>- or as a vox pop.
>>It would also be a chance to report on some of the activity that 
>>has been going on to ensure the commercial & public policy 
>>community would be happy with such a change to the Internet (e.g. 
>>the GIIC thing you went to, and any ISOC activities you're planning).
>
>I'm happy to have _a_ slide pointing out there's a lot of stuff 
>going on in the world, because, as you note, it does communicate 
>that this is getting broad airtime in the world at large.  But for 
>the purpose of the BoF, I think the major emphasis should be on 
>making sure the people in the room have enough information to 
>determine whether they think there is an IETF activity here.
>
>The latter is the theory behind "the problem" item on the 
>agenda.  If there's a better way to handle that piece of it, happy 
>to take sugestions.
>
>And, I'll be sure to announce the ISOC activity here, when it's, 
>errr, announced. :)

[BB] And report on GIIC?

>>can't really discuss how viable it might be to deploy re-ECN 
>>without also considering the viability of alternative approaches to 
>>solve each of the problems we claim to be able to solve (or even 
>>whether alternatives exist).
>>congestion information rate as flow rates & link rates scale. 
>>Whereas if we stick with TCP-friendly, the information rate per 
>>window is already becoming stupidly low (hours between loss 
>>signals) as rates increase over the next few years. To put it 
>>bluntly, as carriers deploy more capacity, we'll be held back by 
>>the e2e protocols, which aren't able to use it.
>>Widespread re-ECN deployment would allow us to relax the 
>>TCP-friendly requirement, so we can bless 1/p controls and shift 
>>onto a sounder evolution path.
>>The alternative seems to be changing every router (and every LSR 
>>and every ethernet switch) in the Internet to do RCP (which still 
>>aims for flow-rate-fairness so it still drives ISPs into net 
>>neutrality issues, yada yada).
>>____
>>in the audience to understand it. You might recall the p2pi 
>>workshop, which covered just a small part of this space. It was a 
>>selected set of participants, but even then the cluefulness factor 
>>from each specialist about a each related area was painfully low.
>>I'm not saying we shouldn't try to talk about viability, but it 
>>will be a complicated session to do justice to the debate.
>
>And, the sad reality is, we will not do it justice.  What we need to 
>do is hit the high points to illustrate that there is viability 
>here, and it's worth chartering an IETF activity to delve into it in 
>more depth.
>

[BB] Yup - fine.
Except, would feasibility be a more precise word than viability?

Cheers

Bob

>>>and/or absent howls of dissent, we'll post the following. Note 
>>>that I have expanded it to allow more discussion of the specific 
>>>We'll work on framing up some discussion for that part of the 
>>>agenda during the next week, or so.
>>>
>>>Leslie.
>>>
>>>
>>>

---

From: John Leslie <[email-masked]>
Date: Mon, 26 Oct 2009 09:36:40 -0400

Bob Briscoe <[email-masked]> wrote:
>>> I've been thinking... We should add an item to the many purposes list:
>>> 

   I've been thinking somewhat along this line, though I wasn't using
the term "dynamic range", but rather considering signal-to-noise in
our feedback function.

   Beyond any question, TCP has been successful at curing "congestion
collapse" -- which is, after all, what it aimed to do.

   But packet loss, as a feedback signal, is frankly terribly unsuited
to fine-tuning how to share bandwidth at the bottlenecks. It's not,
IMHO, "bashing" TCP to point out this difference.

   Network operators, at best, can only estimate packet loss, not
measure it meaningfully, and even if they could measure it with 100%
accuracy, they'd have no hope of relating packet loss to the backoff
it signals.

   That's because the multiplicative decrease it signals is based
on end-to-end flows at a higher layer. We're looking at five or more
orders of magnitude difference in the amount of backoff signaled by
a packet lost. For a network operator, the signal is quite lost in
the noise!

   As we review the history of TCP, we notice that attempts to move
away from packet loss have failed to dependably avoid congestion
collapse: thus implementations tend to depend entirely on packet
loss to signal backoff; and other congestion-notification tends to
be ignored or even suppressed.

   (BTW, that's an issue we need to be prepared to discuss: how can
re-ecn operate when ECN marks are suppressed? Even though most of
the suppression history concerns ICMP, there will be folks who think
ECN will suffer similar suppression.)

   We should recognize that multiplicative-decrease on packet loss
is a proven winner for avoiding congestion collapse, and concentrate
on what's a better signal for management of bottleneck bandwidth.
I propose a few principles:

1) The signal needs to be visible to the network manager managing
   the bottleneck;

2) The signal should be distinct enough to be the basis for cost
   allocation for upgrading the bottleneck;

3) The signal should be visible to end-systems, giving a decent
   measure of how much to backoff their sending rate;

4) The signal should enable "backpressure" to allow network managers
   to avoid forwarding too many packets towards the bottleneck;

5) The signal should not involve packet loss.

   I believe that a properly-designed signalling system can work
for at least eight or nine orders of magnitude of sender bandwidth.
To be complete, a proposal should probably get into how many bits
per signal, but I'm personally convinced that re-ecn can work
beyond five orders of magnitude.

   So, avoiding the bits-per-signal question, are the five principles
above sufficient? Are they necessary? Can we come up with a good
presentation of such principles for the BoF?

--
John Leslie <[email-masked]>

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 26 Oct 2009 21:16:29 +0000

John,

I really like your list of principles. Where in the BoF do you 
suggest we put it?

And I agree with everything you've said in this email, with just a 
couple of inline comments...

>ECN will suffer similar suppression.)

As this sentence is in the passive, I assume you mean suppression by 
the transport or some other link than the congested one (not 
suppression by the congested link itself).

That's why we brought re-ECN to the IETF - because we had solved that 
problem. The draft-briscoe-tsvwg-re-ecn-tcp-motivation-01.txt 
explains the mechanisms that can be built over re-ECN to detect & 
prevent suppression.

I've actually got a PhD thesis of proofs on this now, with pseudocode 
of the mechanisms, simulations etc etc, but I haven't got round to 
asking my company for permission to publish (mainly because they are 
more focused on sacking people than authorising publications at the 
mo). I promise it will be published soon.

>beyond five orders of magnitude.

Have you been following Matt Mathis's work on Relentless TCP? And 
generally on TCP algos with window proportional to 1/p, rather than 
1/sqrt(p) like current TCP. The idea is these maintain the same 
number of loss or ECN signals per window however fast you go.

Is there some reason for choosing 8 or 9 orders of magnitude? I would 
have thought 1/p would scale indefinitely, but you may be thinking of 
other factors I've missed.

Scaling was also one of the main motivations for Kelly's primal algo. 
And it was one of my motivations for introducing re-ECN so we could 
shift from the TCP-friendly (1/sqrt(p)) track painlessly onto a 
scalable 1/p track without worrying about flow fairness.

Bob

---

From: Leslie Daigle <[email-masked]>
Date: Tue, 27 Oct 2009 08:37:47 -0400

I like the principles a lot, too.

I'm thinking that could be an excellent jumping off point for 
"constraints", in the agenda.   Phil's currently slated to do that 
section.  Are you (John) going to be in Hiroshima?  And/or, can you work 
with Phil to integrate these points into the discussion-leading material?

Leslie.

Bob Briscoe wrote:
> John,
> 
> I really like your list of principles. Where in the BoF do you suggest 
> we put it?
> 
> As this sentence is in the passive, I assume you mean suppression by the 
> transport or some other link than the congested one (not suppression by 
> the congested link itself).
> 
> problem. The draft-briscoe-tsvwg-re-ecn-tcp-motivation-01.txt explains 
> the mechanisms that can be built over re-ECN to detect & prevent 
> suppression.
> 
> I've actually got a PhD thesis of proofs on this now, with pseudocode of 
> the mechanisms, simulations etc etc, but I haven't got round to asking 
> my company for permission to publish (mainly because they are more 
> focused on sacking people than authorising publications at the mo). I 
> promise it will be published soon.
> 
> 
> 1/sqrt(p) like current TCP. The idea is these maintain the same number 
> of loss or ECN signals per window however fast you go.
> 
> shift from the TCP-friendly (1/sqrt(p)) track painlessly onto a scalable 
> 1/p track without worrying about flow fairness.
> 
> 
> Bob
> 
> 

--

---

From: Matt Mathis <[email-masked]>
Date: Tue, 27 Oct 2009 21:22:56 -0400

So I have two issues that I would like to see addressed, I think as
part of existing agenda items, but I want to be sure that they are not
ruled out of order by the anti-rat hole police.

First, I really think we need to broaden Re-ECN to RE-Feedback, even
under the still broader umbrella congestion exposure.  RE-ECN
presupposes the deployment of ECN, which has not deployed in 10 years.
 I believe that loss based RE-Feedback can provide much of the
benefits of RE-ECN, but with a far easier deployment strategy.
Furthermore unifying the semantics of the two approaches will make
both stronger.

My point is really that we need to view loss based and ECN based RE feedback
as equals, both likely to be present in steady state, and not one
merely as a deployment strategy for the other.  Our terminology should
reflect this.

Second, we need a task to actively promote ECN deployment.  Many
people have have worked on this, and it has been all but vetoed by a
couple of persistent problems.   Although shepherding deployment is
not a traditional IETF activity, it really think it is the right venue
to bring the right people to bear on the problem.   We need to divide
the problem, such that improved technology in a couple of specific
areas will help the broader community to help us to eliminate the tiny
persistent problems that have thwarted ECN deployment.   I would
really like to see the technological part done under ConEX.

(The required technology is to make OS releases that are targeted for
technology users and beta testes run ECN black hole detection with
automatic diagnosis and reporting.  As we get more experience with
these algorithms and they get enabled for progressively wider pools of
users we will gradually eliminate the bugs in the infrastructure).

Thanks,
--MM--
-------------------------------------------
Matt Mathis      http://staff.psc.edu/mathis
Work:412.268.3319   Home/Cell:412.654.7529
-------------------------------------------
Evil is defined by mortals who think they know
"The Truth" and use force to apply it to others.

On Tue, Oct 27, 2009 at 8:37 AM, Leslie Daigle <[email-masked]> wrote:
> I'm thinking that could be an excellent jumping off point for "constraints",
> in the agenda.   Phil's currently slated to do that section.  Are you (John)
> going to be in Hiroshima?  And/or, can you work with Phil to integrate these
> points into the discussion-leading material?
>
> Leslie.
>
> Bob Briscoe wrote:
>>
>> John,
>>
>> I really like your list of principles. Where in the BoF do you suggest we
>> put it?
>>
>> And I agree with everything you've said in this email, with just a couple
>> of inline comments...
>>
>> transport or some other link than the congested one (not suppression by the
>> congested link itself).
>>
>> problem. The draft-briscoe-tsvwg-re-ecn-tcp-motivation-01.txt explains the
>> mechanisms that can be built over re-ECN to detect & prevent suppression.
>>
>> the mechanisms, simulations etc etc, but I haven't got round to asking my
>> company for permission to publish (mainly because they are more focused on
>> sacking people than authorising publications at the mo). I promise it will
>> be published soon.
>>
>>
>> 1/sqrt(p) like current TCP. The idea is these maintain the same number of
>> loss or ECN signals per window however fast you go.
>>
>> Is there some reason for choosing 8 or 9 orders of magnitude? I would have
>> thought 1/p would scale indefinitely, but you may be thinking of other
>> factors I've missed.
>>
>> Scaling was also one of the main motivations for Kelly's primal algo. And
>> it was one of my motivations for introducing re-ECN so we could shift from
>> the TCP-friendly (1/sqrt(p)) track painlessly onto a scalable 1/p track
>> without worrying about flow fairness.
>>
>>
>> Bob
>>
>>

---

From: Leslie Daigle <[email-masked]>
Date: Wed, 28 Oct 2009 07:30:20 -0400

[Vaguely suspecting you sent your message from a non-subscribed e-mail 
address, as this doesn't appear to have come through on the main mailing 
list.]

I don't see any issue with talking about re-ECN and re-Feedback, and 
being clear about the distinction, in the context of the BoF.  I'm not 
sure how much airtime we need/want to spend on comparing and contrasting 
them there, beyond ensuring that it is clear there is at least one 
reasonable to pursue in a WG.

In terms of supporting ECN deployment -- I'm not sure how much airtime 
that requires at the BoF, beyond possible discussion of whether it is a 
gating factor for viable progress, and whether there is a work item on 
an eventual charter.  Is that reasoanble?  However, I think it is 
important to make sure it remains a collateral point.

Leslie.

--

---

From: John Leslie <[email-masked]>
Date: Wed, 28 Oct 2009 11:56:08 -0400

Leslie Daigle <[email-masked]> wrote:
> section.  Are you (John) going to be in Hiroshima?

   Alas, no.

> And/or, can you work with Phil to integrate these points into the
> discussion-leading material?

   I'd be delighted to.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 29 Oct 2009 15:08:54 +0000

Matt,

I'm more sold than ever on the idea of incorporating explicit 
re-insertion of loss into the requirements for a protocol out of ConEx.

It's v important for incremental deployment. Of all the possible 
first steps to deployment, I think end-systems before networks is the 
most likely. OS & app authors have an incentive to declare that they 
are NOT the source of a large amount of congestion in a standard way. 
Then traffic mgmt boxes will be able to check the veracity of these 
claims and reward the traffic accordingly (without inspecting packets 
deeply). Consider Microsoft BITS (a precursor to LEDBAT) that is used 
to download Windows Update and could be used by anti-virus updates 
etc. Or filesharing using scavenger transports like LEDBAT.

It's fairly straightforward to do but I describe some wrinkles below, 
if I can be allowed to talk in terms of our specific re-ECN protocol 
spec (this documents a conversation Matt & I have been having offline):

For those who don't want the technical details, here's a summary
1/ Since draft-00 (Oct 2005), the re-ECN spec has always said re-echo 
whether congestion is drop or ECN-marking.
2/ Re-ECN currently doesn't re-echo on drop unless the receiver is at 
least ECN-capable. But there's an easy way to make that happen, but 
with possible devil in details.
3/ Yes, black hole detection is critical (to re-ECN on drop as much as to ECN).

==========================================================================
There are two senarios with very different constraints. In both cases 
you have to have a re-ECN-capable sender (or proxy) to do the 
re-echoing. The distinction is whether the recevier is 
non-ECN-capable or at least ECN-capable (which includes re-ECN capable).

== 1/ At least ECN-capable receiver (ie ECN or re-ECN capable recevier) ==

Since draft-00, the re-ECN spec <draft-briscoe-tsvwg-re-ecn-tcp> 
S.6.1.1 has always said:
"                                               Whenever the ECI field
       increments by D (and/or d drops are detected), the sender MUST
       clear the RE flag to "0" in the IP header of the next D' data
       packets it sends
"
Indeed, explicitly reinserting a notification following a drop could 
be called reinserted explicit congestion notification, strangely the 
acronym for thatwould be re-ECN :)

We have recently realised there is an issue with re-echo on drop if 
the drops are from a policer (re-echoing unnecessarily gets the 
sender into deeper and deeper trouble). But I have a flakey solution 
to that, which we can work on writing in I-D language.

== 2/ Receiver that doesn't understand ECN or re-ECN. ==

The sender shouldn't use an ECN-capable codepoint for re-echoing if 
the receiver doesn't understand ECN. Otherwise, if these packets go 
through a congested ECN-capable router, they will get marked, but the 
receiver won't understand ECN marks and just ignore them.

For low levels of congestion, this seems like a corner case; it is 
only a problem for proportion p^2 of packets, where p is the loss 
fraction. So if p=1% the receiver misses a congestion notification on 
0.01% of packets. But if 20% of packets are being dropped in a 
pathological case, 20% of the sent packets will be made ECN-capable, 
so theoretically 4% of throughput could be ECN-marked and ignored by 
the receiver.

Actually, at 20% loss levels, RFC3168 says a queue shouldn't still be 
ECN-marking, but the complexity around this debate is beyond the 
scope of this email...

One solution would be to define a non-ECN-capable codepoint for 
re-echoing drop. As there's only one non-ECN-capable codepoint of the 
ECN, field (00), with only one bit available in IPv4 (the reserved 
flag or RE), there seems to be only one choice:
         ECN = 00, RE = 1

This combination is already defined in re-ECN for meaning "Cautious 
expectation of downstream congestion" or "I am declaring my 
expectation of congestion conservatively, because I've not seen 
enough recent feedback to know the path". This is intended for use at 
the start of a flow, or after an idle period.

We would have to fully work out what it would break to overload this 
meaning with "I've seen actual congestion but it was drop and the 
receiver doesn't understand ECN."

There's also a wrinkle in that this combination can be considered 
ECN-capable by routers under strong conditions given in the draft. I 
think these conditions make it safe for re-echoing drop as well.

So this offers a significant ray of hope that this will be 
straightforward to specify and implement. But the devil may be in the details.

3/ End-point OSs & apps will only deploy re-ECN opportunistically 
(without any certain benefit from the network) if there is absolutely 
no downside. Therefore, we will need an aggressive programme to 
remove black holes for the use of the reserved flag (if that's what 
we go for), and code to do fast blac-hole detection learning from the 
experience with ECN, as Matt says.

I imagine a stds track RFC saying the reserved flag has been 
reassigned from reserved to experimental use (without saying what 
use). That can be used to beat equipment vendors over the head with, 
if they drop or clear that flag.

Experience with ECN showed that happened over 2-3yrs, but the 
residual problems of home gateways weren't ever attacked sufficiently 
hard. The deployment incentive for ECN was a small performance gain. 
The incentives for re-ECN could be a lot stronger.

Cheers

Bob

At 11:30 28/10/2009, Leslie Daigle wrote:
>[Vaguely suspecting you sent your message from a non-subscribed 
>e-mail address, as this doesn't appear to have come through on the 
>main mailing list.]
>
>being clear about the distinction, in the context of the BoF.  I'm 
>not sure how much airtime we need/want to spend on comparing and 
>contrasting them there, beyond ensuring that it is clear there is at 
>least one reasonable to pursue in a WG.
>
>In terms of supporting ECN deployment -- I'm not sure how much 
>airtime that requires at the BoF, beyond possible discussion of 
>whether it is a gating factor for viable progress, and whether there 
>is a work item on an eventual charter.  Is that 
>reasoanble?  However, I think it is important to make sure it 
>remains a collateral point.
>
>Leslie.
>
>Matt Mathis wrote:

---

From: João Taveira Araújo <[email-masked]>
Date: Thu, 29 Oct 2009 15:49:56 +0000

Bob,

I talked to you offline about this the other day, but here goes my main 
concern..

> whether congestion is drop or ECN-marking.

Which suggests there is good reason to separate both. The endpoint 
behaviour doesn't change, but the network's understanding of what is 
happening does, and so you'll get an overestimation of upstream 
congestion. With a loss echo, you can give a neutral value (as with 
cautious packets) but still rate limit at the ingress. However...

> with possible devil in details.

You don't want to mix FNE's with l-echos. Besides being a weak 
abstraction, the FNE flag is critical in creating state, which you use 
for policing flows and sampling interdomain. Another problem with loss 
echos without re-ECN is it makes it impossible for networks to verify 
the percentage of e2e loss over bulk traffic, since they don't know what 
traffic marks loss or not.

> 3/ Yes, black hole detection is critical (to re-ECN on drop as much as 

I'm using the currently unused codepoint to signal loss echos. 
Correlating the amount of loss, congestion and RECT traffic to a 
destination might be a better way of detecting ECN black holes, 
especially as ISPs have an incentive to prod the deployment of something 
like this along. It might not be the easiest solution, but I think it's 
cleaner: this way you can tell if it's an "evil bit" black hole (dropped 
FNEs), or just ECN averse (FNE sets up state but packet which follows 
through is lost. Multiple loss echos follow?).

Joao.

---

From: <[email-masked]>
Date: Thu, 29 Oct 2009 17:23:48 -0000

> > discussion-leading material?

thanks John, that would be excellent.

I've been trying to reach you direct, but am getting bounces from your
mailbox [[email-masked]] - is there a different email?

I will send some first ideas to the ML shortly

Thanks
phil

---

From: Matt Mathis <[email-masked]>
Date: Thu, 29 Oct 2009 14:58:28 -0400

I made a comment on the list some time ago that pertains to your point below:

2009/10/29 João Taveira Araújo <[email-masked]>:
> Which suggests there is good reason to separate both. The endpoint behaviour
> doesn't change, but the network's understanding of what is happening does,
> and so you'll get an overestimation of upstream congestion. With a loss
> echo, you can give a neutral value (as with cautious packets) but still rate
> limit at the ingress. However...

My goal would  be to tweak RE-ECN and loss based RE feedback, such
that the signal semantics are congruent.  After all a loss and ECN
mark are supposed to be equivalent.

>> least ECN-capable. But there's an easy way to make that happen, but with
>> possible devil in details.
>
> You don't want to mix FNE's with l-echos. Besides being a weak abstraction,
> the FNE flag is critical in creating state, which you use for policing flows
> and sampling interdomain. Another problem with loss echos without re-ECN is
> it makes it impossible for networks to verify the percentage of e2e loss
> over bulk traffic, since they don't know what traffic marks loss or not.

The Ah Ha is that this condition detectable/enforceable.   There has
to be a one-to-one correspondence between RE marks and
retransmissions.  For unencrypted connections, these can be detected
by looking at TCP sequences numbers.  Yes it requires per flow state,
but it does not need to be done at the most heavily aggregated points
in the network

Thanks,
--MM--

---

From: Matt Mathis <[email-masked]>
Date: Thu, 29 Oct 2009 15:47:12 -0400

I would like to harp on one of Bob's remarks:

> We have recently realised there is an issue with re-echo on drop if the
> drops are from a policer (re-echoing unnecessarily gets the sender into
> deeper and deeper trouble). But I have a flakey solution to that, which we
> can work on writing in I-D language.

I think part of the problem here is a flaw in the way that RE-ECN
policers are envisioned.  A small tweak might make everything fit
whole lot better.    As currently envisioned a policer monitors the
total RE marks, which reflect the total congestion for the entire
path.  This then forces the policer action to be some other signal,
otherwise the police action self regenerates.

A better design would be if the policer only acted on the down stream
congestion, as indicated by RE marks minus ECN marks (remember, ECN
marks at the policer reflect upstream congestion).  With this design
the policer's action can be to apply additional ECN marks at its own
ingress interface.   This does not directly effect the downstream
congestion signals (until latter RTTs when the transport protocols
have had an opportunity to respond).

The unifying ah-ha that connects loss based RE-feedback and RE-ECN is
the observation that  "RE marks minus ECN marks" is exactly the same
signal as counted duplicate data - It is the number of packets that
will be (have been) marked or dropped further down stream.

This policer design also has a simple physical interpretation: it is a
bottleneck that is specifically designed to take the pressure off of
downstream bottlenecks.

After mulling this over, I am suspicious that there are other places
where "RE marks" should be replaced by "RE marks minus ECN marks" for
example at settlement boundaries.   Ok I will make the brash claim
that a naked "RE marks" is wrong anywhere except when talking about
the end systems.

Thanks,
--MM--

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 30 Oct 2009 16:02:50 +0000

Matt,

At 19:47 29/10/2009, Matt Mathis wrote:
>path.

You're right that the policer in Appx B of
         draft-briscoe-tsvwg-re-ecn-tcp-motivation-01
monitors whole path congestion. But since implementing it, we've 
moved to the same idea as you - using downstream congestion - we 
haven't written it up yet tho.

>This then forces the policer action to be some other signal,
>marks at the policer reflect upstream congestion).

It does (now).

>With this design
>congestion signals

I'll let this one pass for now, but it isn't quite true. (The policer 
might well be introducing a fair bit of drop - if the policer matched 
25% policer drop with 25% CE marking introduced at its ingress, that 
would hide 25% of any ECN marking further downstream.)

>(until latter RTTs when the transport protocols

Yes.
Except duplicates have to be measured per flow and at the transport 
layer, and they're specific to certain transports - as already argued 
out on this list.

>bottleneck that is specifically designed to take the pressure off of

Sort of true - I'd rather say taking the pressure off downstream 
bottlenecks is a side-effect. The design objective it has to be 
written against is to create sufficient disincentive to cause 
congestion, which gives a precise way to calculate exactly how much 
it should drop.

>example at settlement boundaries.

Absolutely - right from the very first twinkle of re-feedback in the 
eye - that was the original design goal - to be able to measure 
downstream congestion simply in bulk at borders. The aim was 
something passive and simple that would work with all-photonic 
borders in the future.

>Ok I will make the brash claim

Correct. Definitely... Brashly even. :)

Cheers

Bob

---

From: João Taveira Araújo <[email-masked]>
Date: Fri, 30 Oct 2009 16:25:39 +0000

Quick comment (more of a doubt actually)

Matt Mathis wrote:
> retransmissions.

There is some sort of equilibrium property, but re-ECN (for now) has the 
notion of building up credit at flow start almost as a sign of goodwill 
to the policer. From my understanding this opens up the possibility of a 
retransmission NOT being marked. For example, if a flow is finishing off 
with sufficient credit to offset CE markings, it could choose to not 
re-echo congestion. Bob should be able to clear this one up.

This also made me wonder whether the one-to-one correspondence is simply 
a trademark of TCP or whether it's a characteristic we want to enforce 
on every future transport protocol. Considering most short flows don't 
get out of slow start, if some were to transfer all their data in a 
burst of (re) marked packets, is this inherently wrong?

Joao