Subject: [re-ECN] Two questions about CONEX

From: Stewart Bryant <[email-masked]>
Date: Sat, 08 May 2010 07:24:26 +0100

As I was reading the recent email on CONEX I got the impression that a 
lot of the interest in CONEX is to instrument the network so that 
operators can identify the root causes of congestion and understand the 
impact on the network in more detail.

  I also get the impression that at least some of the operators are 
looking for a relatively long time constant in the feedback loop.

That causes me to wonder where CONEX fits in relative to IPFIX which is 
a mechanism that is designed to monitor the flows in a network and 
report this information to the network operator.

As I noted in a earlier thread, I am also interested in understanding 
why the host needs to use the data packets themselves to indicate the 
expected congestion state to the network rather than using a fate 
sharing OAM mechanism?

- Stewart

---

From: "Woundy, Richard" <[email-masked]>
Date: Sat, 8 May 2010 09:40:37 -0400

So a couple of questions back to you.
 
1. I gave the 'CMTS congestion management' as a representative example. But what if I need similar measurements/mechanisms at my backbone interconnects? (That's probably the second place I would need to worry about congestion.) Can these OAM messages make it there?
 
2. A typical subscriber host is behind one (or more) Ethernet/WiFi home gateways, then behind a broadband modem that may or may not use Ethernet as a layer two transport over the broadband access network. So are we talking about a layer 2 or 3 OAM message? If layer 3, how would a host behind a NAT know who to address it to? (I don't personally like NAT66 but that may be the subscriber's choice rather than mine.) If layer 2, how do I get the subscriber to upgrade all of their home CPE? Plus how many layer 2 technologies need to define this OAM message? Or do we need to define a new home network layer 2 (please no)?
 
3. What if the OAM message needs to be originated by the application/content server side, rather than the subscriber side? How would that server know which CMTS / interconnect router to send it to?
 
-- Rich
 
P.S. The long time constant in the CMTS congestion management feedback loop is dependent on the state of the art today (DOCSIS specs require 15 minute IPDR polling cycle support for the CMTS). In some ways this is a bug, in some ways this is a feature. :)

________________________________

From: [email-masked] on behalf of Stewart Bryant
Sent: Sat 5/8/2010 2:24 AM
To: [email-masked]
Subject: [re-ECN] Two questions about CONEX

- Stewart

_______________________________________________
re-ECN mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/re-ecn

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 08 May 2010 18:32:00 +0100

> 1. I gave the 'CMTS congestion management' as a representative example. But what if I need similar measurements/mechanisms at my backbone interconnects? (That's probably the second place I would ne…
That would depend on how they are carried. For example (and only for 
example) if they were embedded in the transport protocol itself they 
would go where ever the transport went.

See above.
> 3. What if the OAM message needs to be originated by the application/content server side, rather than the subscriber side? How would that server know which CMTS / interconnect router to send it to?
See above.
> P.S. The long time constant in the CMTS congestion management feedback loop is dependent on the state of the art today (DOCSIS specs require 15 minute IPDR polling cycle support for the CMTS). In s…
I agree, but I was not thinking of 15mins, on the other hand I was not 
thinking of packet rate which is where I believe current thinking is.

- Stewart

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: "Woundy, Richard" <[email-masked]>
Date: Sun, 9 May 2010 16:04:50 -0400

>For example (and only for example) if they were embedded in the transport protocol itself they would go where ever the transport went.

True. But let's follow this example all the way through. The congestion signal would need to be defined for every transport protocol, right? And how do you account for UDP, since UDP doesn't have any unused/option bits? At a minimum, the congestion signal would need to be defined for RTP/RTCP, SNMP, and proprietary protocols such as uTP.
 
Are you implying that congestion control and monitoring in the network will require deep packet inspection by network elements? Sorry, I do NOT want to be the one carrying *that* message to the network neutrality crowd...
 
-- Rich

From: Stewart Bryant [mailto:[email-masked]]
Sent: Sat 5/8/2010 1:32 PM
To: Woundy, Richard
Cc: [email-masked]
Subject: Re: [re-ECN] Two questions about CONEX

See above.
See above.

- Stewart

---

From: Kevin Mason <[email-masked]>
Date: Mon, 10 May 2010 10:56:12 +1200

> expected congestion state to the network rather than using a fate

I believe it is highly desirable to keep the feedback mechanism as simple as practically possible.

There are broadly two groups involved in an end to end path that should/could have an interest in congestion information. These are

1/ the end user (in reality the user's host on their behalf)
2/ A network provider proving transport resources over part of the end to end path.

IF the feedback was contained in some other protocol like an OAM message then this protocol would have to address the end user's hosts (other wise the end user application will not receive it) and all devices end to end would have to be enabled to forward that additional protocol. Transit devices would need to be able to snoop the separate protocol and relate it to the flow of interest, end hosts would need to support the protocol and also relate it to the flow. To me this is getting more complex than having information inside the underlying protocol that can either be ignored or inspected by transit devices and the flow that the information relates to is unambiguous.

If the feedback is implemented using a layer 4 mechanism (like RTCP)then again this becomes more complex as every layer 4 protocol would need to develop their own implementation and all transit devices would need to recognise all these variants.

Both the above approaches require information to be put in separate packets.

I would imagine that the extraction of this information by a network provider for accounting, be it to trigger end user policy changes and/or to underpin some other commercial construct, is only one use. My assumption is that this feedback information may be used for more immediate mechansims, even to inform some forwarding decisions in intermediate devices. So I would not like to see this potential unnecessarily excluded or made more complex than it needs to be without good reason.

So a key issue seems to be how frequently any interested user host or intermediate party is ever likely to need congestion feedback information and therefore, is putting that information in the lowest layer end to end protocol reasonable to make it available to all likely audiences, or would it be more efficient overall to convey the information less frequently in separate packets using a separate protocol.

IT is conceivable that congestion reporting may be exported by a device using an IPFIX mechanism for some uses. But I do not see that as being suitable for all devices/uses.

IP is the lowest layer that is preserved end to end in most communications today. Some claim end to end Ethernet will become more common, but this has not occurred yet and is not likely to soon, the Internet remains an IP network.

So the key question seems to me to be; 

What is the down side of putting the feedback information in the IP "header"? 

>From a simple transport overhead view, at what point is adding any extension header (assuming that is the IPv6 approach) less efficient than putting the information is a separate packet. 

For some protocol a separate packet might be needed anyway if the frequency of backward responses inherent in the protocol in use is too low for the information to be usefully acted on. I suspect any application using such a protocol will be highly inelastic and would need some higher layer longer duration controls in place to avoid congestion impacts anyway.

Regards

Kevin Mason

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Woundy, Richard
Sent: Monday, 10 May 2010 8:05 a.m.
To: [email-masked]
Cc: [email-masked]
Subject: Re: [re-ECN] Two questions about CONEX

 
 
-- Rich

See above.
See above.

- Stewart

---

From: "Woundy, Richard" <[email-masked]>
Date: Mon, 10 May 2010 08:23:21 -0400

If folks in general (not just the IESG) think there are a lot of alternatives to Conex, maybe the ADs should form a design team that uses a real-time form of communication to talk this through? I'm thinking any combination of f2f meeting, conference call, or Webex to start...
 
-- Rich

See above.
See above.

- Stewart

---

From: ken carlberg <[email-masked]>
Date: Mon, 10 May 2010 09:17:37 -0400

just a couple of various thoughts....

if there are alternatives to Conex, cool.  It would be nice to hear people volunteering for a design team and see what they come up with.  But i think that effort then probably needs to go to the IETF list to kick it off for wider distribution/advertisement, and then possibly have another list to go into the specifics and start separate BoF.  

I found that I agreed a fair amount what was previously sent to the list by Kevin Mason.  I'd add that if OAM was the direction to take, then isn't the direction of a solution now being pared down to the boundaries of a specific transit -- and really just the leaf transit?  My impression with Conex was that it was not constrained (even policy-wise) to a single transit but was independent to that because we are placing the info in the IP packet.  But please correct me if I'm missing something.

-ken

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 10 May 2010 15:46:47 +0100

I would like to clarify what my concern is here.

If the marking is in every packet then the forwarder needs to take 
action on every packet.

The thing that makes IP (and MPLS) cheap and fast is that we only do the 
minimum amount of processing in forwarding and do as much as we can by 
other means. In particular everything other than forwarding happens at a 
much slower rate, is often asynchronous WRT forwarding, and usually 
happens in a process that can schedule the processing at some convenient 
time.

I used the term OAM rather loosely earlier to indicate a packet that 
traversed the path, fate sharing with the data, containing information 
of note intended for the intermediate systems.  I make no comment on how 
such packets might be carried or fate shared.

My question here is that whilst I can see that we need to do better than 
the 15mins that systems seem to do at the moment, do we need to go all 
the way to making the reporting the congestion state on every packet.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 10 May 2010 15:50:16 +0100

Ken

Please remember that part of the problem here is to test whether CONEX 
is sufficiently cooked that it is an engineering problem, or whether it 
is still a research phase problem.

Stewart

---

From: ken carlberg <[email-masked]>
Date: Mon, 10 May 2010 11:27:17 -0400

Stewart,

I appreciate the added clarity, and its a fair question to ask.  My own impression is that one could/should have some wiggle room to report the congestion state in some subset of packets to provide hints versus a granular in-depth approach of relying on state placed in every packet.  I assume your concern stems from a possible excess examination of state, which is fair concern.  Ideally, information gleaned from the experimental status of the effort would inform us if the effectiveness of Conex is only tangible if the congestive state is reported in every packet during that condition. 

but before I box myself into a corner :-), an acceptance of different modes of operation doesn't (in my opinion) place the effort as purely research.  rather, its an identification of one item that needs to be fleshed out in this role as an IETF experimental effort.  your follow up statement of...

> Please remember that part of the problem here is to test whether CONEX is sufficiently 
> cooked that it is an engineering problem, or whether it is still a research phase problem.

...is also an understandable one to press the group on.  My understanding is that the lads at BT have pushed this beyond just a research phase such that we at least have a specific direction to aim at.  Do we have a bleeding edge engineering effort being placed on our lap?  Probably.   But my impression is that this falls within the scope of what an experimental IETF effort should be.  Certainly, the proposed CONEX effort is nothing near to what was put forward with IPv5 and the Stream Protocol -- another experimental effort of ages ago.

cheers,

-ken

On May 10, 2010, at 10:46 AM, Stewart Bryant wrote:

> If the marking is in every packet then the forwarder needs to take action on every packet.
> 
> The thing that makes IP (and MPLS) cheap and fast is that we only do the minimum amount of processing in forwarding and do as much as we can by other means. In particular everything other than forwarding happens at a much slower rate, is often asynchronous WRT forwarding, and usually happens in a process that can schedule the processing at some convenient time.
> 
> I used the term OAM rather loosely earlier to indicate a packet that traversed the path, fate sharing with the data, containing information of note intended for the intermediate systems.  I make no comment on how such packets might be carried or fate shared.
> 
> My question here is that whilst I can see that we need to do better than the 15mins that systems seem to do at the moment, do we need to go all the way to making the reporting the congestion state on every packet.
> 
> - Stewart
> 
> 
>

---

From: John Leslie <[email-masked]>
Date: Mon, 10 May 2010 13:10:35 -0400

> If the marking is in every packet then the forwarder needs to take 

   There might be forwarders expected to act on every packet, but I
don't believe any current design calls for that (beyond what forwarders
might already do in Explicit Congestion marking). In particular, it
seems thoroughly unlikely that backbone routers will be expected to
do anything.

> happens in a process that can schedule the processing at some convenient 

   Agreed.

   But "middleboxes" that do various translations and such on every
packet are common and becoming even more common.

> such packets might be carried or fate shared.

   Thanks for the clarification; I was quite confused by that term.

> My question here is that whilst I can see that we need to do better
> than the 15mins that systems seem to do at the moment, do we need to
> go all the way to making the reporting the congestion state on every
> packet.

   I'm not sure...

   The reECN proposal marks very few packets as Congestion Expected;
and in the ideal case it hopes that they will pick up Explicit
Congestion markings as they go through forwarders experiencing
congestion.

   I'm not aware of any expectiations of processing beyond that for
reECN.

   But it's early to say whether we might encode differently in IPv6
ConEx. We do expect middleboxes (for example, policers) to examine
ConEx markings, and it's conceivable they might forward ConEx flows
differently. This issue may or may not be discussed, but actual design
of such middleboxes is clearly out-of-scope in the initial charter.

   Thus, there _might_ be a reason to mark every packet of a ConEx
flow as "ConEx-aware" -- strictly for the convenience of middleboxes.

   More specifically, to what I think is your question: "is there a
need to report congestion on every packet?"...

   There is a clear benefit attached to a transport protocol learning
of congestion without waiting to declare a packet has been dropped.
This has been clearly recognized in all the Explicit Congestion
Notification work. ConEx adds nothing to that issue, but does stand
to benefit from it.

   There is also, IMHO, a potential benefit from estimating potential
congestion _before_ waiting for a full Round-Trip-Time. Such work is
clearly research-level, and is no way "included in" ConEx; but could
be enabled by finer gradations of "pre-congestion-vs-time" information.
As our work proceeds, we will see whether a finer gradation of
Congestion-Expected marking turns out to be practical; I suspect it
may be.

   Do note, there's a real issue _today_ of TCP not filling pipes
whose bandwidth-delay is too large. IMHO, this will _need_ to be
solved by transport protocols that back-off more gently upon
"signals" of congestion or impending congestion. This work goes
beyond any foreseen ConEx charter; but I'd like to believe that
ConEx work could enable better precision of congestion signals.

   Signaling "expected congestion" is necessarily a technology
which will enable all kinds of optimizations which we can't yet
imagine. We really need to standardize _something_ in this realm,
without worrying how the things it will enable will impact the
job of current router boxes.

   Do we need to go all the way to reporting congestion state on
every packet? No, but it might be good to have an option which
allows for that. And the farther from that we settle, the greater
the danger that the packets we _do_ mark will turn out to not be
typical of the whole flow. This is an issue that needs a WG to
find the reasonable consensus: I don't believe we can solve it
in a charter.

--
John Leslie <[email-masked]>

---

From: Ron Bonica <[email-masked]>
Date: Mon, 10 May 2010 15:23:56 -0400

Richard,

I like this proposal. *alot*.

I think that we should charter a WG and task that WG with the
development of exactly one document. That document captures use cases in
which the ip-layer does something useful with congestion information.
(This is currently out of scope for CONEX).

When that WG has completed its work, we should have another BoF in which
multiple solutions (including CONEX) can be entertained.

I don't know if there will be any other solutions, but it's always a
good idea to put the problem statement ahead of the solution.

                                    Ron

On 5/10/2010 8:23 AM, Woundy, Richard wrote:
> If folks in general (not just the IESG) think there are a lot of alternatives 
to Conex, maybe the ADs should form a design team that uses a real-time
form of
communication to talk this through? I'm thinking any combination of f2f
meeting,
conference call, or Webex to start...
> .

---

From: ken carlberg <[email-masked]>
Date: Mon, 10 May 2010 16:38:12 -0400

I'm sorry for getting lost, but how does capturing use cases produce or correlate to a problem statement?  And given that topic, could you then expand on the issues you have with the current Conex problem statement?

Outside of that, I look forward to reading your contribution(s) to this new effort you are in favor of.

kindest regards,

-ken

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 17 May 2010 14:03:23 +0100

Stewart,

At 15:46 10/05/2010, Stewart Bryant wrote:
>The thing that makes IP (and MPLS) cheap and fast is that we only do 
>the minimum amount of processing in forwarding and do as much as we 
>can by other means. In particular everything other than forwarding 
>happens at a much slower rate, is often asynchronous WRT forwarding, 
>and usually happens in a process that can schedule the processing at 
>some convenient time.

That was exactly the goal of re-ECN (which is a candidate protocol for ConEx).

The design of re-ECN is I think the same as the approach you loosely 
called 'OAM-based', then clarified below. Traffic control boxes take 
note of the per-packet re-ECN markings. But all other forwarding 
boxes ignore them (preferably marking ECN though, when congested).

The idea is to *remove* per-packet operations from forwarding boxes, 
shifting information from forwarding boxes to edge control boxes, and 
doing the traffic control once at the outermost edges of the internetwork.

One of our more ambitious goals was to be able to do all-optical 
packet forwarding at inter-domain borders (and within domains) with 
reduced photonic processing per packet. That requires pushing the 
incentives to control traffic back to the first ingress to the internetwork.

At the outer edge, the minimum per-packet policing function we have 
managed to achieve without weakening the incentives is a single token 
bucket per user - but unlike a traditional token bucket it regulates 
only the bit-rate of re-ECN marked packets, not all packets.

>traversed the path, fate sharing with the data, containing 
>information of note intended for the intermediate systems.  I make 
>no comment on how such packets might be carried or fate shared.
>
>go all the way to making the reporting the congestion state on every packet.

This question might be born of complacency that it would be easy to 
stretch to a coarser timescale. Perhaps this stems from assuming 
congestion and utilisation are related. But the two are linked 
differently for different transport behaviours.

This is why LEDBAT-like transports can send a lot of data, helping to 
create high utilisation, but not be responsible for hardly any of the 
congestion. While other transports can contribute a lot to the 
congestion, without sending much data (e.g. unresponsive streaming).

That's why it's important that ECN markings (and drops) are 
per-packet. Also, by using in-band markings, you get a secure binding 
for free between the congestion information and the data. That buys a 
lot of simplicity.

I think worrying about over-precision is a red herring. For a traffic 
control box to add up the sizes of marked packets really isn't heavy 
processing.

If it's cheapest to get a precise result, why make the system 
complicated in order to make it imprecise?

Bob

>>if there are alternatives to Conex, cool.  It would be nice to hear 
>>people volunteering for a design team and see what they come up 
>>with.  But i think that effort then probably needs to go to the 
>>IETF list to kick it off for wider distribution/advertisement, and 
>>then possibly have another list to go into the specifics and start 
>>separate BoF.
>>
>>I found that I agreed a fair amount what was previously sent to the 
>>list by Kevin Mason.  I'd add that if OAM was the direction to 
>>take, then isn't the direction of a solution now being pared down 
>>to the boundaries of a specific transit -- and really just the leaf 
>>transit?  My impression with Conex was that it was not constrained 
>>(even policy-wise) to a single transit but was independent to that 
>>because we are placing the info in the IP packet.  But please 
>>correct me if I'm missing something.
>>
>>-ken
>>
>>
>>
>>>If folks in general (not just the IESG) think there are a lot of 
>>>alternatives to Conex, maybe the ADs should form a design team 
>>>that uses a real-time form of communication to talk this through? 
>>>I'm thinking any combination of f2f meeting, conference call, or 
>>>Webex to start...
>>>
>>>-- Rich
>>>
>>>>1. I gave the 'CMTS congestion management' as a representative 
>>>>example. But what if I need similar measurements/mechanisms at my 
>>>>backbone interconnects? (That's probably the second place I would 
>>>>need to worry about congestion.) Can these OAM messages make it there?
>>>>
>>>>
>>>2. A typical subscriber host is behind one (or more) Ethernet/WiFi 
>>>home gateways, then behind a broadband modem that may or may not 
>>>use Ethernet as a layer two transport over the broadband access 
>>>network. So are we talking about a layer 2 or 3 OAM message? If 
>>>layer 3, how would a host behind a NAT know who to address it to? 
>>>(I don't personally like NAT66 but that may be the subscriber's 
>>>choice rather than mine.) If layer 2, how do I get the subscriber 
>>>to upgrade all of their home CPE? Plus how many layer 2 
>>>technologies need to define this OAM message? Or do we need to 
>>>define a new home network layer 2 (please no)?
>>>
>>>See above.
>>>
>>>>3. What if the OAM message needs to be originated by the 
>>>>application/content server side, rather than the subscriber side? 
>>>>How would that server know which CMTS / interconnect router to send it to?
>>>>
>>>>
>>>See above.
>>>
>>>>-- Rich
>>>>
>>>>P.S. The long time constant in the CMTS congestion management 
>>>>feedback loop is dependent on the state of the art today (DOCSIS 
>>>>specs require 15 minute IPDR polling cycle support for the CMTS). 
>>>>In some ways this is a bug, in some ways this is a feature. :)
>>>>
>>>>

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 17 May 2010 17:50:16 +0100

Kevin,

At 23:56 09/05/2010, Kevin Mason wrote:
> >From a simple transport overhead view, at what point is adding any 
> extension header (assuming that is the IPv6 approach) less 
> efficient than putting the information is a separate packet.
>
>For some protocol a separate packet might be needed anyway if the 
>frequency of backward responses inherent in the protocol in use is 
>too low for the information to be usefully acted on. I suspect any 
>application using such a protocol will be highly inelastic and would 
>need some higher layer longer duration controls in place to avoid 
>congestion impacts anyway.

We deliberately designed re-ECN so that an e2e transport with 
infrequent feedback suffers the consequences of the extra risk it 
takes. This encourages it to
- either use sufficient feedback,
- or, in the absence of sufficient feedback, for its increases in 
sending rate to be more cautious.

Cheers

Bob

---

From: Fred Baker <[email-masked]>
Date: Mon, 17 May 2010 10:24:28 -0700

> (This is currently out of scope for CONEX).

I think this is too narrow a scope. I think the document should describe use cases for conex. 

Consider RFC 3168 as a prototype for this document. It identifies that the network can easily and cheaply flag congestion events (far more cheaply than with IPFIX, and far more accurately), but it also identifies that the responsibility for responding to the event lies in the transport layer. If you had put this requirement on the authors of RFC 3168, they would not have been able to describe what is in fact a very useful capability. (IMHO, either RFC 3168 or a delay-based congestion avoidance procedure in the transport would be very sufficient for ledbat, for example).

I don't think you actually need a WG for this document. What you need is for individuals that have use cases to post drafts describing them. If conex wants to pull those together into a use case document, they can do that at their leisure.

---

From: Fred Baker <[email-masked]>
Date: Mon, 17 May 2010 22:59:43 -0700

On May 7, 2010, at 11:24 PM, Stewart Bryant wrote:

> That causes me to wonder where CONEX fits in relative to IPFIX which is a mechanism that is designed to monitor the flows in a network and report this information to the network operator.

That is a fairly complicated question.

First, IPFIX is not at all lightweight. At the high end, to counter that, it is often statistical - collecting data on all flows is considered a lot of data and, if a reliable transport is used to deliver it, potentially a large amount of memory dedicated to the function. Setting or not setting a two bit field in the IP header is something that any device that can decrement a TTL or hop limit can easily do with effectively no additional overhead. If you're trying to consistently manage high end users, which is what conex is ultimately about, which can be implemented anywhere and gives you full information?

Second, IPFIX doesn't really measure the same thing as ECN does. What ECN does is note that at the point in time that a datagram is traversing a node, the node is or is not experiencing "congestion" as defined locally. The definition for that class of traffic at that point could be "integrated throughput rate exceeds <threshold>", "instantaneous queue depth exceeds <threshold>", or "mean queue depth exceeds <threshold>", and it might mean "marked with some probability" or "marked if the case is true". Whatever its definition, you can determine that the node in question saw something it called "congestion". IPFIX tells you what went through a system - between <time1> and <time2>, data flow <selector> passed some number of bytes and packets. You can now feed that into a simulator of some type, but at best it will give you an approximation of the behavior of the link conditioned by your assumptions, not "the behavior of the link".

Where I scratch my head with re-ecn is the impact of the re-introduction of the flag stream. It looks to me like a control system, which is presumed to be somewhere at a network edge, is supposed to be scoreboarding marks against IP addresses or IP prefixes, which sounds to me like the kind of overheads one sees in XCP, RCP, or the like, but with storage and discrimination in some form of IP address database. That worries me from a scaling perspective. I'm in favor of having ISPs be able to limit congestion in a reasonable way, but I'm interested in approaches that scale. I'm also concerned about what looks to me like a persistent under-run. If Alice sends Bob a stream of traffic, Bob replies noting her mark count, and she re-replies re-inserting that mark count, there is always some probability that her re-inserted reply is itself marked and Bob no longer cares. If the network is drawing broad conclusions from aggregated marks, it seems it would need to account for the under-run.

Another way in which I scratch my head is what one actually does. If we're OK with maintaining per-address or per-prefix state of some sort, I would find myself wondering about a variation on fair queuing that uses the later of "a time or sequence number dictated by the source address" and "a time or sequence number dictated by the destination address", and only keeps track of addresses/prefixes that have been recently used. That allows the ISP to be in control of its own congestion without the compliance of neighboring networks or the hosts in them. But that's just me.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 18 May 2010 10:21:38 +0100

Fred,

At 06:59 18/05/2010, Fred Baker wrote:

> > That causes me to wonder where CONEX fits in relative to IPFIX 
> which is a mechanism that is designed to monitor the flows in a 
> network and report this information to the network operator.
>
>First, IPFIX is not at all lightweight. At the high end, to counter 
>that, it is often statistical - collecting data on all flows is 
>considered a lot of data and, if a reliable transport is used to 
>deliver it, potentially a large amount of memory dedicated to the 
>function. Setting or not setting a two bit field in the IP header is 
>something that any device that can decrement a TTL or hop limit can 
>easily do with effectively no additional overhead. If you're trying 
>to consistently manage high end users, which is what conex is 
>ultimately about, which can be implemented anywhere and gives you 
>full information?
>
>Second, IPFIX doesn't really measure the same thing as ECN does. 
>What ECN does is note that at the point in time that a datagram is 
>traversing a node, the node is or is not experiencing "congestion" 
>as defined locally. The definition for that class of traffic at that 
>point could be "integrated throughput rate exceeds <threshold>", 
>"instantaneous queue depth exceeds <threshold>", or "mean queue 
>depth exceeds <threshold>", and it might mean "marked with some 
>probability" or "marked if the case is true". Whatever its 
>definition, you can determine that the node in question saw 
>something it called "congestion". IPFIX tells you what went through 
>a system - between <time1> and <time2>, data flow <selector> passed 
>some number of bytes and packets. You can now feed that into a 
>simulator of some type, but at best it will give you an 
>approximation of the behavior of the link conditioned by your 
>assumptions, not "the behavior of the link".

Agree.

As John Leslie said in his talk to the transport area in LA, network 
layer people use the word congestion as synonymous with utilisation. 
Transport layer people use congestion to mean queue growth.

Average utilisation isn't sufficient to be able to determine what the 
different contributions to congestion (queue growth) are of different 
users, if some are running unresponsive transports, and others are 
running highly yielding transports (e.g. LEDBAT).

* In ConEx, we're trying to find the product of bit-rate and queue 
congestion over time.
* IPFIX finds bit-rate over time.

>Where I scratch my head with re-ecn is the impact of the 
>re-introduction of the flag stream. It looks to me like a control 
>system, which is presumed to be somewhere at a network edge, is 
>supposed to be scoreboarding marks against IP addresses or IP 
>prefixes, which sounds to me like the kind of overheads one sees in 
>XCP, RCP, or the like, but with storage and discrimination in some 
>form of IP address database. That worries me from a scaling perspective.

It's not intended to be like XCP/RCP. The fine control is intended to 
still be at the end-system not in the network equipment. The 
re-ECN/ConEx info merely allows the network to encourage all the 
end-system controls to balance their aggressiveness, to achieve a 
'fair' amount of harm to others over time.

The minimum a network needs to store is one number per directly 
connected customer (whether end-customer or neighbouring network). 
And there's no need for each router to do this. One traffic control 
device per connected 'customer' needs this info, and the packets 
bring this info to it.

That's surely the opposite of a scaling problem.

This 6pp paper is the best we could do to succinctly describe how the 
most minimalist control system could work:
   Policing Freedom to Use the Internet Resource Pool
   <http://www.bobbriscoe.net/projects/refb/#polfree>
Or I can provide much more detail on request.

>I'm in favor of having ISPs be able to limit congestion in a 
>reasonable way, but I'm interested in approaches that scale. I'm 
>also concerned about what looks to me like a persistent under-run. 
>If Alice sends Bob a stream of traffic, Bob replies noting her mark 
>count, and she re-replies re-inserting that mark count, there is 
>always some probability that her re-inserted reply is itself marked 
>and Bob no longer cares. If the network is drawing broad conclusions 
>from aggregated marks, it seems it would need to account for the under
>  -run.

Ah, this is probably a misconception because we often describe re-ECN 
as simply reinsertion of black packets in response to feedback 
notifying red packets. Then, yes, black would lag red.

But one level of detail further down in the re-ecn-tcp I-D, the 
source has to keep the balance positive by sending sufficient green 
packets from the start. Green and black count the same in the 
network, but they can be distinguished if you need to separate actual 
congestion from just being cautious. This is a subtle, but crucial 
part of the protocol (IMHO), which allows the anti-cheating 
mechanisms to be much simpler.

For TCP, we've calculated the SYN, and the first & third data packets 
should be green, altho Mirja has pointed out an error in our maths on 
this, so we have to correct that.

>Another way in which I scratch my head is what one actually does. If 
>we're OK with maintaining per-address or per-prefix state of some 
>sort, I would find myself wondering about a variation on fair 
>queuing that uses the later of "a time or sequence number dictated 
>by the source address" and "a time or sequence number dictated by 
>the destination address", and only keeps track of addresses/prefixes 
>that have been recently used.
>
>That allows the ISP to be in control of its own congestion without 
>the compliance of neighboring networks or the hosts in them. But 
>that's just me.

Don't understand. Do you want to say more?

BTW, we designed re-ECN so that you don't have to trust that the 
source address has any meaning - it can just be used as an opaque 
label, not a locator. So a receiving network doesn't have to care 
about source addresses in upstream networks - it's independent of 
whether mechanisms like src addr validation (SAVI) are widely 
deployed or not. And it's OK with NATs & tunnels.

Bob

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 18 May 2010 14:33:49 +0200

Hi,

> I think this is too narrow a scope. I think the document should describe
Having see all the discussion on the mailing list about use cases, I have to 
say yes and no to the question if there should be uses cases mentioned in the 
charter. 
I think the use case of providing an incentive for endsystem to do appropriate 
congestion control is an important one. There is already mentioned something 
in the charter 

'Today, the network signals congestion by ECN marking or dropping packets, the 
receiver passes this information back to the sender in transport layer 
acknowledgements, where the sender makes an adjustment to its window size or 
data rate as appropriate.'

but with conex there is actually a strong reason for the sender to really 
adjust something. Whereelse with ECN-only the sender might be better off to 
not reducing its sending rate and wait for a stronger sight of congestion 
like a drop when at the same time someone else reduces its sending rate 
already with the ECN marking.

Such an incentive can be introduced by an policier at network ingress or 
something else... the actually realization in the network might be out of 
scope for the charter but the mechanism of motivating the sender to actually 
do appropriate congestion control (which is not given with ECN-only) should 
be mentioned more explicit IMHO.

I would hope having this use case in mind makes one benefit of conex more 
clear even though there are more use cases especially when regarding the 
network components using this information. But at least for me the 
appropriate congestion control in the endsystem is the focus (in  the first 
step).

> Consider RFC 3168 as a prototype for this document. It identifies that the
> network can easily and cheaply flag congestion events (far more cheaply
> than with IPFIX, and far more accurately), but it also identifies that the
> responsibility for responding to the event lies in the transport layer. If
> you had put this requirement on the authors of RFC 3168, they would not
> have been able to describe what is in fact a very useful capability. (IMHO,
> either RFC 3168 or a delay-based congestion avoidance procedure in the
> transport would be very sufficient for ledbat, for example).
>
> I don't think you actually need a WG for this document. What you need is
> for individuals that have use cases to post drafts describing them. If
> conex wants to pull those together into a use case document, they can do
> that at their leisure.
I don't see anything in the charter that prohibits to put together such a 
document that collect all the uses cases within the w-g (e.g. starting with 
the ones mentioned on the mailing list already). All those ideas might/will 
influence the protocol design. But it might not be possible to fullfill all 
requirements of all use cases so its good to have one major uses case in mind 
which should be described in the charter.

Btw. shouldn't the current charter be on the following page...?
http://trac.tools.ietf.org/area/tsv/trac/wiki/re-ECN

And I guess the URL should be change to 'conex' instead of 're-ECN' at a 
time...?

Mirja

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 18 May 2010 13:48:10 +0100

Mirje,

>And I guess the URL should be change to 'conex' instead of 're-ECN' at a

Some time ago, Lars decided to keep the mailing list (and by 
extension this WiKi page) with the current name until a w-g is 
chartered - just in case the IESG wanted a different name from ConEx 
for the w-g.

However, I doubt he expected the IESG would still be deciding...

Bob

---

From: "Toby Moncaster" <[email-masked]>
Date: Tue, 18 May 2010 14:22:58 +0100

Hi All,

After four hectic months including a change of career, I have decided to get
involved again in the discussions on this list. In particular I wanted to
follow up on this email regarding use cases in the charter. Please accept my
apologies if I am stating the bleeding obvious or repeating something people
have already said, it is going to take me a bit of time to get back up to
speed with all this! I also fear I may be rambling a bit...!

I think we do need to tread a very careful line with the proposed charter.
There are 3 things that it has to achieve:

1) It must be limited enough to give real results quickly (we don't want to
spend years of effort only to end up with a protocol that is superseded
before it is even finalised).

2) It must be open enough to allow a variety of eventual uses for the
resulting protocol. In particular, whilst the charter may well put
significant areas "off limits" it must be careful not to close them off from
future work (ie "x is out of scope for this document" is OK, but "x must
never be done" is not OK).

3) It must be "inclusive". We know (all too well) that there are many
opinions for and against the proposed work. Even those in favour of it
disagree about quite fundamental things. The charter has to acknowledge
this, and has to make it clear that this won't act as a complete block to
progress (luckily the IETF doesn't operate a veto system else we would
really be stymied!). 

Bearing those 3 things in mind we can look at the case for putting use cases
in the charter. My own opinion is that we should be careful about use cases.
Yes, we need to identify that there are several useful use cases in order to
justify putting so much effort into this protocol. However we must avoid the
temptation to get locked into a detailed technical discussion about the
final solution. The IETF guidelines are that the working group should decide
that. At chartering stage it should be sufficient to identify that there are
genuine interesting problems and that there is a genuine potential solution
(both of which have been done repeatedly). 

Having said that, I think there is potentially some benefit in publishing a
(probably very messy) ID gathering together ALL the possible use cases
people have discussed on- and off-list. This wouldn't be part of any formal
process, and it wouldn't matter if some of the cases were contradictory. It
would just give a single point of reference for all these discussions...

Some inline {TM}...

Toby

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Mirja Kuehlewind
Sent: 18 May 2010 13:34
To: [email-masked]
Subject: [re-ECN] use cases in charter? Was: Re: Two questions about CONEX

Hi,

say yes and no to the question if there should be uses cases mentioned in
the 
charter. 
I think the use case of providing an incentive for endsystem to do
appropriate 
congestion control is an important one. There is already mentioned something

in the charter 

'Today, the network signals congestion by ECN marking or dropping packets,
the 
receiver passes this information back to the sender in transport layer 
acknowledgements, where the sender makes an adjustment to its window size or

{TM} Well, there is an incentive if there is congestion policing. What conex
actually does is add a mechanism to do something that can't be done at the
moment, namely let the network know how well behaved you are. Currently many
(most?) ISPs assume their customers are being selfish and misbehaving, conex
allows the honest majority to show that they are in fact honest...

{TM} We have to be a little cautious there. Yes, end-system CC is important,
but it is kind of out of immediate scope for conex. The IETF already has
lots of guidance on what end-systems must do in response to congestion.
Frankly, if it doesn't suit them to follow the guidance, application writers
just ignore it! What conex does is delve deeper than that. It adds the
concept of accountability to things. Then the motivation to behave yourself
is created by the risk of being caught misbehaving. Accountability is a
cornerstone of finance, and (with a few sad exceptions) it keeps things
working as they should do.

> responsibility for responding to the event lies in the transport layer. If

{TM} But PLEASE can we make sure our documents are more clearly written. One
of the weaknesses of RFC3168 is that, because it is relevant to both TCP and
IP it is very hard to cleanly separate out all the different requirements
contained within it...

requirements of all use cases so its good to have one major uses case in
mind 

{TM} I would actually argue that such a document allows us to then
rationally compare different conex proposals to show how good they are at
being broadly applicable to a range of problems. I do still think we mustn't
limit the charter to a single use-case. Perhaps the solution is to have an
initial use case (or preferably a few initial use cases) in the charter but
make it absolutely clear that these are a just starting point (sort of like
the original PCN charter limitation to only examine cases where trust is
assumed).

Mirja

tel: +49(0)711/685-67973