Subject: Re: [re-ECN] Charter Question

From: "Woundy, Richard" <[email-masked]>
Date: Fri, 7 May 2010 20:19:57 -0400

Bill,

Maybe I can give one specific example of how a router might use the conex bit to do something lightweight and simple. This is specific to my type of network, and may not work in John's network or other networks -- it is just one way some of us have envisioned Conex to make things better than today.

In my cable network, I have a broadband access router called the CMTS. It forwards traffic between the cable (DOCSIS) network and the regional access network. It counts transmitted and received bytes per cable modem and per interface, and it can assign priorities to cable modems.

We have implemented a congestion management mechanism on top of this infrastructure, which is protocol- and application agnostic. See <http://tools.ietf.org/id/draft-livingood-woundy-congestion-mgmt-03.txt> for details. In short, the CMTS reports its congestion state to a backend system, and reports on the subscribers contributing to that congestion (byte counts per modem and per interface). The backend system then changes the relative priority of modems to balance the traffic among subscribers during brief times of congestion.

What's good about this system is that it seems to work, and is not targeted against a specific protocol/application mix that might change over time. Plus it avoids many net neutrality-related concerns raised against other congestion management schemes.

One open issue is that this scheme cannot tell the difference between LEDBAT-enabled applications (eg uTorrent) and non-LEDBAT applications. A LEDBAT application might generate a lot of traffic in some short period of time, but it will back off when it senses congestion in the network (based on measuring the change in end-to-end delay). This is bad for both parties, because the ISP has a reduced ability to provide incentives to LEDBAT applications.

So now let's add Conex marking, in which the end hosts add markings to send feedback about end-to-end congestion into the network. Now the CMTS has a new data point -- information about whether there is upstream (ECN) or downstream (Conex) congestion experienced/expected for a particular packet flow. The CMTS can count 'total bytes' as well as 'congestion bytes' on a per modem basis. As a result, the backend system can make a much better determination about whether a subscriber's overall traffic is dominated by LEDBAT or non-LEDBAT applications, without resorting to DPI and other such schemes. And the CMTS only maintains another set of packet counters, without disrupting its primary job of packet forwarding.

We haven't worked out all the details of this use case, of course, but is that enough detail to answer your basic questions?

-- Rich

----- Original Message -----
From: [email-masked] <[email-masked]>
To: [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>
Sent: Fri May 07 17:56:24 2010
Subject: Re: [re-ECN] Charter Question

[email-masked] <[email-masked]> wrote:
> On Fri, May 07, 2010 at 04:02:35PM -0400, John Leslie wrote:
>> 
>> I don't believe any of us are talking a "congested/notcongested"
>> binary switch, and I'm certainly not.
> 
> well, we -could- be, if this is truely hop/hop - then the
> only link you care about is the link betwn the two hops.

   IMHO, even if this ends up in an IPv6 "hop-by-hop" option, it won't
concern the link between two hops. This is about _exposing_ information
now at the transport layer _to_ the forwarding layer.

>> I agree with Bill that the binary switch he mentions isn't really
>> applicable to the problem, though I don't agree the "inspecting
>> forwarder" is a likely cause of congestion.
> 
> perhaps I am jaded, but for many years there were real problems
> with forwarders being able to operate at "line-rates".

   I remember that too...

> If the forwarder spends too much time "inspecting, detecting,
> neglecting, infecting" the datagrams - it could be considered a point
> of congestion.. 
> Or I am misunderstanding (again)

   It is, of course, always possible to create yet-another-point-of-
congestion. :^(

   I would, however, view it as an abuse for any forwarder to spend
much time evaluating ConEx marking.

   There is an unstated presumption that "true backbone" links will
have sufficient aggregation to have negligible congestion except
during extraordinary DoS conditions. Thus, there would be no point
in putting congestion-policing at these. If their design allows, we'd
like them to ECN mark (in the rare case); but dropping packets during
congestion is perfectly fine.

>> I'm a bit confused by Bill's "how is it different" question: to
>> me "a range" is totally different from QoS...
>> 
>> Nevertheless, congestion-expected marking is a fundamentally
>> different thing than QoS marking, and rather orthogonal IMHO. QoS
>> deals with what _kind_ of special treatment you request; CE marking
>> deals with passing information about an estimate of future
>> congestion the packet might encounter.
> 
> qos is a range,

   Not, IMHO, to any forwarder that actually acts on it...

> ecn is (for this thread) a range.

   Yes.

> based on other parts in this thread, ecn markers could be
> used to tag/flag which flows needed priority - which sounded
> like QOS to me.

   I don't believe any of us intend ConEx marking to indicate a
"need" for priority; nor do I believe there is any current proposal
calling for giving such packets priority (though I personally don't
want to rule out such things in some future charter -- if it might
lead to better congestion management).

> CE marking sounds more like a warning to the destination that the
> path taken by the source should be avoided (congested) if possible.

   The reECN proposal calls for CE marks in just a few packets of a
flow. Treating those packets any differently than non-CE-marked
packets seems counter-productive.

   (There is, of course, ongoing work in MPTCP; and I don't mean to
try to forbid a case where MPTCP pays attention to CE markings.
But IMHO there doesn't seem to be any _reason_ for them to do so.)

>> Thus, IMHO, it's quite reasonable to imagine QoS marking in
>> addition to CE marking, to designate what you want a forwarder to
>> do in the event of lengthening queues, for example (though I don't
>> mean to say the two markings _should_ overlap in the slightest!).
>> Or it might be the case that a forwarder could benefit from CE
>> markings to estimate whether a particular QoS path is becoming
>> congested... Any such work goes _far_ beyond our initial charter.
> 
> i'm ok with this idea.  what would be useful here is some
> guidance from the IAB (the architecture people) about the 
> 10'000m view.

   Hmm... I'm willing to push for that, but the IAB appears to have
a pretty full plate of liaison issues, and I doubt we'd get anything
from them in less than 3-6 months.

> granted we may be working on a small piece, but it would be nice to
> know that when we start mass hallucination that we are either
> ontrack or way out of scope. e.g. I'd like to see a larger context
> in which this work would be a contributing part.

   I think we'd have to start that discussion here, and hand IAB
members much more specific questions to see whether they bite...

>> I believe it is our intent to specify the meaning of CE marks,
>> expecting nothing of intermediate forwarders than that they signal
>> congestion experienced, whether by ECN, dropping packets, or some
>> other method entirely.
> 
> ok.  hop/hop or end/end?

   End-to-end, IMHO. We're talking about making transport-layer
information visible at a lower layer -- the transport-layer info
is strictly end-to-end.

> can intermediate nodes modify CE marking?

   I suppose they "can", but it doesn't seem helpful to do so.

--
John Leslie <[email-masked]>
_______________________________________________
re-ECN mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/re-ecn

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 08 May 2010 06:38:59 +0100

On 08/05/2010 01:19, Woundy, Richard wrote:
> We have implemented a congestion management mechanism on top of this infrastructure, which is protocol- and application agnostic. See<http://tools.ietf.org/id/draft-livingood-woundy-congestion-mgmt-03.txt>  for details. In short, the CMTS reports its congestion state to a backend system, and reports on the subscribers contributing to that congestion (byte counts per modem and per interface). The backend system then changes the relative priority of modems to balance the traffic among subscribers during brief times of congestion.
>
This sounds like a relatively slow process - i.e. over a few packets the 
host tell anyone who wishes to listen that it is seeing congestion on a 
flow. That sounds like it could be addressed by having the host send an 
OAM packet from time to time over the path rather than including the 
information in every packet.

Stewart

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 17 May 2010 11:56:04 +0100

Stewart,

At 06:38 08/05/2010, Stewart Bryant wrote:
>>So now let's add Conex marking, in which the end hosts add markings 
>>to send feedback about end-to-end congestion into the network. Now 
>>the CMTS has a new data point -- information about whether there is 
>>upstream (ECN) or downstream (Conex) congestion 
>>experienced/expected for a particular packet flow. The CMTS can 
>>count 'total bytes' as well as 'congestion bytes' on a per modem 
>>basis. As a result, the backend system can make a much better 
>>determination about whether a subscriber's overall traffic is 
>>dominated by LEDBAT or non-LEDBAT applications, without resorting 
>>to DPI and other such schemes. And the CMTS only maintains another 
>>set of packet counters, without disrupting its primary job of 
>>packet forwarding.
>>
>>-- Rich
>This sounds like a relatively slow process - i.e. over a few packets 
>the host tell anyone who wishes to listen that it is seeing 
>congestion on a flow. That sounds like it could be addressed by 
>having the host send an OAM packet from time to time over the path 
>rather than including the information in every packet.
>
>Stewart

An OAM based solution would be in scope as a solution to the ConEx 
problem statement. However, we would need someone to write up a 
protocol spec in detail, including inter-domain, then prove it is 
resistant to cheating, etc etc. If this happens (unlikely) the w-g 
can choose between candidate solutions.

Anyway, granular isn't necessarily bad. For instance, operators 
generally use data volume on slow timescales, but a volume counter 
still has to accumulate packet by packet. Congestion-volume is no 
different - except you just count the volume of packets that are 
congestion marked.

We should be open to any solution, whether granular or coarse, as 
long as it solves the problem ConEx aims to address: that network 
operators currently have no reliable way (fast or slow) to measure 
which users contribute how much to congestion - at least not in cases 
where the congestion is occurring at a different point to where it is 
being measured and controlled - whether the two points are within 
your own network, as in Rich's example, or in different networks.

Bob

PS. sorry for late reply.

PPS. The only feasible OAM-based system I am aware of is Katerina 
Argyraki's (ref  below). I'm sure the IESG would not want this level 
of complexity in the Internet. Also it only solves one side of 
congestion accountability: network to users, whereas re-ECN aims to 
make networks accountable to users *and* users accountable to networks.

Argyraki, K., Maniatis, P., Irzak, O., Ashish, S. & Shenker, S., 
"Loss and Delay Accountability for the Internet," In: Proc. IEEE 
International Conference on Network Protocols IEEE (October 
2007)  URL: http://www-dsg.stanford.edu/papers/astra/astra-hotnets04-paper.pdf

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: "David Harrington" <[email-masked]>
Date: Mon, 17 May 2010 08:32:34 -0400

Hi,

Do we already have protocols that could help solve this problem
intra-domain, such as ipfix and psamp?

One of the concerns that has been raised is whether domains will be
willing to share this potentially-sensitive information across
domains.
As a vendor, if I supported a re-ecn, I would think I would be
motivated to allow the operator to decide not to share the info, and I
would implement a scrubbing mechanism (allow border routers to clear
the conex markers in traffic exiting a domain). The IETF cannot
dictate that deployments reveal this sensitive information, any more
than it could dictate that SNMP traps must be allowed cross-domain.

So maybe we should approach the problem in two steps:

1) determining how well existing standard protocols can be used to
effectively address the intra-domain congestion problem, and whether
specific enhancements, such as congestion-specific ipfix IEs, are
needed. This would give us forward progress on standard approaches for
monitoring congestion that might be deployed fairly quickly.

2) how to share information across domains. Since the major question
here seems to be whether domains would be motivated to share, maybe an
effort akin to INCH is needed for this piece. After we solve the first
problem, we can try to extend it to cross-domain, or use a different
mechanism for cross-domain reporting.

dbh

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Bob Briscoe
> Sent: Monday, May 17, 2010 6:56 AM
> To: [email-masked]
> Cc: [email-masked]; Woundy,Richard; [email-masked]

> 2007)  URL: 
> http://www-dsg.stanford.edu/papers/astra/astra-hotnets04-paper.pdf
> 
> 
>

---

From: John Leslie <[email-masked]>
Date: Mon, 17 May 2010 10:37:17 -0400

> willing to share this potentially-sensitive information across

   I could pose that question to NH ISPs -- but I suspect the response
would be, "Huh? Whazzat?"

   And I'm not sure I could answer!

   The only thing which seems to fit "potentially-sensitive" is what
congestion there is within your domain. And that would be downright
hard to extract from ConEx markings.

   ConEx markings will go with end-to-end flows. A sender will signal
expected congestion, which may or may not be experienced _anywhere_
along the path. There is a hope that ConEx-marked packets will get
ECN markings instead of being dropped, but that's wishful thinking.

   The ECN markings are where information on congestion lives. And
they indicate congestion _anywhere_ on the path so far. You'd have
to sample flows _before_ they enter a domain and _after_ they leave
to get useful "potentially-sensitive" information about congestion.
There are easier and better ways to derive this information.

   For the ConEx marking itself, the information it carries is about
the sender's expectations. The sender might or might not be in your
domain. If it is, you're welcome to scrub any ConEx marks you don't
like. If it's not, what could possibly be "potentially-sensitive"?

> As a vendor, if I supported a re-ecn,

   Wait a second... what might it mean for a "vendor" to "support"
ConEx?

   The first example is an Operating-System vendor. An OS "supports"
ConEx by putting ConEx markings in IP packets it originates --
otherwise the markings can't get there. It is likely that an OS
vendor would automate what ConEx markings based upon congestion
feedback, within its TCP/IP stack.

   The next example is a middlebox vendor. A middlebox might perform
accounting and/or policing for congestion-expected packet volume.

   IMHO, a backbone-router vendor is _not_ an example. Backbone
routers either do or don't support ECN marking in place of dropping
packets. (A end-customer-router might well perform middlebox tasks,
but I prefer to group it with other middleboxes.)

> I would think I would be motivated to allow the operator to decide
> not to share the info, and I would implement a scrubbing mechanism
> (allow border routers to clear the conex markers in traffic exiting
> a domain).

   Scrubbing ConEx congestion-expected markings seems pointless.
They show _nothing_ about your domain unless somebody can be bothered
to identify which packets originated in your domain; and even then
they show the varying expectations of your customers, not what
happened to the packets while you were forwarding them.

   Scrubbing ECN markings _might_ hide something about your fabric,
but it seems really far-fetched to hide what little they show. If
you did scrub them, that wouldn't prevent ConEx from working: it
would merely slow down the reaction to congestion, inevitably
leading to _more_ congestion somewhere.

   (Clearly, an OS vendor or a middlebox vendor might have good
reason to program an option to disable, or even scrub, ConEx markings.
But it doesn't look as if that's your concern.)

> The IETF cannot dictate that deployments reveal this sensitive
> information, any more than it could dictate that SNMP traps must be
> allowed cross-domain.

   The IETF cannot "dictate" much of anything; but please explain
what "sensitive information" you're concerned about if I've missed
your point.

>    effectively address the intra-domain congestion problem,

   I don't believe ConEx is intended to "address" any "intra-domain
congestion problem". It would produce a tool which might later be
useful for such problems; but that is not the problem we set out
to solve.

>    and whether specific enhancements, such as congestion-specific
>    ipfix IEs, are needed. This would give us forward progress on
>    standard approaches for monitoring congestion that might be
>    deployed fairly quickly.

   This may well be work worth doing, but it is not ConEx-related
work. ConEx does not intend to "monitor" congestion. It means to
encode total-path expected-congestion, allowing ISPs to discern
which end-users are demanding services on congested paths.

> 2) how to share information across domains. Since the major
>    question here seems to be whether domains would be motivated
>    to share, maybe an effort akin to INCH is needed for this piece.

   Again, that may be work worth doing. It doesn't fit well with
the intended ConEx work, though; and I suspect we wouldn't be
happy if you tried to squeeze it into our charter.

> After we solve the first problem, we can try to extend it to
> cross-domain, or use a different mechanism for cross-domain
> reporting.

   I think you need an entirely different BoF for this problem
you are describing.

---

From: "McCann Peter-A001034" <[email-masked]>
Date: Mon, 17 May 2010 11:33:48 -0400

> but I prefer to group it with other middleboxes.)   

It would still be desirable to implement the preferential dropping
scheme on congested links in the backbone, no?  Or are you assuming
that with universal deployment of ECN, we would never get the congestion
in the backbone that would necessitate dropping?

-Pete

---

From: John Leslie <[email-masked]>
Date: Mon, 17 May 2010 12:21:10 -0400

McCann Peter-A001034 <[email-masked]> wrote:
> John Leslie wrote:
>> 
>> Backbone routers either do or don't support ECN marking in place of
>> dropping packets.
> 
> scheme on congested links in the backbone, no?

   This probably deserves some discussion; and I see no reason we
can't discuss it before ConEx is chartered.

   I'm not sure whether preferential dropping belongs in ConEx at all.
It seems like an ECN thing to me.

   The benefit of ECN marking instead of dropping is that a congestion
signal can trigger a slowing of sending rate more quickly if we don't
have to wait for a packet drop to be recognized.

   The downside is that we can't be sure that an ECN mark will actually
make it through and result in a backoff of sending rate. A misguided
operator might scrub ECN markings; an ECN-marked packet might encounter
_other_ congestion and be dropped; or the signal could fail to make it
back to the sender for a number of reasons.

   The issue is whether anything about ConEx markings could signal a
trustworthy assurance that ECN marking is more likely to result in
earlier backoff.

   This doesn't feel particularly in-charter to me (at least not in
the initial charter); but we have until Thursday at least to discuss
things that are _not_ in-charter.

   ;^)

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 17 May 2010 17:30:36 +0100

David,

Allow me to talk in terms of the only concrete proposal we have (re-ECN):

1/ Whole path congestion marking

Re-ECN starts with the end systems telling all networks on the path 
what the e2e congestion is. They use an e2e marking that the 
end-systems can cover with e2e authentication. If a network scrubbed 
re-ECN markings, it would/could break e2e authentication - probably 
worse than sending TCP resets ;)

Just re-ECN on its own, without ECN, has a number of interesting 
uses. I didn't think much about re-ECN with drop (not ECN) when I 
designed re-ECN, but Matt Mathis convinced me.

The incentive for senders to declare re-ECN markings is strongest for 
the large majority of systems that are *not* the main contributors to 
congestion. They are saying "It's not me". OS and/or app developers 
have an incentive to implement this for the majority of their 
customers. E.g. BitTorrent has an incentive to make LEDBAT traffic 
prove that "It's not me".

2/ ECN marking

Your question is, however, most relevant to whether an operator will 
turn on ECN, because that's the protocol that reveals whether the 
congestion is upstream or downstream of a point, when combined with 
ConEx info. Yes, of course an operator doesn't have to turn on ECN; 
so why would it?...

Once ConEx is deployed in a major OS or app, there is an incentive 
for well-provisioned operators (the majority) to turn on ECN, again 
to say "It's not me"... Here's why:

Imagine e2e ConEx markings are revealing a high level of congestion 
in the traffic passing between networks A&B - somewhere on their e2e 
paths in A or B. If you are network A and your network is not to 
blame, you can prove this to B by turning on ECN. Alternatively, if 
network B is not to blame, it can accuse A of being responsible for 
the whole path congestion, unless A turns on ECN to prove how much of 
the congestion is in B.

3/ Congestion as a business secret?

Networks already cannot help but reveal to end-systems that there is 
congestion somewhere on the path. ConEx allows end-systems to reveal 
this info back to networks. That would be a result for the IETF if it 
took off, even if ECN was still not deployed.

I've given plausible scenarios for how ConEx then ECN might get 
deployed. But ultimately I cannot say whether or not networks that 
have something to hide will predominate. If so, we might get ConEx 
but not ECN. If not, we might get both ConEx and ECN.

But this issue is beyond the IESG's remit to judge - it should surely 
"let the market decide" on this point. Let's get moving.

Bob

---

From: "McCann Peter-A001034" <[email-masked]>
Date: Mon, 17 May 2010 12:39:17 -0400

> It seems like an ECN thing to me.

My understanding of the re-ECN proposal is that any congested
link, anywhere along the path, should drop negative or neutral
packets before dropping any re-ECN marked packets.  This seems
to require some sort of "re-ECN support" on even backbone routers.
 
>    The benefit of ECN marking instead of dropping is that a
> congestion signal can trigger a slowing of sending rate more quickly
> if we don't have to wait for a packet drop to be recognized.  

Right, which might be one argument in favor of not needing to
implement the preferential drop mechanism in backbone routers.
 
>    The downside is that we can't be sure that an ECN mark will
> actually make it through and result in a backoff of sending rate. A
> misguided operator might scrub ECN markings; an ECN-marked packet
> might encounter _other_ congestion and be dropped; or the signal
> could fail to make it back to the sender for a number of reasons.    

All signs of a broken network, IMHO.
 
> things that are _not_ in-charter.  

Well, this question might be a factor in deciding whether to pursue
a per-packet marking or an out-of-band signaling (OAM) type solution.
If
only the first and last hop ISPs need to understand ConEx to implement
the ingress policing and the egress dropping, then there seems to be
some hope that out-of-band signaling might work.  On the other hand,
if the backbone routers need to see the ConEx markings, then they would
either need to keep per-flow state in the out-of-band signaling case
or there would need to be some node aggregating the signaling, similar
to what has been tried for RSVP deployments.  It could be a mess.

-Pete

---

From: John Leslie <[email-masked]>
Date: Mon, 17 May 2010 13:37:41 -0400

> to require some sort of "re-ECN support" on even backbone routers.

   It's not _my_ understanding that any such support on backbone
routers is expected, least of all "required"

   Perhaps Bob or Matt could correct me if I misunderstand...

> a per-packet marking or an out-of-band signaling (OAM) type solution.

   While I can agree that an Out-of-Band mechanism would be easier
to "secure" and thus perhaps be more "trustworthy" for securing
preferential dropping, that's really quite a different problem than
ConEx set out to solve.

   Furthermore, it doesn't seem sufficiently "scalable" to me. How
would we manage a Key-Distribution-System to all the routers that
may need to decide whether to preferentially-drop certain packets?

> If only the first and last hop ISPs need to understand ConEx to
> implement the ingress policing and the egress dropping, then there
> seems to be some hope that out-of-band signaling might work.

   Indeed, there is "some hope" it could work. But the scalability
and key-distribution issues make me very hesitant to pursue it.
(If nothing else, keeping flow-state synchronized between ends feels
pretty daunting to me.)

> On the other hand, if the backbone routers need to see the ConEx
> markings, then they would either need to keep per-flow state in
> the out-of-band signaling case or there would need to be some node
> aggregating the signaling, similar to what has been tried for RSVP
> deployments.  It could be a mess.

   I am willing to discuss such issues, either before or after
chartering. But I think it most unwise to have a charter that
requires us to design an out-of-band system.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 17 May 2010 18:41:06 +0100

John,

>can't discuss it before ConEx is chartered.

Yes, let's have some technical discussion for a change, on the 
understanding pref drop is not a candidate for the initial charter. 
Those who want charter discussions can tune out.

>It seems like an ECN thing to me.

A queue can do some cool OPTIONAL protection against DDoS using 
preferential drop based on ConEx (specifically re-ECN) markings. This 
might be in a server's IP stack, or in a router. I would expect more 
relevant for access routers, assuming core routers might better 
absorb flooding with raw power.

Here's the intuition. A queue under a flooding attack will drop lots 
of packets and nearly all ECN-capable packets that manage to get 
through will be marked CE (red). So a behaving re-ECN sender should 
have marked nearly all arriving traffic for ~100% expected congestion 
(~100% black). Therefore, a flooded queue can focus drop on any 
packets that claim to support re-ECN but are not black. And it can 
further focus drop on any packets that don't support re-ECN.

A network using re-ECN for traffic management will police heavy 
sources of black packets at every entrance to the network. So, a DDoS 
source is caught between a rock and a hard place. To avoid focused 
drop at the congested queue it has to colour most packets black. But 
to avoid drop at the ingress policer it has to not colour packets 
black. Either way, its packets generally get dropped before harming 
packets that are behaving.

See the second bullet "Preferential Drop" in S.5.3 of the re-ECN draft:
<http://www.bobbriscoe.net/projects/refb/draft-briscoe-tsvwg-re-ecn-tcp-08.html#retcp_Router_Forwarding_Behaviour>
[Yes, I know, it's expired - now I'm back in circulation I'll rev it.]

The analysis I did on this showed that re-ECN can raise the bar 
against DDoS so that the army has to be N times bigger to achieve the 
same *sustained* attack force, where N=1/p and p is the 'normal' 
level of congestion in the network. E.g. if normal congestion is 
0.001%, a sustained attack against a re-ECN internetwork has to be 
100,000 times bigger to cause the same damage as it could against a 
network without re-ECN.

Obviously, there are assumptions and details, but that was the 
take-home message. If anyone wants more details, just ask.

>_other_ congestion and be dropped;

SUch a drop will still be an (eventual) congestion signal

>or the signal could fail to make it

So far I'm not convinced by preferential drop based on ECN markings.

>things that are _not_ in-charter.

I also believe strongly that pref drop is off-charter, at least initially.

Incidentally, I've also worked out that it is perfectly possible to 
add the above preferential drop behaviour to the specs later, without 
creating an incremental deployment problem.

Bob

---

From: "McCann Peter-A001034" <[email-masked]>
Date: Mon, 17 May 2010 16:11:23 -0400

Hi, Bob,

Thanks for the explanation.

> take-home message. If anyone wants more details, just ask.

I'm curious about your definition of "damage".  Was it just "causing
a given loss probability on a particular target link"?  It seems like
there are other costs in a DDoS attack on a re-ECN enabled network, such
as causing well-behaving users to consume their black token allotment,
making them less able to respond to congestion for some time after the
attack.  But then, I guess the philosophy of re-ECN is that if you
experience
congestion, you are also causing it, so those well-behaving users that
marked near 100% should also be "held responsible" for the DoS attack...

My assumption was that this dropping should happen at any congested
link, not just the egress, for maximum effectiveness.  (Btw, is this
different from the behavior you propose for an "egress dropper"?  That
wasn't clear to me from the documentation.)  

How often do you think this technique would need to be invoked on a core

Internet router?  Do you agree that an out-of-band per-flow signaling
solution
would have difficulty supporting this feature, if the core routers
needed to implement it?

(I agree this is all out-of-scope of the charter discussion and that
preferential dropping shouldn't be in the initial charter anyway.  I
am just trying to understand whether this feature would be a motivation
for per-packet as opposed to out-of-band signaling solutions).

-Pete

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 20 May 2010 18:20:51 +0100

Pete,

Still off the topic of chartering...

At 21:11 17/05/2010, McCann Peter-A001034 wrote:
>a given loss probability on a particular target link"?

Correct

>It seems like

Correct. Or slowing their bit-rate to consume 
their token allotment more slowly.

>attack.

I think you mean "less able to contribute to congestion..."

>But then, I guess the philosophy of re-ECN is that if you
>experience

Indeed.

I have pointed out before that attacking during a 
flash crowd is the most cost-effective strategy 
an attacker can adopt — then the infrastructure 
is most valuable to most genuine users, and it 
takes least extra malicious effort to tip it into 
overload. See <http://www.bobbriscoe.net/projects/refb/#refb_dplinc>

Imagine three types of users in such a situation:
1) a 'genuine' member of the flash crowd who slows in response to congestion
2) a 'genuine' member of the flash crowd who is unresponsive to congestion
3) an attacker who is unresponsive to congestion

IMHO it is not unreasonable for the system to 
throttle down 2 & 3 the same, at least by 
default. They may have different intent 
(selfishness as opposed to malice), but they have 
the same outward effect on everyone else.

>wasn't clear to me from the documentation.)

Sorry; I'll be clearer: This OPTIONAL 
preferential dropping could be at any IP-aware 
link vulnerable to DDoS. It's not needed at all 
for re-ECN to work. It's not proposed for initial 
chartering (this thread is for light techie 
relief). Nonetheless, if a network is using 
re-ECN already for other reasons, and a queue 
under attack has to drop stuff, re-ECN is 
designed so that a router can easily distinguish 
stuff not obeying the protocol and drop it first.

The 'egress dropper' proposed in 
draft-briscoe-re-ecn-tcp-motivation is very 
different. It uses drop to punish cheating (not 
because it's overloaded itself). This egress 
dropper (or something like it) is needed to 
asssure the integrity of re-ECN markings against 
cheaters. It's only needed at the egress of the 
internetwork, not at every link. A simplifed 
variant could be useful at an interior egress 
(border), but may not be necessary.

>How often do you think this technique would need 
>to be invoked on a core Internet router?

You might be able to protect core routers from 
DDoS by good network design (e.g. ECMP and 
over-provisioning). But if this still isn't 
enough, preferential drop based on re-ECN might 
be the lightest possible *self-protection* a core 
router could adopt against DDoS. It doesn't rely 
on the addresses in packets so we claim it is 
robust against SYN attacks and spoofed source 
addresses (unlike the scrubbing solutions used 
today, which rely on attacks not bothering to obfuscate themselves very well).

>Do you agree that an out-of-band per-flow signaling
>solution

I can't imagine how an OOB solution would allow 
the data plane to discriminate attack traffic, but...

<RANT> <!--not against you-->
We cannot know whether an OOB per-flow signalling 
solution support any feature, because it's currently vapourware.

People have proposed systems of echo-reflectors 
to check the QoS performance of networks. None 
I've seen have been designed to be robust against 
even simple cheating strategies. Katarina 
Argyraki's Loss & Delay accountability idea is 
the only example I know designed to be robust 
(see my posting to Stewart Bryant on 17-May). But 
all these things only address the easy half of 
the ConEx problem: holding networks accountable 
for too little capacity, not also holding users 
accountable for too much traffic. Despite this, 
they are already highly complicated beasts.

It took literally years to solve the really hard 
problem of making inter-domain congestion 
exposure robust to self-interest and attacks, 
including DDoS. Then as people came up with some 
really perverse attacks against it, we spent a 
further year working out how to harden it (and 
simplified it in the process). I'm not saying 
it's perfect, but it's pretty damn solid.

Hey, I'm open to someone solving this problem a 
different way. But let's see it before we believe it.

Yes, I'm grateful to the guys from the IESG who 
have opened up with their concerns and engaged in 
debate. But, let's keep this in perspective...

Re-ECN has been accused of being research. I hope 
it's accepted that was a confusion - "an output 
of research" rather than "still needing more research"....

But now we're discussing an OOB solution, without 
even a vaguest whiff of where a design might 
start, let alone any research, evaluation, running code, yada yada.

</RANT>

>for per-packet as opposed to out-of-band signaling solutions).

Understood

Cheers

Bob

---

From: "McCann Peter-A001034" <[email-masked]>
Date: Thu, 27 May 2010 17:12:42 -0400

Hi, Bob,

> Still off the topic of chartering...

Indeed.  Thanks for the discussion, it is helping me wrap my brain
around this problem space.
 
>>> against DDoS so that the army has to be N times bigger to achieve
>>> the same *sustained* attack force, where N=1/p and p is the 'normal'

Based on the statement above and reading the paper you point to below,
it seems you are making an assumption about the token bucket parameters
of the ingress policing and their relationship to the access network
link speed and the link speed of the congested, target link.  If you
take the 'normal' level of congestion as a setpoint for the ingress
policer, your example above seems to imply that the token bucket rate
would be .001% of line rate for a given customer.  As for the depth
of the token bucket, I have no intuition about how to pick a good value.

If we consider operating regions where the token bucket is fast and
deep, the access link speed is high, and the target congested link
is very bandwidth constrained, we could have a situation where the
link remains near 100% congestion for a long period of time or even
in steady state, when senders are marking exactly at their token
bucket rate, even when N isn't inordinately large.  Even with
preferential
drop we might be forced to start discarding packets with positive worth.

Perhaps this is an example of a link that was poorly dimensioned for
the expected load, but that kind of argument seems to be a variant
of the "just overprovision all the links" point of view.

> I think you mean "less able to contribute to congestion..."

Yes, thanks.  I guess the effectiveness of this attack depends on the
relative depth of the token buckets.
 
> IMHO it is not unreasonable for the system to throttle down 2 & 3 the
> same, at least by default. They may have different intent
> (selfishness as opposed to malice), but they have the same outward
> effect on everyone else.   

Regardless of intent, I wonder if a better outcome could be achieved
if the owner of the congested resource made a per-flow admission control
policy decision instead of lumping all the flows together in one
statistical
drop bucket.  Especially for inelastic flows, it would be better to
time-shift in units of whole flows rather than start dropping a fraction

of the traffic across all flows.

The use of Re-ECN marking to resolve contention for a congested link
is kind of like a game of chicken.  If N is sufficiently large, and
all parties are willing and able to mark up to 100% of their flows,
then no one gets out of the way and the system collapses in overload.
Undoubtedly, Re-ECN would make this situation (possibly much) less
likely due to the damping effect of the ingress policers.  However,
it still wouldn't be entirely safe to put a mission critical resource
behind a low-bandwidth link (e.g., a 56kbps modem) exposed to the 
Internet.  That situation still seems to require some sort of security
gateway in front of the link to discard unauthorized traffic, and a
signaling protocol to establish state in that gateway.

> I can't imagine how an OOB solution would allow the data plane to
> discriminate attack traffic, but... 
> 
> We cannot know whether an OOB per-flow signalling solution support
> any feature, because it's currently vapourware. 

Agreed; I am just trying to figure out whether Re-ECN completely
eliminates the need for protocols like RSVP or NSIS (or even non-e2e
IKE, to the extent that IPSec gateways are used to prevent DoS) or 
whether a signaling protocol would still be needed to establish 
per-flow state in middleboxes for the purpose of DoS prevention and
congestion control in general.  The case of a link that is a few
orders of magnitude slower than the average access link still seems
to require this.  Perhaps such a protocol could be tweaked to work
in concert with Re-ECN; for example, you could directly carry the
path congestion metric p in the signaling, maybe eliminating the need
to fight over the evil bit.  This would allow for policing on ingress
as well as auditing on egress.  On the other hand, you would lose the
ability to do inter-domain settlement and preferential dropping without
keeping per-flow state.  You would also have to invent some way to tie
the signaling to the bearer flow (robust against spoofed headers).  It 
might be desirable in some circumstances to interact with your first-hop

policer; e.g., to request a token bucket upgrade in the event of an 
emergency.  If the congestion is near the receiver, then it might be
good to give that receiver some input into which flows get admitted.

Do you see co-existence between Re-ECN and NSIS?  Or is the signaling
just a big can of worms that isn't justified given the lower
effectiveness
of DoS in a Re-ECN enabled network?

-Pete