Subject: [Tsvwg] draft-briscoe-tsvwg-cl-architecture-02.txt

From: ken carlberg <[email-masked]>
Date: Tue, 18 Apr 2006 10:38:28 -0400

Bob (and gang :-),

here are some comments/questions about your draft.

-ken
===================================

Page 4, Figure 1: Overall QoS Architecture.  Would there be a problem  
in extending the target architecture from that shown in Figure 1 to  
one that potentially could include two or more "islands" of CL- 
regions?  The reason I ask is that the CL-region over the internet is  
only as large as one can extend diff-serv to multiple autonomous  
domains, which is difficult at best.  But if one could accomplish  
what you set out to do between two or more leaf ISPs with non-diff- 
serv transit backbones, then it would seem that your design is more  
flexible, and possibly more tractable.

page 5, 3'rd paragraph.  This is definitely just a nit, but could you  
add some text to be more precise when you state that PCN "can  
supplement" any PHB?  The word "can" leaves the reader with a measure  
of doubt, and the possibility that the word "supplement" may involve  
more than just a superset of functionality.

page 6, 4'th paragraph.  When you mention the dependency on  
measurement, I was wondering if relatively short transaction flows  
becomes an issue with your design.  Phone calls today still last just  
~4.5 minutes in duration, while on the other side of the spectrum,  
SMS over telco's obviously are just brief blips.  If we extend these  
to the thousands and on, will a need arise for more than one  
measurement algorithm?  Note, this is just food for thought and  
doesn't really require an answer in your draft.  But this does lead  
to the next point....

page 6, 5'th paragraph.  Here, and in other parts of your draft, you  
state the use of EWMA to dampen and smoothen the measurements of CL  
traffic.  In reality, what you want is a moving average of some type  
-- it could be simple, weighted, non-weighted, etc.  But given that  
this is an architecture document, is it necessary to pin it down to  
one specific type of moving average?  perhaps EWMA could be the  
default, but even then one would eventually need to state the best  
common practice regarding its parameters.

page 7, 1'st paragraph.  In this segment, and throughout the draft,  
you state that interior nodes within the CL-region do not hold  
"state".  Isn't it more accurate to say that they do not hold "per- 
flow state"?  The diff-serv capable interior nodes still hold  
aggregate state, correct?

Small nit.  You use the term "nodes" throughout the draft.  When I  
hear the term node, I tend to view it as an IP capable object that  
could be either a router or a host/server.  And yet in that first  
paragraph of page 7, the "node" you speak of is just a router.  I  
think it would be better if you could use the term routers and hosts  
when it is clear that is what you are speaking about, instead of  
adding ambiguity when using the term "node".

Section 1.1.2 Flow Pre-emption.  When I first came across this  
section, the questions that came to mind were: where does this occur  
-- at the edge, or interior to the CL-region?...and how does it occur  
-- through the use of signaling, or in-band state via DCSP?  These  
questions didn't get (somewhat) answered until a hint is given on the  
last paragraph of Page 24 and Section 7 on Security.  I think you  
need to address these questions in Section 1.1.2

Also, when capacity constraints occur, would it be possible to  
introduce another model besides pre-emption that focuses on  
probability using engineered spare capacity?  Instead of a lengthy  
explanation written here, let me point you to
the following draft:

ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lefaucheur- 
emergency-rsvp-01.txt

and specifically, Section 2.1.3.  If you have further questions on  
the draft or my request, we can discuss that off-line.

Page 8, 1'st paragraph.  Small nit, but the first sentence is unclear.

Page 12, 4'th Item.  The timeline of 1-2 seconds really applies to  
voice.  User expectations for video would not seem to be under the  
same constraints.

Page 13, 1'st Item.  When you speak of "session priorities", are you  
referring to the priorities one may see from signaling protocols like  
SIP and the Resource Priority header?  Or are you referring to the  
session as a flow identified by RSVP?

Page 15, 2'nd Item.  You should either substitute "NSIS" with "GIST",  
or expand the current text to say "signaling protocol produced by the  
NSIS working group".

Page 21, 1'st Item & 1'st para.  You state "packets are pre-emption  
marked".  My question is, how and where does this occur?  Is this a  
specific (potentially new?) DSCP marking?  If so, it would be clearer  
to state this explicitly as opposed to referring to a new behavior.

Section 4.6 Tunneling.  Do you foresee marking the DSCP field in the  
outer packet header?

Section 6.1.  I ended up getting confused by what you are trying to  
say.  It may be helpful to break up the first sentence into 2 or more  
smaller ones.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 25 Apr 2006 13:40:53 +0100

Ken,

Responses embedded to your very helpful, carefully thought through points. 
Many thanks.

But I have a feeling we've made something ambiguous that underlies all your 
questions. Or perhaps you are approaching the draft with some 
pre-conception on how it works that we haven't managed to demolish. I can't 
quite work out what it is, but we're interested in fixing it...

At 15:38 18/04/2006, ken carlberg wrote:
>what you set out to do between two or more leaf ISPs with non-diff- serv 
>transit backbones, then it would seem that your design is more

We considered an edge-to-edge region would be the most useful deployment 
model, but you are right that we have unnecessarily constrained ourselves. 
We should alter the draft so that the end-to-end architecture covers any 
number of big hops across PCN regions requiring admission control.

But I think your question is about islands separated by nodes doing *no* 
admission control, rather than *other* admission control. This could only 
work if you could be sure that the transit network between the leafs would 
never be congested. This would be difficult, given the motivation for flow 
pre-emption is to cope with failure scenarios. If an operator chose to take 
this risk, it could, but I believe we have a better solution...

The original reason we brought this CL over PCN work to the IETF was 
because we believed we had a way of getting it to work across multiple 
trust domains. But that aspect was a bit researchy at the time, so we 
decided to write the cl-architecture draft around a deployment model that 
addressed just one trust domain. Then, in Feb 06, I posted 
draft-briscoe-tsvwg-re-ecn-border-cheat-00.txt to document the ideas on how 
multiple domains can use this technology even if they don't trust each 
other not to cheat.

I am in the process of completing draft-01 of that. It is almost complete 
barring a fix to an attack Joe Babiarz pointed out (off-list). The latest 
sub-version is at:
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#repcn>. [The fix is 
relatively simple -- I had unnecessarily over-penalised an attacker, which 
could be used by others as an attack.] I shall write in the correction when 
I have some time in the next couple of weeks, but the draft at the above 
URL is a better read than the one posted to internet-drafts.

I would be interested to hear if this addresses your disquiet about 
extending diffserv to multiple autonomous domains, given this draft 
proposes a different (mostly passive) approach to that used in classic 
Diffserv for inter-domain policing.

>more than just a superset of functionality.

How about changing this sentence and moving it to the end of the para as 
follows?:

"  In order to support CL traffic we would expect it to supplement the 
existing Expedited Forwarding (EF). Within the controlled edge-to- edge 
region, a particular packet receives the Pre-Congestion Notification 
behaviour if the packet's DSCP (differentiated services codepoint) is set 
to EF and also the ECN field indicates ECN Capable Transport. However, PCN 
is not only intended to supplement EF. The PCN draft has been written as a 
building block so that authors of other PHBs can choose to add the PCN 
marking behaviour to their scheduling behaviour."

>measurement, I was wondering if relatively short transaction 
>flows  becomes an issue with your design.  Phone calls today still last just

Any admission control scheme will have a signalling etc effort which
may not be worthwhile for a very short flow. I'm not sure SMS is a good 
example, but there certainly will be cases where short flows still might 
need admission control. We envisage one measurement algorithm for short and 
long flows. That is, one marking algorithm on interior routers and one 
measurement alg in egress gateways. Having found an algorithm that gets a 
fast answer with minimal cost, there would be little point in having a 
second, slower alg.

>common practice regarding its parameters.

You're probably right that we have over-constrained implementation. We are 
now discussing how to specify the minimum necessary constraint on the 
algorithms for the next draft. Exponential decay is generally what is most 
desirable and it is also extremely cheap to implement, so I don't often 
consider other algs, but I'm sure we can allow any moving average.

>"state".  Isn't it more accurate to say that they do not hold "per- flow 
>state"?  The diff-serv capable interior nodes still hold

You're right that they do hold some state. But it is per outgoing interface 
card, not per aggregate (assuming the term aggregate means a 
CL-region-aggregate [ie the set of microflows between a specific pair of 
ingress and egress gateways]). We will clarify this fully in the Key 
Benefits section (2.3). Elsewhere we tried to say "no per-flow state", but 
we may have missed a few. We will check.

>adding ambiguity when using the term "node".

OK

>need to address these questions in Section 1.1.2

On re-reading 1.1.2, it looks pretty clear to us which node is doing what. 
But it's obviously not clear to you, so we need to understand what 
ambiguity has crept in. Would the following change to the second para of 
1.1.2 help?

"   The solution involves four steps:

    First, triggering the ingress gateway
    to test whether pre-emption may be needed. An interior router enhanced 
with
    Pre-Congestion Notification may optionally include an algorithm that
    sets packets into the Pre-emption Marked state. Such a packet alerts
    the egress that pre-emption may be needed, which in turn sends a Pre-
    emption Alert message to the ingress.

    Secondly, calculating the right
    amount of traffic to drop. This involves the egress gateway
    measuring, and reporting to the ingress gateway, the current amount
    of CL traffic received from that particular ingress gateway. The
    ingress gateway compares this measurement (which is the amount that
    the network can actually support, and which we thus call the
    Sustainable-Aggregate-Rate) with the rate that it is sending and
    hence determines how much traffic needs to be pre-empted.

    Third, choosing which flows to shed in order to drop the traffic 
calculated in the second step. Information on the priority of flows may be 
held by the ingress gateway, or by some out of band policy decision point. 
How these systems co-ordinate to determine which flows to drop is outside 
the scope of this document, but between them they have all the information 
necessary to make the decision.

    Fourth, tearing down the chosen flows. The ingress gateway triggers 
standard tear-down messages for the signalling protocol in use.

>ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lefaucheur-emergency-rsvp-01.txt
>

I guess Francois will want to comment on this part! I believe a similar 
effect can be engineered by allowing emergency calls in without admission 
control, having appropriately set the configured admission and pre-emption 
rates on interior routers. Then emergency calls can use the capacity 
between these two rates.

Alternatively, an ingress could pre-empt other flows already admitted to 
the same CL-region-aggregate whenever it admits an emergency call while 
admission control is telling it not to.

However, the purpose of this informational draft was to give one clear 
working example. We were trying not to give a load of options.

Francois, can we add some text & a ref in section 1.1.3 [which talks about 
whether both adm ctrl and prmpt are needed]?

>Page 8, 1'st paragraph.  Small nit, but the first sentence is unclear.

Assume this is the sentence "The solution has two aspects." See above - it 
is now: "   The solution involves four steps. " Better?

>voice.  User expectations for video would not seem to be under the

How about this alternative bullet in the key goals:

"   o It needs to be possible to complete flow pre-emption within 1-2 
seconds.  Operators will have varying requirements but, at least for voice, 
it has been estimated that after a few seconds many affected users will 
start to hang up, making the flow pre-emption mechanism redundant and 
possibly even counter- productive. Until flow pre-emption kicks in, other 
applications using CL (e.g. video) and lower priority traffic (e.g. AF) 
could be receiving reduced service. Therefore an even faster flow 
pre-emption mechanism would be desirable (even if, in practice, operators 
have to add a deliberate pause to ride out a transient while the natural 
rate of call tear down or lower layer protection mechanisms kick in). "

>session as a flow identified by RSVP?

How about this alternative text:

"   o The CL service should support emergency services ([EMERG-RQTS],
       [EMERG-TEL]) as well as the Assured Service which is the IP
       implementation of the existing ITU-T/NATO/DoD telephone system
       architecture known as Multi-Level Pre-emption and Precedence
       [ITU.MLPP.1990] [ANSI.MLPP.Spec][ANSI.MLPP.Supplement], or MLPP.
       In particular, this involves admitting new flows that are part of 
high priority sessions even when admission control would reject other 
routine flows. Similarly, when having to choose which flows to pre-empt, 
this involves taking into account the priorities and properties of the 
sessions that flows are part of. "

>or expand the current text to say "signaling protocol produced by the

Suggest the latter. I'm not sure that GIST is correct anyway, surely
QOS-NSLP?

>to state this explicitly as opposed to referring to a new behavior.

Just before the phrase "packets are pre-emption marked", we say "...each 
node in the CL-region runs an algorithm...". Is this sufficient to say how 
and where this algorithm runs?

As to how pre-emption marking (and indeed admission marking) are encoded in 
a pkt, this is under discussion in the cl-phb draft (so we didn't want to 
pre-judge the outcome in this informational draft). New DSCP is one option, 
but there are others using the ECN field. Please see Appendix C 
"Pre-Congestion Notification marking" draft-briscoe-tsvwg-cl-phb-01.txt - 
we also had some discussion at the "bar BoF" in Dallas.

>Section 4.6 Tunneling.  Do you foresee marking the DSCP field in the

Whatever encoding is used, whether DSCP or ECN, we envisage it being in the 
outer header.

>say.  It may be helpful to break up the first sentence into 2 or more

OK

Hope this all helps.

Bob Briscoe & Phil Eardley (& team)

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: ken carlberg <[email-masked]>
Date: Fri, 28 Apr 2006 11:58:08 -0400

Bob,

Sorry about my tardy response.

> We considered an edge-to-edge region would be the most useful  
> deployment model, but you are right that we have unnecessarily  
> constrained ourselves. We should alter the draft so that the end-to- 
> end architecture covers any number of big hops across PCN regions  
> requiring admission control.

I think this would be helpful in gaining wider acceptance for your  
proposed architecture.

> But I think your question is about islands separated by nodes doing  
> *no* admission control, rather than *other* admission control. This  
> could only work if you could be sure that the transit network  
> between the leafs would never be congested. This would be  
> difficult, given the motivation for flow pre-emption is to cope  
> with failure scenarios. If an operator chose to take this risk, it  
> could, but I believe we have a better solution...

understood.  the information I tend to come across is that more times  
than not, its the Tier-2 & 3 ISPs that have a higher propensity to  
suffer congestion in comparison to the Tier-1's.  And that many of  
the Tier-1 ISPs have over-provisioned to such a degree that several  
simultaneous failures would need to occur to be subject to  
congestion.  On the other hand "many" does not equal all, so I'm  
sympathetic to your approach.

I guess what I had in mind is that if one had disjoint islands (eg,  
leaf ISPs) cooperating with each other using your proposed  
architecture, then there wouldn't be a dependency of a continual path  
of ISPs using some form of admission control.  In a sense, one would  
follow the old "smart edge, dumb core" approach and move the concept  
of the edge to the leaf ISP and the core to the Tier-1 ISP.

> The original reason we brought this CL over PCN work to the IETF  
> was because we believed we had a way of getting it to work across  
> multiple trust domains. But that aspect was a bit researchy at the  
> time, so we decided to write the cl-architecture draft around a  
> deployment model that addressed just one trust domain. Then, in Feb  
> 06, I posted draft-briscoe-tsvwg-re-ecn-border-cheat-00.txt to  
> document the ideas on how multiple domains can use this technology  
> even if they don't trust each other not to cheat.
>
> I am in the process of completing draft-01 of that. It is almost  
> complete barring a fix to an attack Joe Babiarz pointed out (off- 
> list). The latest sub-version is at:
> <http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#repcn>. [The fix  
> is relatively simple -- I had unnecessarily over-penalised an  
> attacker, which could be used by others as an attack.] I shall  
> write in the correction when I have some time in the next couple of  
> weeks, but the draft at the above URL is a better read than the one  
> posted to internet-drafts.
>
> proposes a different (mostly passive) approach to that used in  
> classic Diffserv for inter-domain policing.

I'll need to go over the ecn-border-cheat-01 draft more carefully  
before I can say that the proposed supplemental solution would  
address concerns about extending your architecture across multiple  
administrative/trust domains.  But my gut feeling is that I think its  
easier to gain deployment/usage when an approach such as yours that  
extends over the Internet can work more as an overlay-with-gaps  
instead of natively from leaf-ISP to leaf-ISP.  Also, I wouldn't  
treat my concern as something to hold up your document, since I'm  
just speaking from a personal preference.

> How about changing this sentence and moving it to the end of the  
> para as follows?:
>
> "  In order to support CL traffic we would expect it to supplement  
> the existing Expedited Forwarding (EF). Within the controlled edge- 
> to- edge region, a particular packet receives the Pre-Congestion  
> Notification behaviour if the packet's DSCP (differentiated  
> services codepoint) is set to EF and also the ECN field indicates  
> ECN Capable Transport. However, PCN is not only intended to  
> supplement EF. The PCN draft has been written as a building block  
> so that authors of other PHBs can choose to add the PCN marking  
> behaviour to their scheduling behaviour."

that's fine.

> may not be worthwhile for a very short flow. I'm not sure SMS is a  
> good example, but there certainly will be cases where short flows  
> still might need admission control. We envisage one measurement  
> algorithm for short and long flows. That is, one marking algorithm  
> on interior routers and one measurement alg in egress gateways.  
> Having found an algorithm that gets a fast answer with minimal  
> cost, there would be little point in having a second, slower alg.

understood.  I wasn't so much advocating a "slower" algorithm, but  
rather contemplating the benefit of different measurement algorithms  
whose selection were perhaps dependent on the type of data that was  
being measures.  But again, this was just food for thought.

> You're probably right that we have over-constrained implementation.  
> We are now discussing how to specify the minimum necessary  
> constraint on the algorithms for the next draft. Exponential decay  
> is generally what is most desirable and it is also extremely cheap  
> to implement, so I don't often consider other algs, but I'm sure we  
> can allow any moving average.

ok.  I was mostly curious about leaving that door open.

> You're right that they do hold some state. But it is per outgoing  
> interface card, not per aggregate (assuming the term aggregate  
> means a CL-region-aggregate [ie the set of microflows between a  
> specific pair of ingress and egress gateways]). We will clarify  
> this fully in the Key Benefits section (2.3). Elsewhere we tried to  
> say "no per-flow state", but we may have missed a few. We will check.

ok.

> On re-reading 1.1.2, it looks pretty clear to us which node is  
> doing what. But it's obviously not clear to you, so we need to  
> understand what ambiguity has crept in. Would the following change  
> to the second para of 1.1.2 help?
>
>    to test whether pre-emption may be needed. An interior router  
> enhanced with
> calculated in the second step. Information on the priority of flows  
> may be held by the ingress gateway, or by some out of band policy  
> decision point. How these systems co-ordinate to determine which  
> flows to drop is outside the scope of this document, but between  
> them they have all the information necessary to make the decision.
>
>    Fourth, tearing down the chosen flows. The ingress gateway  
> triggers standard tear-down messages for the signalling protocol in  
> use.

that's helpful.  thanks.

> I guess Francois will want to comment on this part! I believe a  
> similar effect can be engineered by allowing emergency calls in  
> without admission control, having appropriately set the configured  
> admission and pre-emption rates on interior routers. Then emergency  
> calls can use the capacity between these two rates.
>
> Alternatively, an ingress could pre-empt other flows already  
> admitted to the same CL-region-aggregate whenever it admits an  
> emergency call while admission control is telling it not to.
>
> However, the purpose of this informational draft was to give one  
> clear working example. We were trying not to give a load of options.

ok, understood.

> Assume this is the sentence "The solution has two aspects." See  
> above - it is now: "   The solution involves four steps. " Better?

yes, thanks.

> "   o It needs to be possible to complete flow pre-emption within  
> 1-2 seconds.  Operators will have varying requirements but, at  
> least for voice, it has been estimated that after a few seconds  
> many affected users will start to hang up, making the flow pre- 
> emption mechanism redundant and possibly even counter- productive.  
> Until flow pre-emption kicks in, other applications using CL (e.g.  
> video) and lower priority traffic (e.g. AF) could be receiving  
> reduced service. Therefore an even faster flow pre-emption  
> mechanism would be desirable (even if, in practice, operators have  
> to add a deliberate pause to ride out a transient while the natural  

That's fine.

>       In particular, this involves admitting new flows that are  
> part of high priority sessions even when admission control would  
> reject other routine flows. Similarly, when having to choose which  
> flows to pre-empt, this involves taking into account the priorities  
> and properties of the sessions that flows are part of. "

Sorry to be a pain about this, but again, when you bring up the term  
"session", I (the novice reader) will wonder the context you are  
talking about.  My assumption is the session/flow that  was conveyed  
by a signaling protocol like RSVP.

> Suggest the latter. I'm not sure that GIST is correct anyway, surely

ok

> Just before the phrase "packets are pre-emption marked", we say  
> "...each node in the CL-region runs an algorithm...". Is this  
> sufficient to say how and where this algorithm runs?

ok, that helps.

> As to how pre-emption marking (and indeed admission marking) are  
> encoded in a pkt, this is under discussion in the cl-phb draft (so  
> we didn't want to pre-judge the outcome in this informational  
> draft). New DSCP is one option, but there are others using the ECN  
> field. Please see Appendix C "Pre-Congestion Notification marking"  
> draft-briscoe-tsvwg-cl-phb-01.txt - we also had some discussion at  
> the "bar BoF" in Dallas.

ok.  I missed the bar BoF

> Whatever encoding is used, whether DSCP or ECN, we envisage it  
> being in the outer header.

ok, it may be helpful to state this in the draft.

cheers,

-ken