Subject: End to end QoS guarantee for users in VPN

From: Chris Lewis <[email-masked]>
Date: Sun, 22 Feb 2004 21:08:04 -0600

Neil,

I understand that this may be a longish discussion, but would you be 
prepared to engage on a couple of points from this topic on the alias?

The couple of points I'm interested in hearing more from you on are;

1. The need for over-provisioning to ensure end to end QoS, which you 
(correctly IMHO) identify as not a good solution
2. The need for per VPN QoS

On the first one, Steven Casner in NANAOG 22 presented a nice study of a 
tier 1 backbone that showed that for pretty much all of the time, if there 
was twice the bandwidth available than offered load, there was never 
congestion/delay/jitter, even under failures. This does of course trade off 
the cost of bandwidth with a simple network planning model. The downsides 
however do include cost, unexpected shifts in traffic, provisioning 
failures, internet traffic affecting VoIP and so forth. One proposal is to 
over provision (or under provision) on a per class basis, so that say the 
controlled latency class has 2 times the bandwidth available when compared 
to the offered load, however the best effort class may be under provisioned 
and not get all it needs in times of congestion. The obvious benefit here 
is that the cost of over-provisioning is lowered and is only done for the 
classes that really need it.

This leads on to point 2, while it is true to say it is impossible to come 
up with a portable definition of what is mission critical and what is not 
for an enterprise, it is possible to identify different levels of service 
within a provider VPN that the enterprise can match its traffic to, such as 
voice requiring 150 ms end to end delay, or a transaction program requiring 
X hundred msecs of delay, or a file transfer requiring Y Mbps.

So is it not feasible to suggest that a provider can offer these different 
service types, then knowing what has been subscribed to, engineer a network 
for the aggregate, meaning that it is known that X hundred Mbps of voice 
transport has been sold, and therefore 2X is provisioned in the network? If 
this then meant that all voice traffic was treated properly, there would be 
no need for per VPN QoS to support voice. The same could then be true of 
other traffic classes (even though those other classes would have different 
under or over provisioning ratios).

Cheers

Chris

This needs a longish response .....so I'll get back to you off-list.
regards, Neil
-----Original Message-----
From: Jeff Helmer [mailto:[email-masked]]
Sent: 20 February 2004 23:50
To: Harrison,N,Neil,IKR2 R
Cc: [email-masked]
Subject: Re: FW: End to end QoS guarantee for users in VPN

Neil,
I wanted to understand the phrase "traffic isolation on a per VPN basis" 
that you used. Do you mean per-VPN LSPs (without any merging) between the 
PE routers (assuming a MPLS core)? This would ensure each VPN has a 
guaranteed end-to-end bandwidth which may not be possible if all VPNs 
between a pair of PE routers used the same tunnel LSP. The DS codepoint 
would then signify the relative importance of different kinds of traffic in 
the same VPN.
- Jeff

Jeff Helmer <[email-masked]> wrote:
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Li 
Xiangjun
Sent: 19 February 2004 02:08
To: [email-masked]
Subject: End to end QoS guarantee for users in VPN

hi;
Traditionally we only manages the toal QoS of VPN. In fact,many users in 
the enterprise use the VPN,that means the VPN carries different kinds of 
services(VoIP.HTTP.VoD et ac.) of different users in the enterprise. How to 
assure every user's QoS, that is to say every flow's QoS ?
Is there any suggestion about it?
Excellent question. And I agree with you. As a general observation it is 
not possible to differentiate 'urgency' (ie how fast?) from 'importance' 
(ie how critical is this pkt vs others?) from just a DS codepoint at the 
pkt level. One can imagine cases in 1 VPN where a given voice call is 
mission critical and others where it is not....same applies to data 
applications. Further, there is a difference between the treatment of 
public traffic and private traffic.....and in fact some public traffic that 
belongs to the same 'urgency' class can also have different 'importance'. 
One can use the LSP as a survivavbility differentiator, ie to isolate 
different VPN instances......however, this requires p2p LSPs because if the 
traffic gets merged on lower mp2p LSPs then per VPN 
QoS/perf/failure-handling behaviour is not simple to provide. The usual 
response to such uncertainty in these quantities is to significanly 
overprovide.......which is not a good solution.
There are deeper issues here of pricing for 'fair/meaningful' services 
that, in addition to the urgency vs importance angle, is affected by BW 
borrowing between DS classes.....and these can't be easily sorted IMO if 
the network arch is unable to deliver traffic isolation on a per VPN basis.
regards, Neil

---

From: "Li Xiangjun" <[email-masked]>
Date: Wed, 25 Feb 2004 11:21:22 +0800

Service Level Agreement about VPN between enterprise and Service Provider maybe is as following(I think enterprise customer do need these requirements): 
    VPN Service SLA(an Example):
|---------------------------------------------------------------------------------------------------------------------------------
|                                                                                                                                      performance parameters & target value
|            Application                                                 Data Rates                                      ----------------------------------------
|                                                                                                                                           Delay       Jitter     Packet Loss Ratio
|---------------------------------------------------------------------------------------------------------------------------------
|            Total QoS                                                    2 M
|--------------------------------------------------------------------------------------------------------------------------------
|                                           Min Session Num  Max Session Num   PerSession Requirement
|---------------------------------------------------------------------------------------------------------------------------------
|                                VoIP                10                        50                         10 Kbit/s                < 150ms          < 1ms        < 3%
|                               -------------------------------------------------------------------------------------------------------------
|  Restriction              VoD                  2                          5                          200Kbit/s                <10ms                             <1%
|                                    -------------------------------------------------------------------------------------------------------------
|                                   Best Effort   Min Total  500K
|----------------------------------------------------------------------------------------------------------------------------------
Best Effort's priority is the lowest and others can preempt it's bandwidth.

    The question is how to guarantee each session's QoS, not each class. Even though each class' Qos can be guaranteed, for each session ,maybe it's bad one.
    For example, 1 M bandwidth is allocated to VoD class service absolutely, that is mean, 5 sessions' QoS can be guaranteed. If 10 persons(maybe more) in the enterprise request VoD service concurrently, if no resource control entity exists and all the request are treated fairly, then 10 VoD session maybe exist and each has bad QoS. The point is that 5 VoD sessions' QoS can be guaranteed if others 5 request are not allowed. 
    If a call admission control entity exists, but for real-time service,like VoIP, how to guarantee it's real time requirement?

Regard,
    
----- Original Message ----- 
From: [email-masked] 
To: [email-masked] 
Sent: Tuesday, February 24, 2004 1:00 AM
Subject: L3vpn digest, Vol 1 #137 - 1 msg

Today's Topics:

   1. End to end QoS guarantee for users in VPN (Chris Lewis)

Neil,

Cheers

Chris

---

From: [email-masked]
Date: Wed, 25 Feb 2004 07:28:35 -0000

*re-posting in plain text converted from original HTML format of original posting*

Hi Chris....I can't really do justice to this on this list..........and where to start! However, let me have a go (though I really don't want to open up any protracted discussion here).
 
Its not as simple as homing-in on a single topic like QoS support in VPNs (whatever that truly means...see later).........we really have to go right back to the fundamental principles of the 3 networking modes (ie co-cs, co-ps and cl-ps) and look across the whole arch vista to understand how QoS fits in the big picture.

This means we need to start with respecting the func arch principles of G.805/809 for the mode considered, because everything flows from these. >From this we can then define a network and NMS/OSS architecture that: 
- delivers a minimised opex/capex solution across all public and private services (this is multi-faceted) 
- is secure for both customer traffic and internal operator control/management traffic......so, for example, an OOB network is required for internal/sensitive operator control and management traffic 
- allows the creation of service models (which include appropriate differential pricing models) that are both meaningful and fair (to all parties) across all networking loading points and across all public and private services........if we don't get this right we will probably damage the industry for all parties (ie end-users/customers, vendors and operators). 
- allows a network design that is proofed against all future (unknown) traffic types/vectors.
 
Now in our view IETF have created some networking problems that won't allow us to easily (if at all) achieve the above objectives. I have been posting mails on these problems for a long time now, esp on what is wrong with MPLS as-is and some of its consequences like PWE3. However, and for whatever reasons, we have not been able to get IETF to take our requirements into account. I suspect one of the problems for IETF these days is an unclear target audience: 
- is it the Internet? 
- is it the ISP/alt-nets? 
- is it the trad operators like BT? 

There are conflicting requirements here. We are no longer operating in the old Arpanet days when the Internet was a big experiment......so the IETF modus operandi that worked fine then looks highly suspect to us now (and we can't fix that here anyway). The bubble on misreported BW demand has burst and so trad operators are being courted again by vendors.......but our commercial requirements are not the same as the others above.
 
I can do no better than to advise anyone who really wants to understand our thinking on networking (eg why we must start with G.805/809 func arch models) to please read chapters 3 and 4 (at least) of a colleague's book (Andy Reid, co-authored with Mike Sexton) on 'broadband networking'. I know this stuff is anathema to some folks, but for us operators it is crucial since this is the basis for developing the management information models.....and NMS/OSS are key areas to get right for any major operator ........but these can only be as good as the underpinning network arch models allow. Break the arch rules and it will all slowly start to go pear-shaped. 

I could provide you with a long list of stuff that is broken in our view. However, that would take pages.
 
Some further remarks in-line below. 

regards, Neil 

: Chris Lewis [mailto:[email-masked]]
Sent: 23 February 2004 03:08
To: Harrison,N,Neil,IKR2 R
Cc: [email-masked]
Subject: End to end QoS guarantee for users in VPN

	Neil,
	
	I understand that this may be a longish discussion, but would you be prepared to engage on a couple of points from this topic on the alias?

	1. The need for over-provisioning to ensure end to end QoS, which you (correctly IMHO) identify as not a good solution 
	2. The need for per VPN QoS 

NH=> I think QoS is much abused term Chris. Few networks (not just comms) can create a viable commercial model from multiple 'QoS levels'....look at successful transport networks say, eg we have a very successful airline in the UK called Easyjet, and its based on a 1-class service model....just '1st come 1st served'. ATM tried lots of QoS/traffic classes. They were not a success..........simple ATM services are a success however......and one of the key strengths of ATM is that we have a trail that can be managed quite separately from the traffic flowing on that trail (this is related to the 'urgency' vs 'importance' issue that I will return to shortly). In fact all the networks that have been successful to date have not really 'done QoS' as a conscious act, eg PSTN, Sonet/SDH, Internet as-is, FR.

QoS is also intrinsically related to a notion of BW squeezing, ie trying to get more traffic on a given resource. It is, however, an efficiency illusion. BW is not a scarce resource any longer and the costs are falling each day......complexity is a major problem however, and its costs are rising each day. Multiple traffic/QoS classes create complexity/capex costs in both the data-plane (buffers, schedulers, etc) and the control-plane protocols to support this, and there are associated complexity/opex costs with the design/operation/maintenance of such networks. Real networks, as you also note below, don't run heavily loaded. Why? Well, there are lots of reasons such as:
- the topology of a given layer network is created from trails in a lower server layer network, eg optics, SDH VC4s........ so planning-cycle growth comes in large BW quanta ;
- layer networks require ~ 30% redundancy for protection/restoration 
- there is a varying demand over usage cycles, eg daily, so capacity has be related to some peak usage demand 
- if there is uncertainty (or a lack of control) of routing this encourages over-dimensioning
- similarly, uncertainties in traffic forecast can lead to over-dimensioning (and possibly under-dimensioning in some cases). 

Now given all service requirements have to be met at the peak time and at the *end* of the current planning-cycle, then over the life of real networks it is not surprising they seem to be under-utilised.

This creates a serious commercial challenge for operators in that QoS has to be seen to be 'fair and meaningful' over a large range of loads.......and the really big challenge is ensuring the right behaviour for:
-	Normal Operation...load <~50% for maybe >99% of time?
-	Abnormal Operation...load -> 100% for maybe <1% of time?

The network behaviour required is different in each of these operating regions......in the former (ie nearly all of the time) its providing fair/meaningful service (ie 'you get what you pay for not what someone else has paid for'), whilst in the latter its about protecting the network (and this is a one reason the control/management protocols have to go OOB, another reason is security) from overload (ie must remain stable) and ensuring that the right serivices survive......this is the 'urgency' vs 'importance' problem.

Now any technology that allows BW sharing across different QoS classes will not allow a solution to the commercial problems. Indeed, over provisioning simply makes these commercial problems worse! 

There is also the problem of 'urgency' vs 'importance'. Under failures its not QoS considerations that should determine which traffic survives, but some 'importance' semantic........which IMO is something one has to associate with the trail entity, esp in the case of private vs public traffic say. This survivability issue also implies a required 'ordering behaviour'....because fault-handling and survivability/availability (which are related) are far more important metrics than QoS, and must be considered before QoS has any meaningful basis. That is: 
- the arch of the network mode considered must be right 1st 
- which then allows the appropriate defects to be identified 
- which then allows the appropriate OAM to be defined ......such that defects have the appropriate entry/exit criteria and consequent actions specified 
- which then allows the (un)available states to be defined 
- which then, and only then, provides the temporal basis (ie the available state) where QoS has any meaning. 

These aspects are not the same for the cl-ps, co-ps and co-cs modes, ie a different approach is required for each mode.....and this is assuming one has (i) respected the func arch of the mode considered, and (ii) that the technology used is correctly and adequately functionally specified. 

And remember what I said earlier about the importance of management information models? Well, once one starts to define the fault-handling and availability/QoS registers in network elements, and how these interface to the NMS/OSS and stuff like billing/SLAs/etc, then it is rather important that the above thinking has flowed through from a proper understanding of the underpinning network architecture of the mode being considered. 

You can use a co-ps mode like MPLS to provide the missing survivability/availability control (at the trail level rather than the pkt level) for VPNs. This is fine providing it respects the allowed arch of that mode (ie p2p or p2mp only) but fails if there is merging, ie LDP mp2p....and PHP is also a merging device. The mp2p/PHP violations of the co-ps mode also create significant fault management problems too ....and these get compounded if we add a proprietary ECMP behaviour to the brew. And noting what I said earlier, if we can't get the defects/OAM sorted first then we don't even have a basis for addressing the traffic/QoS aspects anyway. I therefore find it hard to take seriously anyone who tells me LSP-Ping is 'simple/efficient' (as the draft currently says)....esp compared to Y.1711. Though I have to own up to self-interest here as I penned to original version of Y.1711. But please note this was not specifically written for MPLS, this is how IMO we should do the basic defect detection/handling in any co-ps mode technology that is architecturally compliant.

CL=> On the first one, Steven Casner in NANAOG 22 presented a nice study of a tier 1 backbone that showed that for pretty much all of the time, if there was twice the bandwidth available than offered load, there was never congestion/delay/jitter, even under failures. This does of course trade off the cost of bandwidth with a simple network planning model. 

NH=> So if there is BW sharing between classes and if the network is lightly loaded over much of its life (which it has to be) then just how can one make a good commercial model for 'service differentiation'?

CL=>	The downsides however do include cost, unexpected shifts in traffic, provisioning failures, internet traffic affecting VoIP and so forth. One proposal is to over provision (or under provision) on a per class basis, so that say the controlled latency class has 2 times the bandwidth available when compared to the offered load, however the best effort class may be under provisioned and not get all it needs in times of congestion. The obvious benefit here is that the cost of over-provisioning is lowered and is only done for the classes that really need it. 

NH=> I think there are alternative ways of looking at the overall service problem we operators have, but this requires one to have an open-mind wrt to the solutions and the real value of QoS in relation to the network mode.....see my next comment. 
	
CL=>	This leads on to point 2, while it is true to say it is impossible to come up with a portable definition of what is mission critical and what is not for an enterprise, it is possible to identify different levels of service within a provider VPN that the enterprise can match its traffic to, such as voice requiring 150 ms end to end delay, or a transaction program requiring X hundred msecs of delay, or a file transfer requiring Y Mbps.
 
NH=> It's not just private traffic we have to address, but also public traffic.

Let's assume we are addressing an operator's commercial service requirements and not just those of the Internet. Suppose we had 2 simple modes of (1 class in each) cl-ps and co-ps (emulating co-cs, ie CBR) working in harmony, and where the co-ps core can offer both on-demand SVC-like services for public traffic (but only as/when required and charged for accordingly, else use the cl-ps mode) and S-PVC-like based services for private traffic when strong SLAs and strong traffic isolation are required (one can still do VPNs using a proper cl-ps server layer, but the SLAs and traffic isolation available now are much weaker).

Further, for VPNs we could have more generic models......where a (so-called) L3 VPN is nothing more than adding a client layer reachability component to a base-case simple managed BW (so-called) L2 VPN.

In the public service case we let the application/user choose the mode (we obviously need different pricing models here). You then don't need a load of complex QoS classes in each mode that won't work anyway, just simple peak rate models work fine....the biggest 'QoS bang per buck' comes from mode selection and we can thus enforce fair/meaningful services over all load points.....users now only get what they pay for, not what someone else has paid for. One could argue that this is using BW wisely to solve both technical and commercial problems that are otherwise hard or intractable.

Add an OOB control/management-plane and the top-level objectives I mentioned earlier can now really start to be addressed. 

CL=>	So is it not feasible to suggest that a provider can offer these different service types, then knowing what has been subscribed to, engineer a network for the aggregate, meaning that it is known that X hundred Mbps of voice transport has been sold, and therefore 2X is provisioned in the network? If this then meant that all voice traffic was treated properly, there would be no need for per VPN QoS to support voice. The same could then be true of other traffic classes (even though those other classes would have different under or over provisioning ratios). 

NH=> As I noted earlier, QoS is an illusion if seen solely as a means to an end in a single network mode. The successful networks to date have never done 'QoS' and I don't see any reason why this should suddenly change......QoS should be a non-issue IMO if you get the right network arch in place for the 3 modes, and you use them wisely working in harmony.

We have various internal papers that go into some of the stuff I have briefly touched on above, ie what precisely is wrong/missing with MPLS as-is and the stuff its spawned, like in PWE3, and also the development of an alternative network service architecture based on the 3 modes working together. 

regards, Neil

	Cheers
	
	Chris
	
	-----Original Message-----
	From: Jeff Helmer [<mailto:[email-masked]>]
	Sent: 20 February 2004 23:50
	To: Harrison,N,Neil,IKR2 R
	Cc: [email-masked]
	Subject: Re: FW: End to end QoS guarantee for users in VPN
	
	Neil, 
	I wanted to understand the phrase "traffic isolation on a per VPN basis" that you used. Do you mean per-VPN LSPs (without any merging) between the PE routers (assuming a MPLS core)? This would ensure each VPN has a guaranteed end-to-end bandwidth which may not be possible if all VPNs between a pair of PE routers used the same tunnel LSP. The DS codepoint would then signify the relative importance of different kinds of traffic in the same VPN. 
	- Jeff
	
	-----Original Message-----
	From: [email-masked] [<mailto:[email-masked]%5DOn>On <mailto:[email-masked]%5DOn> Behalf Of Li Xiangjun
	Sent: 19 February 2004 02:08
	To: [email-masked]
	Subject: End to end QoS guarantee for users in VPN
	
	hi; 
	Traditionally we only manages the toal QoS of VPN. In fact,many users in the enterprise use the VPN,that means the VPN carries different kinds of services(VoIP.HTTP.VoD et ac.) of different users in the enterprise. How to assure every user's QoS, that is to say every flow's QoS ? 
	Is there any suggestion about it? 
	Excellent question. And I agree with you. As a general observation it is not possible to differentiate 'urgency' (ie how fast?) from 'importance' (ie how critical is this pkt vs others?) from just a DS codepoint at the pkt level. One can imagine cases in 1 VPN where a given voice call is mission critical and others where it is not....same applies to data applications. Further, there is a difference between the treatment of public traffic and private traffic.....and in fact some public traffic that belongs to the same 'urgency' class can also have different 'importance'. One can use the LSP as a survivavbility differentiator, ie to isolate different VPN instances......however, this requires p2p LSPs because if the traffic gets merged on lower mp2p LSPs then per VPN QoS/perf/failure-handling behaviour is not simple to provide. The usual response to such uncertainty in these quantities is to significanly overprovide.......which is not a good solution. 
	There are deeper issues here of pricing for 'fair/meaningful' services that, in addition to the urgency vs importance angle, is affected by BW borrowing between DS classes.....and these can't be easily sorted IMO if the network arch is unable to deliver traffic isolation on a per VPN basis. 
	regards, Neil

---

From: [email-masked]
Date: Wed, 25 Feb 2004 07:28:42 -0000

Can you please use plain text for mails in future.......I tried responding to an orginal mail from yourself that Chris Lewis had tagged with some supplementary questions (I note you have attached some of this below) and found it ended up too large (the list moderator culled it).
 
The basic answer to your question is this......'there is no such thing as a free lunch'.  Has it ever crossed your mind that the reason this stuff is not easily solveable is that perhaps the underpinning approach and network arch is all wrong?  I'll re-post my response to Chris I noted above in plain text where I discuss this aspect for both public and private services (you can't just consider these in isolation).......not that I expect many dyed-in-the-wool IETF folks will like it because, as I say, it questions the fundamantal approach being used so far.
 
thanks...regards, Neil

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Li Xiangjun
Sent: 25 February 2004 03:21
To: [email-masked]
Subject: Fw: End to end QoS guarantee for users in VPN 

 
Regard,
    
----- Original Message ----- 
From: [email-masked] 
To: [email-masked] 
Sent: Tuesday, February 24, 2004 1:00 AM
Subject: L3vpn digest, Vol 1 #137 - 1 msg

Today's Topics:

Neil,

Cheers

Chris