Subject: RE: [pim] Fwd: Multicast in MPLS/BGP IP VPNs

From: [email-masked]
Date: Thu, 20 Oct 2005 08:21:45 +0100

Dino,

If you don't think PE-based VPNs will scale for multicast services, please can you explain how you think a CE-based VPN approach will scale better? I would be particularly interested in hearing how you think bandwidth consumption will compare using the CE-based approach, bearing in mind that all replication will need to take place at the CE, i.e. for N receivers the source CE access link will need to carry N copies of the same packet.

> There are scaling problems with basic PE-based VPNs.

All solutions/technologies have scaling limits that when reached processing/bandwidth resources need to be increased or an alternative solution/technology needs to be selected. Please can you elaborate as to what the scaling limits are when using PE-based VPNs that cannot be overcome by increasing network resources incrementally as the number of customers increases, e.g. adding new route reflectors? 

> How do you expect anything that overlays on top of it to work well?

I assume here you are talking about overlaying multicast VPNs over PE-based unicast VPNs? If so, what exactly is it that you believe is overlaid and over what? In the case of IP/GRE PE-based VPN multicast solutions for example, IP/GRE tunnels are used for multicast traffic whereas completely separate MPLS tunnels are used for carrying unicast traffic, i.e. there is no overlay. In addition, depending on the implementation, the multicast memory/processing resources may be completely separate from unicast memory/processing resources. So, exactly what overlay are you talking about?
 
> PE-based VPNs will not scale, it is the wrong place to put the technology.

Really? I think there may be a few operators that will disagree with you. Again, please can you explain what the scaling limits are and why they can't be overcome by increasing network resources?

>  Putting VPN support at the edges (CE-based VPNs) allows them to scale in a
>  distributed manner

So?.. Distributed scaling is not unique to the CE based approach, PE based VPNs can also scale in a distributed manor. For example, route reflectors can be added as the number of customer routes increases and new blades can be added to PE chassis incrementally as customer demand increases.  

> and is more cost-effective because you can avoid the huge honkin PE box cost

Please can you explain how the CE-based approach is more cost-effective than the PE-based approach? You don't need to pay for a "huge honkin PE box" from the outset. PE chassis are relatively cheap and blades can be added incrementally as customer demand increases. Putting device equipment costs aside, how do you think bandwidth costs/efficiency for multicast traffic compares using the CE-based approach (where N streams will need to be carried for N receivers on the source access link)? 

> with all sorts of additional issues like lack of privacy, single point of failure, etc...

Regarding privacy, operators have spent a significant amount of time and money investing in network provisioning/management systems to ensure VPN isolation is maintained. If you believe privacy is an issue, what do you think the issues are? Regarding single points of failure, failure of a CE connected to a central office or hub site in the CE-based VPN approach is just as detrimental to the customer as a PE failure in the PE-based approach. In addition, in the CE based approach a CE still needs to connect to an edge router in the providers network, which without any redundancy will be a single point of failure. The only way to protect against failure is to use redundancy (in the form of devices, routes, links, cards, etc.), this is orthogonal to whether a CE or PE based VPN approach is used.

>     People say that CE-based VPNs are too hard to manage by a service provider.
>     I don't believe it for one minute.

Who says that? Providers offer managed and unmanaged VPN solutions (where management refers to the management of the CE) so the provider may need to manage the CE whether a CE-based or PE-based approach is being used anyway.

> If cable and DSL providers can put boxes in our homes, we can have script-based or 
> auto-configuring CE-based devices in remote offices. And if and only if you want 
> network based encryption, then you get encrypted packets starting out 
> on the CE-PE link.

Now you've lost me... What sort of CE-based VPN do you know of that doesn't require CE-PE encryption? If there isn't any encryption and the traffic is being carried over a shared infrastructure then the provider somehow needs to provide VPN isolation, which requires partitioning of resources at the edge (PE), i.e. a PE based VPN is required.

>     So let's fix the fundamental problem and not finger point to PIM or any
>     other protocol.

What's the fundamental problem then?

> BTW, we could get BGP to scale better too if it didn't have to support VPNs.

And?... So what's your point? If I take the back seats out of my car to improve the power-to-weight ratio then it might go a few miles an hour faster, but it won't be quite as useful. Anything can be optimised, but it still needs to support key functions, and I think VPN support is quite important to the majority of providers.

Regards,
Richard

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2005 09:38:53 -0700

>> If you don't think PE-based VPNs will scale for multicast services, please can you explain how you think a CE-based VPN approach will scale better? I would be particularly interested in hearing ho…

    Mike McBride and Yiqun Cai will present their draft at the L3vpn session.

>> All solutions/technologies have scaling limits that when reached processing/bandwidth resources need to be increased or an alternative solution/technology needs to be selected. Please can you elab…

    Yes, you are right. Increasing resources costs money. There are times 
    when you just can't solely throw hardware resources at a problem.

    In a CE-based solution, the hardware resource is added for the new site
    (if not already there) and the PE doesn't need add anything.

>> I assume here you are talking about overlaying multicast VPNs over PE-based unicast VPNs? 

    Yes.

>> Really? I think there may be a few operators that will disagree with you. Again, please can you explain what the scaling limits are and why they can't be overcome by increasing network resources?

    Yes, they will disagree but not for this reason.

>> So?.. Distributed scaling is not unique to the CE based approach, PE based VPNs can also scale in a distributed manor. For example, route reflectors can be added as the number of customer routes i…

    Distribute even more, closer to the edges so the CE boxes handle only the
    VPN it is attached to. And you get inter-AS VPNs for free.

    Do you realize how complicated intra-AS MPVNs are, let alone inter-AS,
    and extranet based multicast services?

>> Please can you explain how the CE-based approach is more cost-effective than the PE-based approach? You don't need to pay for a "huge honkin PE box" from the outset. PE chassis are relatively chea…

    No, you buy a PE box that aggregates connections. That is the physical
    requirement. But adding all the policy-basd edge features and complicated
    control-plane interaction which operators have to deal with (read: high
    OpEx) then the cost of the box and the service goes up. And as you add
    customers and more VPNs, and then more services on top of it (read: 
    multicast) it isn't going to get easier.

>> Regarding privacy, operators have spent a significant amount of time and money investing in network provisioning/management systems to ensure VPN isolation is maintained. If you believe privacy is…

    VPNs are hyped as being "private". But private to me means my data is
    encrpyted before I hit the ISP (I actually prefer my data encrpted before
    it leaves the system I am typing on). But VPNs really deliver "separation"
    and very loosely at that.

    I have also heard that VPNs can be made secure by justing using IPsec
    end-to-end. Well I can do that without VPNs as well. So who is helping
    who here to provide the privacy?

>>Who says that? Providers offer managed and unmanaged VPN solutions (where management refers to the management of the CE) so the provider may need to manage the CE whether a CE-based or PE-based app…

    I talked to dozen of ISP type customers who have mentioned this to me.
    Probably over the last 8 years.

    I am glad to hear you think providers offer managerment VPN services. And
    I agree beacuse my data points validate that too.

>> Now you've lost me... What sort of CE-based VPN do you know of that doesn't require CE-PE encryption? If there isn't any encryption and the traffic is being carried over a shared infrastructure th…
 
    I am talking about PE-based VPNs don't provide encrpytion in and of 
    itself.

>> What's the fundamental problem then?

    The architecture needs to be more distributed so it can scale better.

    It simply comes down to where in the network you put the functionality.

>> And?... So what's your point? If I take the back seats out of my car to improve the power-to-weight ratio then it might go a few miles an hour faster, but it won't be quite as useful. Anything can…

    To follow your analogy, definitely keep the back seats in, but take a
    look at what you are carrying in the trunk. Maybe you should move it
    to your wife's SUV because it is more appropriate to be there.

Dino

---

From: Pedro Roque Marques <[email-masked]>
Date: Thu, 20 Oct 2005 22:05:33 -0700

Dino Farinacci wrote:

>>> Please can you explain how the CE-based approach is more cost-effective than the PE-based approach? 
>>> You don't need to pay for a "huge honkin PE box" from the outset. PE chassis are relatively cheap and
>>> blades can be added incrementally as customer demand increases. Putting device equipment costs aside,
 >>> how do you think bandwidth costs/efficiency for multicast traffic 
compares using the  CE-based
 >>> approach (where N streams will need to be carried for N receivers 
on the source access link)?
>     multicast) it isn't going to get easier.

Dino,
The "control-plane interactions" are not complexity that PE-based VPNs 
intruduce. The problem exists by itself.

e.g. given a VPN with a few hundred sites, CE to CE based routing 
exchange is non-trivial as can be proven by looking at any SP network 
that has that number of edge boxes. The policy-based edge features on a 
PE allow a SP to aggregate the control-plane problem as well as traffic. 
Contrary to your claim, it makes the service more scalable from a 
management point of view.

Clearly, you don't want to have to modify the CE configuration when new 
CEs are introduced... so a "network discovery" procedure is necessary. I 
think we can agree on that one. That is a necessary control plane 
interaction.

 From a point of routing exchange, e.g. move from a NxCE mesh to some 
sort of hierarchical setup (e.g. using BGP route reflection), it also 
helps the SP to do this design function once on the core, instead of per 
VPN basis...

Although one has the option to move this routing exchange to the CE 
(inter-as option c) very few SPs choose to do so, even on managed 
setups. I don't believe they do so out of ignorance.

Also, i'm sure that you still remember the IP-overlay-on-ATM days and 
are quite aware of the issues with overlay networks. The top layer (the 
VPN routing in the case) really needs visibility into the lower transit 
layers if you want routing to actually follow some reasonably sensible 
path on failure.

The best argument against CE-based VPNs, imho, are the current MVPN 
deployements which are VR-based instead of using the same model than 
2547. Having per VPN data groups in the provider core, is undesirable 
from an operations an state point of view.

So contrary to your view, i'd argue that finding ways to aggregate 
control and management state in the PE should be one of the top goals 
for a VPN solution.

 > encrpyted before I hit the ISP

No, it doesn't. They just need to be as private as the leased lines they 
are replacing. Most entreprises used to run their own FR/ATM network; 
VPNs are a tool to outsource that service.

   Pedro.

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2005 22:28:29 -0700

Cc'ing [email-masked] for continuity.

>> management point of view.

    It doesn't make it more scalable it makes it more centralized. And
    distributing the problem doesn't make management harder. There are
    ways to do auto-discovery of CEs for a given VPN.    

>> think we can agree on that one. That is a necessary control plane 

    You don't have to. There are deployed mechanisms that have already solved
    this.

>> helps the SP to do this design function once on the core, instead of per 

    No, not once in the core. Once for each new VPN per PE. Plus with a 
    CE-based solution, nothing needs to change in the P or PE routers. They
    simply move bits.

>> setups. I don't believe they do so out of ignorance.

    Of course they don't choose to do so. There are business reasons to
    lock all sites of a VPN to the same provider. This does not give
    flexibility to their customers. And I'm sure their customers would like
    to have choices. And if they had provider choices, everyone's service
    would improve.

>> layers if you want routing to actually follow some reasonably sensible 

    Tunneling is overlay networks. VPNs are overlay networks. So don't say
    we don't do it anymore. And what do you call VPLS?

>> from an operations an state point of view.

    You don't need per VPN data groups.

>> control and management state in the PE should be one of the top goals 

    You are not aggregating, you are doing all of it in one place. Aggregating
    means to *reduce* information, 2547 does not do that. 2547 is a 
    multiplier of routing tables with only memory separation and no *real*
    privacy.

>> VPNs are a tool to outsource that service.

    No, the design should have gone farther. The requirements should have 
    had security (read encryption) built in. 

Dino

-------------------------------------------------------------------------------

X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="222326298:sNHT32368748"
X-BrightmailFiltered: true
X-Brightmail-Tracker: AAAAAA==
X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="128700253:sNHT15953352"
X-BrightmailFiltered: true
X-Brightmail-Tracker: AAAAAA==
X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="489568432:sNHT22759208"
Date: Thu, 20 Oct 2005 22:05:33 -0700
From: Pedro Roque Marques <[email-masked]>
User-Agent: Mozilla Thunderbird 1.0.7-1.1.fc4 (X11/20050929)
X-Accept-Language: en-us, en
To: Dino Farinacci <[email-masked]>
CC: [email-masked]
Subject: Re: [pim] Fwd: Multicast in MPLS/BGP IP VPNs
In-Reply-To: <[email-masked]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
X-OriginalArrivalTime: 21 Oct 2005 05:05:34.0669 (UTC) FILETIME=[117BB7D0:01C5D5FD]
X-PMX-Version: 4.7.1.128075
X-from-outside-Cisco: 128.107.243.14

Dino,

   Pedro.

-------------------------------------------------------------------------------

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Oct 2005 02:12:00 -0700

Hey Pedro,

> management point of view.

Why is this non-trivial ? AFAIK we can easily deal with few hundred BGP 
sessions on RRs or route servers ? Such a box could be owned by customer 
itself or even provider as a value add service. From CE point of view it 
is the same single bgp session ... if it terminates on PE or some server 
it does not really matter that much ;).

> think we can agree on that one. That is a necessary control plane 

Sure ! AFAIK there are at least few ways to securely auto discover CEs 
memberships in a given VPN which naturally would be followed by auto 
participation. In fact I could argue that it could be made much more 
easy then 2547 ...

In MPLS-VPNs you pretty much need to peer to the PE assigned to you by 
service provider. The same service provider needs to preprovision it for 
you etc ... Imagine a scheme where you preload CE with few information 
allowing it to auto discover not only it's peers, but also VPN topology 
as well as peering device to use for control plane. AFAIK such a scheme 
were highly demanded by many 2547 service providers to reduce their per 
new site addition/deletion operation costs.

> helps the SP to do this design function once on the core, instead of per 

Well I am not 100% sure on that one. One of the main issue here is 
dealing with combined vpnv4 routes as one single entity. Many efforts to 
partition this in order to be able to handle it within provider networks 
are still effective mostly on ppt slides :).

Even our most modern rt-constrain way has a potential to help in the 
cases where intra domain you have decent number of rr clusters as well 
as VPN membership is dense. In the case of VPNs spanning entire country 
gain is pretty much equal to zero as all RRs would need to handle all 
routes anyway. All other methods to manually partition vpnv4 RRs just 
failed suffering non trivial operational complexity.

If on the other hand you look at the problem from different angle and if 
you decide not to turn relatively small buckets of IPv4 routes from each 
customer VPN into one huge vpvn4 barrel you could distribute those 
buckets among many RRs/route servers just fine. No additional machinery 
to help you partition something which does not need to be combined in 
the first place will be needed.

Also notice that the argument of just adding more RRs following "divide 
and conquer" principle does not work here as the first part of the 
equation "divide" becomes not a trivial exercise.

> setups. I don't believe they do so out of ignorance.

Well I would rather lean towards recommendation of some flavor of CSC or 
just pure CE to CE VPNs.

I think when we were rolling 2547 it picked up for a couple of main 
reasons ...

* SPs needed new revenue streams,
* Foo over MPLS was sexy,
* Tunneling over mpls was supported by many vendors faster then 
tunneling over any of the IP based encapsulation,
* Most CEs did not had enough of power for IPSec hardware based encryption

Most of the above reasons (except first one :))) are gone and I think it 
would be nice to look at the problem space with new fresh look.

> layers if you want routing to actually follow some reasonably sensible 

I don't quite agree that VPN routing has to have any visibility into VPN 
data transmit layer.

> are replacing. 

It would be nice if they would be so actually :) All reports in that 
space quite fail to observe and compare effects of compromising PE 
router vs compromising frame/atm switch by a hacker or attacker.

When you are on the traditional L2 switch you can't do much harm to your 
customer VPNs running over PVCs. You may shut some circuits, cross 
connected them all which will cause properly configured routing on 
customer sides to drop adj.

When you access PEs you can do whatever you like .. telnet/ssh to any 
device of all VPN networks given PE is serving, sniff the traffic of 
customers or if you are a bit skilled in the art type few lines of 
script and run it from the PE's OS shell ;). CE-PE encryption does not 
help and CE-CE encryption brings you back to CE based VPNs provided 
proper auto discover is in place.

Cheers,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 24 Oct 2005 10:15:59 -0700

Hi Richard,

Few comments from me .. I am sure Dino will send his points as well ;).

> All solutions/technologies have scaling limits that when reached
> processing/bandwidth resources need to be increased or an alternative
> solution/technology needs to be selected. Please can you elaborate as
> to what the scaling limits are when using PE-based VPNs that cannot
> be overcome by increasing network resources incrementally as the
> number of customers increases, e.g. adding new route reflectors?

Growing number of RRs as proven in many networks including BT's do not 
solve the issue at all simply due to the fact that easy vpnv4 route 
partitioning versus right scheme of per given VPN PE homing is very 
difficult to achieve in practice.

If you have many VPNs which are served by all RRs what have you gained ? 
Do you have at least any examples of real world operators managing VPNv4 
route partitioning among different RR clusters with some success ?

And that is just a tip of the iceberg. Growing number of PEs means full 
mesh of LSPs between them. While with less then 1K it is not an issue 
when you talk about many more PEs it getting you close to cliff effect 
as no one yet measured number of LDP LSPs which are possible to operate, 
no one did real study of the impact to 2457 of network instabilities in 
your customers etc ... (hint: blind dampening is not what some customers 
would expect from a service).

> Really? I think there may be a few operators that will disagree with
> you. Again, please can you explain what the scaling limits are and
> why they can't be overcome by increasing network resources?

Sure everything can be overcome by increasing network resource :) And no 
one will be more happy then vendors ... Maybe few folks just think too 
much out of the box here trying to help those which do not need the help 
:) ?

Or maybe there is a significant group of customer who would like to peer 
to 2-3 cheap DSL providers with few orders lower prices which they would 
otherwise pay for a "2457 VPN service" pick the best path out of their 
sites depending on dynamic measurements, clearly allowing them to 
achieve the same (or better SLAs) which they get today for $$$$ and 
automatically create full and really secure interconnectivy of their 
sites ?

> So?.. Distributed scaling is not unique to the CE based approach, PE
> based VPNs can also scale in a distributed manor. For example, route
> reflectors can be added as the number of customer routes increases
> and new blades can be added to PE chassis incrementally as customer
> demand increases.

You like those RRs don't you :) ? Please describe in detail how do you 
in practice partition your VPNv4 routes ? How your operations team 
consult to which RR provision which PEs provided that such a PEs carry 
more then handful of VPN customers ???

> Please can you explain how the CE-based approach is more
> cost-effective than the PE-based approach? You don't need to pay for
> a "huge honkin PE box" from the outset. PE chassis are relatively
> cheap and blades can be added incrementally as customer demand
> increases. Putting device equipment costs aside, how do you think
> bandwidth costs/efficiency for multicast traffic compares using the
> CE-based approach (where N streams will need to be carried for N
> receivers on the source access link)?

That has been already explained above. Do you charge the same for VPN 
service as one ordinary non business residential customer pays for basic 
DSL to his home ?

What we are saying that this is what the VPN price should be. If that is 
real that is super ! If not this is what we mean by "cost effective". 
The CE routers capable of good IPSec and few BGP session can be 
purchased in Fry's today.

> Now you've lost me... What sort of CE-based VPN do you know of that
> doesn't require CE-PE encryption? If there isn't any encryption and
> the traffic is being carried over a shared infrastructure then the
> provider somehow needs to provide VPN isolation, which requires
> partitioning of resources at the edge (PE), i.e. a PE based VPN is
> required.

The VPNs we have in mind do not have even have a concept of PEs. ;).

Cheers,
R.