Subject: [rbridge] MPLS forwarding

From: Kris Price <[email-masked]>
Date: Wed, 04 Mar 2009 19:53:26 +1300

Hi,

So, further to my previous email about MPLS forwarding: I've been unable 
to get the subject out of my head, and after pondering it at some length 
I thought I would summarise the basics here, in case it interests other 
people too, who think it's worth discussing (perhaps elsewhere), or who 
have considered it and can share the flaws with me. I apologise for any 
controversy.

It seems to me that RBridges could be achieved using MPLS encapsulation 
and forwarding. It may resemble VPLS in many ways, but would use new or 
adapted control-plane protocols operating at layer-2 instead of layer-3.

Such a system would share many similarities to RBridges and the problems 
that TRILL has already solved (or is solving) for RBridges, but the 
encapsulation and forwarding would be different.

Why MPLS? Mainly for the re-use of existing hardware. I assume adding 
new control-planes is much simpler as it is predominantly a matter of 
software development. Re-using hardware would (again, I'm assuming) 
reduce development costs and the time to market.

Also, MPLS seems to have won the race in the transport area, so there 
may be practical advantages to consolidating on using MPLS hardware for 
aggregation and campus as well. (Rather than splitting effort.) But 
that's well beyond me to say.

In summary:

Routers run a link-state protocol, building a topological database from 
which they identify the shortest paths to other routers. Routers set up 
label switched paths (LSPs) to the other routers.

Routers use either control-plane or data-plane learning of reachability 
information. In control-plane learning, routers learn the MAC addresses 
of active end stations on attached links, as a normal IEEE bridge would, 
and distribute this over the link-state protocol so that sending routers 
can identify the appropriate LSPs on which to forward frames. In 
data-plane learning, each router learns the appropriate LSPs by 
examining packets during decapsulation.

When a frame from an end station arrives at a router, the router looks 
up the destination MAC address, identifying the egress router and 
appropriate label switch path. The router encapsulates the frame in an 
MPLS header, and the encapsulated frame is then label switched across 
the network to the egress node. At the egress node the label is popped 
and the frame forwarded out the appropriate interface.

Single-destination frames would be forwarded on multipoint-to-point 
(MP2P) LSPs. Point-to-point (P2P) LSPs do not scale as well. For 
single-destination forwarding, scaling is essentially O(n) using MP2P 
LSPs compared to O(n^2) using P2P LSPs.

It appears from the experience of TRILL that data-plane learning is a 
desirable feature. To achieve this, there are two options apparent to me 
that could be explored:

The first is to place an extra label in the stack when encapsulating a 
frame in MPLS, which can be used by the egress router to identify the 
ingress router. In essence these are P2P LSPs tunnelled over the MP2P LSPs.

The second is to look at using some kind of MAC-in-MAC encapsulation 
before the MPLS encapsulation, with the outer MAC header identifying the 
source and destination addresses of the routers in question (where only 
the source address is actually relevant). The egress router, after 
popping the MPLS label, would learn the ingress router via this outer 
MAC header.

Multi-destination forwarding is naturally trickier. Trees would need to 
be set up, either shared trees as per RBridges, or source-based trees 
similar to PLSB and approaches considered for VPLS. Additional trees 
would be set up for constrained distribution (more later).

With multi-destination forwarding there are so many ways to skin it, and 
it depends on many factors such as: how VLANs and IP multicast is to be 
handled; trade offs between bending the rules in the MPLS architecture, 
any requirements for data-plane learning, and balancing the intricacy of 
the control-plane.

VLANs could be managed in a couple of ways:

Some form of multi-topology routing could be used to discover the 
topology of each VLAN, and when transmitting across each link the outer 
MAC header would contain the VLAN tag of the topology in question, on 
receipt the router can use this additional information to classify the 
packet. This can be seen to increase the MPLS label space 
("per-subinterface") and constrain the forwarding of frames in a VLAN to 
links which are in those VLANs. Directly coupled router to router links 
would simply be full 802.1Q trunks.

Another option is not to be concerned about constraining frames to only 
those links which allow those VLANs, and each router advertises VLAN 
membership, with constrained distribution trees established for each 
VLAN. The outer VLAN tags used on each link wouldn't be relevant.

IP multicast can also be optimised with constrained trees, and the mcast 
group membership information distributed by the link-state protocol.

I did discover that MPLS encapsulation was proposed for RBridges in 
draft-bryant-perlman-trill-pwe-encap-00, which from my understanding 
violates the MPLS architecture, because label assignments are made 
upstream, but that upstream allocation does certainly make things easier 
for the control-plane in a system like this (if it's a violation you can 
live with).

The draft does discuss how the paths would be established and it is 
possible to conceive ways of establishing additional constrained trees 
for VLANs and IP multicast optimisation too. However, after cursory 
examination of methods for this, it would seem to require routers to do 
SPF computations for all routers in the network. A router would need to 
know its position in the tree to decide if it needed to install relevant 
forwarding state. This may be considerable overhead, but that does not 
seem to bother the people behind PLSB, and I do not have the wisdom to 
declare whether it is or is not.

A signalling protocol in the control-plane (perhaps separate, or perhaps 
carried on the link-state protocol) could remove the need to do these 
SPF computations, and remove the upstream assignment of labels, making 
it compliant to the MPLS architecture. But this would be a trade off in 
that the resulting protocol might be considered more intricate.

That is the sum of it. The rest gets into the details of weighing these 
approaches against each other and designing the protocols.

Again, I apologise for any controversy, unfortunately I've no one who 
understands these subjects well enough that I can bounce the idea off. I 
just wanted to put it out there with a slightly better explanation in 
case there were likeminded people who thought it worthy of discussion 
(perhaps in another forum) or were able to open my mind to the flaws in it.

Regards

Kris Price

_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: James Carlson <[email-masked]>
Date: Wed, 4 Mar 2009 09:38:54 -0500

> have considered it and can share the flaws with me. I apologise for any 

I and others have had similar thoughts.  And the hardware reuse is
very attractive indeed.

> ingress router. In essence these are P2P LSPs tunnelled over the MP2P LSPs.

Yes; that's what I'd expect you'd need to do.  It should work, though
it does mean that the ingress somehow (through signaling) needs to
know what label the egress needs to use as the outer label when
sending data in the opposite direction.

It's just not what the working group long ago adopted.  That's the key
problem with the approach.

I realize that doesn't sound like a promising answer, but we're years
down the path of getting consensus.  Developing a brand new protocol,
at least in *this* working group, doesn't seem like a viable option.

> popping the MPLS label, would learn the ingress router via this outer 

That doesn't sound as good to me.  The egress learning function needs
to have a clear path back to the ingress.  In MPLS, that means a
label.  If you're doing MAC-in-MAC, this means that the egress needs
to have some sort of table that maps source MAC addresses back to the
labels that are actually needed, which sounds far more complex than
just including the information as a stacked ready-to-use label.

> that the resulting protocol might be considered more intricate.

Actually, I think the SPF computations are a *good* thing.  The only
issues would be carrying labels in the LSPs instead of nicknames,
determining what to do about any MPLS-only (no RBridgishness) portions
of the core network, and figuring out how label distribution ought to
work.

-- 
James Carlson, Solaris Networking              <[email-masked]>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 04 Mar 2009 10:48:21 -0800

> reduce development costs and the time to market.

Kris,

It has been a while since the MPLS encapsulation was discussed in the WG 
but there are two key things I remember as being issues:
  - TRILL wants a hopcount in the header for loop safety
  - For multipath of multicast (or multi-destination frames in general) 
TRILL needs the ingress rbridge in the header

While one could extend the MPLS encapsulation for TTL, and one can 
somehow use the tag as a tag for the ingress for multi-destination 
frames, doing so probably means that existing MPLS hardware couldn't be 
reused.

You might be able to find more details on the discussion we had in the 
WG email archives.

    Erik

---

From: James Carlson <[email-masked]>
Date: Wed, 4 Mar 2009 14:12:52 -0500

>   - TRILL wants a hopcount in the header for loop safety

MPLS already makes a provision for determining the MPLS TTL from some
other procedure; section 2.4.4 of RFC 3032.

> TRILL needs the ingress rbridge in the header

A stacked header does that reasonably well.

> frames, doing so probably means that existing MPLS hardware couldn't be 

I don't think that part is true.  At least I *know* it's not true for
the hardware that was designed at IronBridge.  It was capable of
pushing multiple labels at ingress and popping multiple at egress,
precisely to support tunnels-in-tunnels.

As for those in the middle of the proposed scheme (the network core),
all they need to do is what any LSR needs to do: look at the outer
label and forward, swapping as usual.

Thus, it would be compatible with existing machines, and is an
interesting thing to explore.  I just don't think it's something that
would be explored in this working group.

---

From: Kris Price <[email-masked]>
Date: Thu, 05 Mar 2009 14:30:43 +1300

[Oops. Resending to list from correct address.]

Hi Erik,

> TRILL needs the ingress rbridge in the header

As I see it, multi-destination would have to be handled using either
point-to-multipoint (P2MP) or multipoint-to-multipoint (MP2MP) LSPs.

*MP2MP*

A MP2MP LSP delivers packets amongst routers in a group. Any router in
the group can transmit a packet and it will be received by all of the
other routers in the group.

I am partial to this, because in many properly designed networks there
are usually only a few such shared trees that are needed.

Identifying the ingress router is a problem with MP2MP LSPs because a
router will recieve packets from all of the members in the group with
the same incoming label. Label stacking could be used but pretty much
requires a special case violation of the MPLS arch to permit upstream
assignment. In that case each router advertises its identifying label
for other routers to install the appropriate state for.

Note the reason for upstream assignment is of course that the same MPLS
packet is sent to multiple downstreams -- so of course there is no way
for it to have a different bottom label for each downstream it reaches.

If this upstream assignment is seen as too ugly, but MP2MP LSPs were
determined necessary, then it is back to some kind of MAC-in-MAC-o-MPLS
possibility. Perhaps think of it like each router has an 802.1ah bridge
layer, that out one side connects to the Ethernet LANs, and the other to
other routers via this MPLS system. (Uglier perhaps.)

Or data-plane learning is dropped should MP2MP LSPs be used.

*P2MP*

A P2MP LSP delivers packets from a single ingress router to a group of
egress routers. This has the advantage that when a router recieves a
packet on the LSP, it knows which source it came from. (Depending in
part on how the LSP is set up.)

At a minimum, there would need to be a default P2MP LSP rooted on every
router, so that broadcasts/unknowns reach all other routers. From there
constrained trees could be set up in any number of ways and it opens up
into a big topic.

Pro: downstream assignment can be used, and data plane learning still
possible.

Con: potential scalability constraints with downstream assignment
dependant on requirements.

> You might be able to find more details on the discussion we had in the 

I did try searching for anything relevant in the archives, but seems my
search foo is just not up to it.

Question: is there any (rough or otherwise) scalability requirements
someone can outline or point me to for systems like this and RBridge?
Maximum number of bridges or end-stations, that kind of thing.

Cheers
Kris