Subject: Re: [rbridge] Orphaned endnodes with partitioned VLANs on a cloud

From: [email-masked]
Date: Sun, 09 Dec 2007 12:53:29 +0800

Russ, I think a way of thinking about it that might be clearer is first to pretend there are no VLANs.
We want there to be just a single RBridge that forwards traffic to/from the link. 

Let's say that there are two RBridges, RB1, and RB2 that both think they should forward traffic to/from
the link, not realizing they are both connected to the same link (because bridges are blocking the data
traffic for some reason, or because there is so much congestion that they are not hearing each other's
IS-IS Hello messages).

RB1 encapsulates a multicast packet off the link. It arrives at all RBridges, including RB2. RB2 decapsulates it
(including losing the TTL), and forwards it on the link. RB1 sees it as a brand new packet, encapsulates it and
forwards it into the campus...and so forth.

The original design was a bit easier to understand -- where just the DRB forwarded traffic to/from the link
on behalf of all VLANs. The VLAN tag was only a way of keeping endnodes separated from each other, as
opposed to a policy decision on inter-RBridge links. The WG concluded that it is simpler and more robust
to assume that all inter-RBridge links are allowed to forward traffic on behalf of all VLANs, and if anyone
wants anything fancier than that, they can use all the traffic engineering that layer 3 people have years of
experience with, to move different types of traffic (whether it be based on VLAN, or deeper inspection into
the packet).

The extra thing -- load splitting -- seems to have been confusing some people, but it works if there is a single
RB forwarding traffic from a particular link on behalf of a VLAN.

But anyway, now let's look at VLANs on the link. Because of bridges, any VLAN might be partitioned with
respect to data traffic, for various reasons.  The WG discussed this at length, and it looked to me, and I strongly
agree with this -- that TRILL should consider partitioned VLANs on a link as a misconfiguration, and should
make sure they are defensive against it (not creating loops), but should not worry about healing
all the partitions.

The RBridges have to stick some VLAN tag on packets when they
talk to each other.  One possibility is to talk with every VLAN tag legal to transit that link.  That is certainly more
overhead than talking with a single VLAN.

And it's good to be really careful of temporary loops, even if they "can't happen". Having two RBs not see each other's
Hellos can happen due to congestion, due to RBridges that can't handle 4000 Hellos from each other RBridge
because they are tagging every Hello with every possible VLAN tag, and sending 4000 times as often, ...

If we attempt to heal a partition of VLAN A by having two forwarders off the link, it removes our ability to be very
cautious by reporting the root bridge, since although VLAN A is partitioned for data traffic, the spanning tree for
VLAN A is not necessarily partitioned, so the two RBridges will both see and report the same root bridge.

So, given the two possibilities:
a) Hellos on every possible VLAN by every RBridge. DRB elections on each possible VLAN. Different pseudonodes per link for each
possible VLAN (with some attempt to consolidate when possible). Not being able to be conservative about loops
(since the "report the root bridge" will not work). Multicast traffic might traverse the traffic multiple times (because
each pseudonode is logically a different node in the topology).
b) DRB sends lots of Hellos (to be as robust as possible about preventing multiple forwarders within the cloud,
although if that doesn't work, the next level of safety -- seeing the root bridge in the LSP), but all other RBridges
just send a single Hello. Single pseudonode. Only downside -- if the cloud is configured to have a particular VLAN
partitioned, then endnodes get orphaned.

I did suggest an enhancement to b) that I don't think is that expensive or complex, to at least detect if a VLAN is
partitioned. It will not work (nor would anything work) if there are endnodes that are not in a partition with an RBridge.
The proposal is that the VLAN-x appointed forwarder send Hellos tagged with VLAN x in addition to the common
VLAN announced by the DRB. Then an RBridge that is attached to VLAN x and which is not the designated forwarder,
can notice it is not receiving any VLAN x Hellos.

Personally, I think we should leave the spec the way it is. It was a careful balancing act between overhead and
safety, plus supporting features such as load splitting based on VLANs.

Radia

----- Original Message -----
From: Russ White <[email-masked]>
Date: Saturday, December 8, 2007 1:44 am
Subject: Re: [rbridge] Orphaned endnodes with partitioned VLANs on a cloud

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> > Not trying to "fix" the failure mode is exactly what the current 
> thing> is doing: it favors simplicity and scaling, and not
> > trying to heal partitions within a cloud, but instead making 
> sure that
> > if there is a partition, it won't cause loops.
> 
> Loops on the edge ports are a problem--I'm cogitating on a set of
> solutions for that which don't require the complexity involved in the
> proposed solution. IMHO, the specification should be flexible 
> here, and
> allow tradeoffs.
> 
> I'm not getting where a loop can form due to a link failure in the
> underlying transport between rbridges. Could someone provide an 
> example?
> :-)
> 
> Russ
> 
> - --
> [email-masked] CCIE <>< Grace Alone
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFHWYZuER27sUhU9OQRAlj5AJ97kpulWWm57hvd9QkX69WrwWGAgQCfZpnb
> FlEWpN8YIhUQUo3DHYuo3TM=
> =Vvu1
> -----END PGP SIGNATURE-----
> 

_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Russ White <[email-masked]>
Date: Sun, 09 Dec 2007 14:27:22 -0500

> We want there to be just a single RBridge that forwards traffic to/from the link. 

What you're talking about here is edge ports, though, not links in the
middle of the cloud--I think there's a radical difference between the
two. The edge case can be solved more easily than what you describe
below--I'm waiting on some validation from some other folks before
sending out a list of ideas to the list.

As for the link in the cloud, I don't think this is a case that we
should worry about.

> make sure they are defensive against it (not creating loops), but should not worry about healing

The current proposal heals partitioned VLANs, from what I understand of
it, intentionally--at least on the links in the cloud. This is what I
specifically think we should avoid doing.

Let's begin with this: rbridges don't care about the state of the links
between rbridges, they only care to solve possible loops for edge ports.

> talk to each other.  

Actually, the don't, from what I understand. They could talk native on
the media, or they could use a vlan, depend on how they're configured.

> partitioned, then endnodes get orphaned.

There are a number of other options, but again, I'm waiting on someone
else to look at them before sending them to the list.

> safety, plus supporting features such as load splitting based on VLANs.

The problems with the current mechanism are:

1. It doesn't solve what it was intended to solve.
2. When it does work, it has unintended side effects.

Again, things are much too mixed up right now--solve one problem, and
one problem, only. Ignore links between rbridges that are designed to be
host free, or transit only. Only worry about edge ports. Don't worry
about misconfigurations, in terms of trying to find a path when
something is misconfigured, etc.

:-)

Russ

- --

---

From: "Eastlake III Donald-LDE008" <[email-masked]>
Date: Mon, 10 Dec 2007 22:44:15 -0500

Hi Russ,

See below at @@@

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Russ White
Sent: Sunday, December 09, 2007 2:27 PM
To: [email-masked]
Cc: Developing a hybrid router/bridge.
Subject: Re: [rbridge] Orphaned endnodes with partitioned VLANs on a
cloud

@@@ First, I think Rbridges should work for general topologies. Second,
our charter speaks of incrementally deployablity. You may be
incrementally deploying Rbridges into what was a bridged network. You
could eventually end up with all Rbridges. But many intermediate states
would have bridged LANs between the Rbridges. Some phrases extracted
from our charter: "... arbitrary topologies ... ensure that the solution
can interwork with bridges in a flexible manner ... incremental
deployment into LANs ...". So I think we do need to worry about that
case, although I don't see anything wrong with being able to configure a
link to be a point-to-point link between Rbridges, provided you also
include some reasonable sanity checks so if you receive frames
inconsistent with the link being a direct point-to-point connection
between Rbridges you will notice and take appropriate action.

> respect to data traffic, for various reasons.  The WG discussed this
at length, and it looked to me, and I strongly

@@@ It's not that simple. Healing partitioned VLANs can mean two things:
partitioned on a link or partitioned between different links, where
"link" may be a bridged LAN.

@@@ The current proposal does not heal a VLAN that is partitioned on a
single link. As far as I know, that could only happen due to the link
being a bridged LAN and the bridge or bridges within it being configured
to partition the VLAN. Since, for simplicity, there is only one Rbridge
forwarding on-to/off-of the link for any particular VLAN, such a
partitioning is considered a misconfiguration and can result in orphaned
nodes.

@@@ The current proposal does heal a VLAN that is partitioned between
different links.  That is, the incremental deployment of Rbridges may
cause what were separate islands of a VLAN in a bridged network to
become connected. The original existence of such islands can also be
considered a misconfiguration or at least a violation of the spirit of
802.1Q, although I'm sure you can find those who would dispute this
view. This island interconnection just happens automatically because of
the global view Rbridges have through the link state database. It would
require more complexity and configuration to keep such islands separate.
Since Rbridges are routing on VLAN ID and MAC address, how would they be
supposed to know there are supposed to be two separate islands of VLAN
X? Seems like you would need to extend the VLAN ID with an island ID or
something. Given Rbridges are required by our charter to be able to
capable of operating with "Minimal or no configuration" the WG has thus
far decided not to go with such "VLAN island IDs" or any similar
mechanism, which in any case would only work with extra configuration
and be more complex.

@@@ If a bridged LAN link between Rbridges is misconfigured to partition
a VLAN, it isn't healed because to do so would make things more complex
and less safe. And if a VLAN is partitioned between separate links that
connect to Rbridges, it is healed because to not do so would make things
more complex and would require additional configuration.

@@@ None of these questions arise, of course, in an all Rbridge LAN or
one where any remaining bridges are configured to provide VLAN
connectivity.

@@@ As above, Rbridges must consider the case where the links between
Rbridges are bridged LANs. Also, it is my impression that a number of
TRILL WG members consider loops to be so harmful that Rbridges need to
take reasonable precautions against them everywhere, not just on "edge"
ports, whatever you consider such ports to be. So I don't see how we can
begin where you suggest we begin.

@@@ Well, they could emit untagged TRILL frames. Note that you are using
the word "native" in a different way that it is used in the TRILL WG and
protocol document. Here "native" currently means any data frame that
isn't encapsulated with the TRILL Ethertype, as opposed to a TRILL frame
which does have the TRILL Ethertype encapsulation. Whether or not the
frame is VLAN tagged is orthogonal to whether or not it is native in
this sense.

@@@ Also, assuming an 802.1Q sort of network, conformant with Annex E of
802.1Q-2005, then I think it is reasonable to think of all frames,
whether they are VLAN tagged or not, as being in/on a particular VLAN.
It is certainly the case that data frames inside an 802.1Q bridge always
have a VLAN ID associated with them. If the frame was untagged on the
wire but was admitted, then the port through which it was admitted
associated a VLAN ID with that frame. And, if the network is Annex E
conformant, all ports attached to that wire which will admit untagged
frames are required to be configured so as to associate the same VLAN ID
with the frame. So it might be reasonable to consider the frame to be in
that VLAN even when it is transiting untagged on the wire.

...

> Personally, I think we should leave the spec the way it is. It was a
careful balancing act between overhead and

@@@ I think it does a reasonable job of solving the problem it was
intended to solve: "... a solution for shortest-path frame routing in
multi-hop IEEE 802.1-compliant Ethernet networks with arbitrary topology
..." I think that, when combined with the incremental deployment
requirement, necessitates consideration of intervening bridges.

@@@ How is anyone supposed to know what side effects you are talking
about?

...

@@@ Thanks,
@@@ Donald

Russ

- --

---

From: "Eric Gray" <[email-masked]>
Date: Tue, 11 Dec 2007 14:44:27 -0600

Donald,

	Unfortunately, I have to pack my computer for a trip and
cannot go into too mych detail right now, but I would like to 
make just a few short comments on this thread.

	Please see a few comments in-line below...

--
Eric Gray
Principal Engineer
Ericsson  

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Eastlake III 
> Donald-LDE008
> Sent: Monday, December 10, 2007 10:44 PM
> To: Developing a hybrid router/bridge.
> @@@ First, I think Rbridges should work for general topologies. 
>	Second, our charter speaks of incrementally deployablity. 
>	You may be incrementally deploying Rbridges into what was a
>	bridged network. You could eventually end up with all Rbridges. 
>	But many intermediate states would have bridged LANs between
>	the Rbridges. Some phrases extracted from our charter: 
>	"... arbitrary topologies ... ensure that the solution can 
>	interwork with bridges in a flexible manner ... incremental
> 	deployment into LANs ...". 
>	So I think we do need to worry about that case, although I 
>	don't see anything wrong with being able to configure a link 
>	to be a point-to-point link between Rbridges, provided you also
>	inconsistent with the link being a direct point-to-point 
>	connection between Rbridges you will notice and take appropriate

>	action.

Yes, ID is a goal.  The current specification is less than adequate
in this respect.  Significantly.

> 
> > But anyway, now let's look at VLANs on the link. Because of bridges,
> > any VLAN might be partitioned with respect to data traffic, for 
> > various reasons.  The WG discussed this at length, and it looked to
> > me, and I strongly agree with this -- that TRILL should consider
> > partitioned VLANs on a link as a misconfiguration, and should
> > make sure they are defensive against it (not creating loops), but
> > should not worry about healing all the partitions.

This is a departure from the intuitive (and common use) meaning of
the term "virtual LAN" in that this assumption does not hold true 
for the current specification if I replace VLAN with LAN.  In that
case, I would expect the fact that there may be multiple RBridges
with access to a LAN segment to "heal" a partition - simply because
they could no longer be neighbors with respect to that LAN and this
would trigger a DRB election and HEAL the LAN.

The same thing MUST be true for any VLAN.

> 
> The current proposal heals partitioned VLANs, from what I understand
> of it, intentionally--at least on the links in the cloud. This is what
> I specifically think we should avoid doing.
> 
>	partitioned on a link or 
>	partitioned between different links, where "link" may be a 
>	bridged LAN.
> 
> @@@ The current proposal does not heal a VLAN that is partitioned 
>	on a single link. As far as I know, that could only happen due 
>	to the link being a bridged LAN and the bridge or bridges within

>	it being configured to partition the VLAN. Since, for
simplicity, 
>	there is only one Rbridge forwarding on-to/off-of the link for 
>	any particular VLAN, such a partitioning is considered a 
>	misconfiguration and can result in orphaned nodes.

Wrong.  In the example I gave, partitioning occurs because of a bridge
failure.  One or more link failures would have the same effect.

> 	different links.  That is, the incremental deployment of
Rbridges 
>	may cause what were separate islands of a VLAN in a bridged 
>	network to become connected. The original existence of such 
>	islands can also be considered a misconfiguration or at least a 
>	violation of the spirit of 802.1Q, although I'm sure you can
find 
>	those who would dispute this view. This island interconnection 
>	just happens automatically because of the global view Rbridges 
>	have through the link state database. It would require more 
>	complexity and configuration to keep such islands separate.
>	 Since Rbridges are routing on VLAN ID and MAC address, how
would 
>	they be supposed to know there are supposed to be two separate 
>	islands of VLAN X? Seems like you would need to extend the VLAN 
>	ID with an island ID or something. Given Rbridges are required
by 
>	our charter to be able to capable of operating with "Minimal or 
>	no configuration" the WG has thus far decided not to go with
such 
>	"VLAN island IDs" or any similar mechanism, which in any case 
>	would only work with extra configuration and be more complex.
> 
> @@@ If a bridged LAN link between Rbridges is misconfigured to 
>	partition a VLAN, it isn't healed because to do so would make 
>	things more complex and less safe. And if a VLAN is partitioned 
>	between separate links that connect to Rbridges, it is healed 
>	because to not do so would make things more complex and would 
>	require additional configuration.

This is because you assume a model for VLAN interactions that is 
different from the model that necessarily applies for LANs.

In fact, the ideal model would not differ and the realizable model
only differs by the fact that VLANs must be grouped to share a
spanning tree (because of the scalability limitation in MSTP).

> 
> @@@ None of these questions arise, of course, in an all Rbridge LAN 
>	or one where any remaining bridges are configured to provide
VLAN
>	connectivity.
> 
> Let's begin with this: rbridges don't care about the state of 
> the links between rbridges, they only care to solve possible loops 
> for edge ports.
> 
> 	Rbridges are bridged LANs. Also, it is my impression that a
number 
>	of TRILL WG members consider loops to be so harmful that
Rbridges 
>	need to take reasonable precautions against them everywhere, not

>	just on "edge" ports, whatever you consider such ports to be. So
I 
>	don't see how we can begin where you suggest we begin.
> 
> @@@ Well, they could emit untagged TRILL frames. Note that you are 
>	using the word "native" in a different way that it is used in 
>	the TRILL WG and protocol document. Here "native" currently
means 
>	any data frame that isn't encapsulated with the TRILL Ethertype,

>	as opposed to a TRILL frame which does have the TRILL Ethertype 
>	encapsulation. Whether or not the frame is VLAN tagged is 
>	orthogonal to whether or not it is native in this sense.
> 
> @@@ Also, assuming an 802.1Q sort of network, conformant with 
> 	Annex E of 802.1Q-2005, then I think it is reasonable to think
of 
>	all frames, whether they are VLAN tagged or not, as being in/on
a 
>	particular VLAN.
>	It is certainly the case that data frames inside an 802.1Q 
> 	bridge always have a VLAN ID associated with them. If the frame 
>	was untagged on the wire but was admitted, then the port through

>	which it was admitted associated a VLAN ID with that frame. And,

>	if the network is Annex E conformant, all ports attached to that

>	wire which will admit untagged frames are required to be 
>	configured so as to associate the same VLAN ID with the frame. 
>	So it might be reasonable to consider the frame to be in
> > careful balancing act between overhead and safety, plus supporting 
> > features such as load splitting based on VLANs.
> 
> 	intended to solve: "... a solution for shortest-path frame
>	routing in multi-hop IEEE 802.1-compliant Ethernet networks with

>	arbitrary topology ..." I think that, when combined with the 
>	incremental deployment requirement, necessitates consideration 
>	of intervening bridges.
>