Subject: [rbridge] potential L2 forwarding loop issue in RBridges

From: James Carlson <[email-masked]>
Date: Thu, 12 Feb 2009 14:19:43 -0500

I've discovered something in my testing of our TRILL/RBridge
implementation that I think might be significant.  If I'm right, it
means that something is missing from our current protocol draft
(likely section 4.2.3.3).

An RBridge by itself (no other RBridges in sight on the wire) will
become DRB on every link it has and, since it sees no other RBridge,
will also become AF for all configured VLANs on each link.

It will begin learning and forwarding on all of those links.
Logically, since there are no other RBridges to talk to, it will (at
least as viewed from outside; the internal design doesn't matter)
simply forward L2 packets from one link to another.

Now suppose we have two RBridges connected like so:

	       hub1
	  |+----------+|
	   |          |
	+--+--+    +--+--+
	| RB1 |    | RB2 |
	+--+--+    +--+--+
	   |          |
	  |+----------+|
	       hub2

What should happen is that RB1 and RB2 will elected a DRB on each one
of those networks (hub1 and hub2), resulting in either L2 forwarding
through just a single RBridge (if AF is the same RBridge on each
network), or in a trip through TRILL encpasulation (if RB1 is AF on
one network and RB2 is AF on the other).

So far, so good.  However, what happens if RB1 and RB2 cannot see each
other because of MTU restrictions in hub1 and hub2?

This is where things seem to get ugly.  If we're using the normal
MTU-padded IS-IS packets for TRILL Hello messages, then those will get
dropped by the hubs.  Both RBridges will declare themselves to be DRB
and AF for everything, and both will begin forwarding L2 frames.
Neither sees the other at all, except that for all frames sized to the
restriction or less, both will attempt to forward.

The loop avoidance in 4.2.3.3 doesn't apply, because the Hellos aren't
getting through; they're choked on MTU restrictions.  The "root bridge
collision check" doesn't apply, because we're not actually
decapsulating a frame from some other RBridge.

Chaos then ensues.

Conceptually, if we were (in this stand-alone L2 forwarding between
ports case) pretending to encapsulate in TRILL and then immediately
decapsulating, then the collision check would apply, but it wouldn't
actually work, because there *isn't* any 802.1D bridged connection
between hub1 and hub2, and thus they do have completely different root
bridges.

This is a more significant problem to me because we will naturally be
dealing with unusual packet sizes for TRILL.  The path between TRILL
RBridges needs to be at least 24 octets bigger than it is for regular
bridges, and if we're to avoid accidentally trying to use unsuitable
paths, such as ordinary (non-jumbo-enabled) switches and hubs, then we
need to check for them, which is what IS-IS's MTU padding does.

In routing, the IS-IS Hello failure means that forwarding for
non-local destinations _stops_ through that link, which means that
loops can't form.  Unfortunately, TRILL defines things such that if
IS-IS Hello fails, forwarding _starts_ through that link, and since
(unlike routing) bridging doesn't distinguish between local and
non-local, loops become possible.  We have it backwards.

I can think of at least two ways to avoid this problem, but I think
I'd like to hear from the group before going ahead with a formal
description of one.

  A.  Require TRILL IS-IS implementations to send very small Hello
      messages, not MTU-padded, and with as little information as we
      can manage.  (If we need "large" Hellos, then send small ones
      occasionally as well.)  This allows the loop avoidance part to
      do its work.

      (Or perhaps the MTU that IS-IS uses should not be the actual MTU
      needed along the path with TRILL overhead, but the MTU for data
      packets, since IS-IS is now going unencapsulated.  But then this
      means that MTU-restricted paths are unprotected and cannot be
      detected by normal IS-IS operation.  We end up with the black
      hole problem.)

  B.  Require RBridge implementations to include STP as well.
      However, the STP portion in this solution has (by design) no
      effect on TRILL itself.  Instead, we use STP's link "forwarding"
      state to gate the AF behavior in the following ways:

	i.  When STP reports that the link is not in "forwarding"
	    state, we refuse to become AF for any VLAN.  We become AF
	    only if STP reports "forwarding" for the link *and* the
	    DRB appoints us to do it.

	ii. When encapsulating from or decapsulating to native format,
	    examine both the AF flag and the STP state.  Discard the
	    packet if not AF or if STP state is not "forwarding."

Thus, I have at least two questions here:

  1.  Have I correctly identified a real problem?  (If not, then a
      bonus question would be "why isn't it a problem even though I've
      run into this?")

  2.  Assuming it is a problem, which solution do we prefer?

(And perhaps a third question: does it bug anyone else that "MTU"
never appears in this document ... ?)

-- 
James Carlson, Solaris Networking              <[email-masked]>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Eric Gray" <[email-masked]>
Date: Thu, 12 Feb 2009 15:04:25 -0600

James,

	One of the basic things we need to be able to assume in
layer 2 forwarding technologies is that broadcast messages -
especially those associated with topology determination - will
be forwarded by the underlying network.

	What would be the effect we could expect if BPDUs were 
not forwarded?

	Hence, the maximum size of our hello messages has to be
determined from the "real" MTU limitations of the underlying
network.

	Configure it if you have to.

--
Eric

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of James Carlson
Sent: Thursday, February 12, 2009 2:20 PM
To: TRILL/RBridge Working Group
Subject: [rbridge] potential L2 forwarding loop issue in RBridges

	       hub1
	  |+----------+|
	   |          |
	+--+--+    +--+--+
	| RB1 |    | RB2 |
	+--+--+    +--+--+
	   |          |
	  |+----------+|
	       hub2

Chaos then ensues.

---

From: James Carlson <[email-masked]>
Date: Thu, 12 Feb 2009 16:18:19 -0500

> 	What would be the effect we could expect if BPDUs were 

It's effectively the same result, at least from a high level.
However, I'm talking specifically about IS-IS as used for TRILL, not
BPDUs.  The two are different.

> determined from the "real" MTU limitations of the underlying

Sadly, in the configuration I described, the "real" MTU limitation
isn't knowable, at least in any automatic sense.

More generally, the silent MTU problem which leads to black holes in
the network (as was once created by FDDI/Ethernet bridges) is exactly
why IS-IS has this feature.

> 	Configure it if you have to.

I don't think it's merely a matter of configuration.  It may be
possible to configure to avoid the problem on a particular network,
but that renders the solution fragile.  Accidents result in
catastrophe, which is a bad thing for a protocol whose sole purpose in
life is to avoid accidents.

(If everyone could just configure L2 topologies without loops, we
wouldn't need any of this fancy loop-detection stuff.)

---

From: "Anoop Ghanwani" <[email-masked]>
Date: Thu, 12 Feb 2009 18:42:11 -0800

I am not going to argue about whether or not this
is a real problem since you have actually run into it.
It may not be one that would happen to an informed
user (per Eric's response), but then there is always
an ill-informed implementer/deployer out there and if
you run things according to spec, then such problems 
are a no-no. 

>       detected by normal IS-IS operation.  We end up with the black

This is certainly doable, but I'm not sure it's
robust enough.  It almost falls in the category of
what Eric said.

> 	    packet if not AF or if STP state is not "forwarding."

The problem I have with this is that thus far we have been
assuming we terminate STP at RBridges.  If we are to require
that RBridges participate in STP, then are the different ports
considered different instances or the same instance?  If they
are the same instance, we risk creating a giant spanning tree
that goes across RBridges.

Or maybe I'm missing something...perhaps you can provide
more detail on what you meant.

Anoop

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 12 Feb 2009 22:58:26 -0500

Hi Jim,

See below...

> Now suppose we have two RBridges connected like so:

Right. The problem is the low MTU of the hubs and that we are using
Hellos for two different things. Their function related to forwarding
is doing fine. In this example, no forwarding is occurring as
evidenced by the fact that there are no TRILL encapsulated frames on
the wire. However, the other function, which might be called
ingress/egress loop safety, isn't working because the Hellos are not
getting through.

> need to check for them, which is what IS-IS's MTU padding does.

Given all the various tags that 802.1 is specifying, many people would
not consider these frame sizes to be "unusual"...

It is interesting that you found devices that strictly enforce the old
Ethernet frame limit. No one has worried about this because (almost?)
everyone has been assuming that equipment you would actually run into
does not have that limit. I don't know that adding just 24 bytes
actually  constitutes a "jumbo" when 802.3 has now defined a larger
size to allow for tag insertion, etc.

(802.3as-2006 specifies three frame sizes with the following size for
the MAC Client Data transported: basic frame = 1500, Q-tagged frame =
1504, and envelope frame = 1982.)

> non-local, loops become possible.  We have it backwards.

Well, as I say above, I don't think that ingress or egress are
forwarding. they are a function not present with IP or other protocols
that have end station participation. Ingress/egress *have* to be
enabled by the non-receipt of frames (currently Hellos) just as bridge
ports have to be enabled by the non-receipt of frames (BPDUs). After
all, if there is actually no RBridge or no bridge out that port then
it has to sooner or later become enabled for RBridge ingress/egress or
bridge forwarding respectively.

>      detected by normal IS-IS operation.  We end up with the black

Yes, that's one possibility. But it seems like a pain to have two
different Hellos sizes...

>            packet if not AF or if STP state is not "forwarding."

You want to be careful not to imply that the RBridge as a whole is
participating in STP. The bigger the STP the more ports are turned off
and, if STP turns off a port, it becomes unavailable for TRILL
encapsulated traffic.

If you wanted to do this, you would have each RBridge port separately
participate in STP as if there was a virtual bridge between the
RBridge and the link. And if you are going to do this, you might as
well make the virtual bridges be maximum priority for root and just
say that you are DRB on a port if and only if your virtual bridge at
that port becomes root. Then, I think, you can leave Hellos the size
they are now and get rid of all Hellos except on the Designated VLAN.

>      bonus question would be "why isn't it a problem even though I've

If you have run into it, it clearly has some reality :-)

Perhaps people were wrong when they thought that the old Ethernet MTU
would not be enforced by equipment people would actually encounter.
Then again, some people have said things very much like "there are no
such things as hubs any more"...

>  2.  Assuming it is a problem, which solution do we prefer?

I think we need some discussion on this point but maybe this will push
us to the "DRB only if root bridge" position...

> never appears in this document ... ?)

It has been discussed but those who were knowledgeable said they
thought it wouldn't be a problem. Perhpas a caveat should be added
that, since neither IS-IS Hellos nor 802.1 BPDUs are fragmentable,
having components of your network with an MTU small enough to block
them but big enough to let some data through means your network is
toast.

Thanks,
Donald

=============================
 Donald E. Eastlake 3rd   +1-508-634-2066 (home)
 155 Beaver Street
 Milford, MA 01757 USA
 [email-masked]
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 12 Feb 2009 23:01:58 -0500

Hi Anoop, see below

On Thu, Feb 12, 2009 at 9:42 PM, Anoop Ghanwani <[email-masked]> wrote:
> ...

We certainly don't want that. See my response to Jim's initial
message, which suggests what might be called a solution C.

Thanks,
Donald

---

From: Dinesh G Dutt <[email-masked]>
Date: Thu, 12 Feb 2009 21:52:51 -0800

HI James,

I'm not overly concerned with the problem you raise because it is not a 
realistic deployment. However, from a protocol correctness perspective, 
we must specify something to prevent the problem that you mention from 
occurring. I'm not in favor of running STP because one of the reasons I 
hear from customers is that they'll look at TRILL because they hate STP 
and want STP to completely disappear, if they can help it. Adding 
another Hello to deal with this problem seems more complicated than 
required to deal with what is a theoretical problem, at least to me.

I favor your third solution and suggest a minimum MTU in a TRILL 
network. Many L2 protocols that I'm aware of are doing this such as 
802.1Q VLAN and FCoE. We can specify that if the MTU on a link is less 
than this min MTU, TRILL forwarding is disabled (i.e. encap/decap).

Dinesh

-- 
We make our world significant by the courage of our questions and by 
the depth of our answers.                               - Carl Sagan

---

From: "Eric Gray" <[email-masked]>
Date: Fri, 13 Feb 2009 07:45:47 -0600

James,

	If you have a management knob to allow you to configure
the "real" MTU, you could use it if you have to.  You would know
that you have to, in the scenario you describe - either before
hand, or shortly there-after.

	Configuring an MTU for individual interfaces, when it is
needed, is not the same as configuring a topology, as you should 
well understand.  It is in fact, no more configuration effort
than we have already bought into in supporting a variety of 
different VLAN topology options.

	In fact, given the number of things we've already decided
to call "configuration errors", this is simply yet another one.
And that is what I was getting at.

	Yes, the technique you describe almost certainly was meant
to detect the situation where an underlying Data-Link Layer will
drop MTU-sized packets.  But we're not using IS-IS to determine
what will happen in L3 forwarding, because of L2 issues - we're
using IS-IS to determine what will happen in L2 forwarding.

	Hence, two things are true:

1) the padding - if (as you assert) it serves only the one purpose
   of finding L2 forwarding problems - is unnecessary and
2) if it is done anyway (for wahtever reason) it must be based on 
   the effective maximum MTU at the Data-Link Layer.

	I propose that configuring the L2 MTU is one option - to 
serve as a proof of concept that the issue you point out can be
dealt with as is.

--
Eric

---

From: James Carlson <[email-masked]>
Date: Fri, 13 Feb 2009 12:12:37 -0500

> robust enough.  It almost falls in the category of

Agreed; it's not so safe.

> that goes across RBridges.

I was thinking of a single instance.

It's true that it creates a single giant spanning tree, but "so what?"
The only case where this spanning tree is clearly needed is when we're
doing plain L2-to-L2 forwarding.

I think the suggestion I made could be narrowed down: the only thing
that really needs to be protected is this special case of forwarding
at L2 without going through at least one TRILL hop to a distinct
RBridge.  That special case currently has no protection against loops
*if* IS-IS fails to hear peers.

Donald Eastlake writes:
> Hellos for two different things.

Yes.  Another way to look at it is that when L3 routing fails, it
fails "safe" in that it doesn't forward non-local data, and thus can't
loop.  When this L2 mechanism fails, it fails "unsafe" because it
enables non-local forwarding.  Failing "unsafe" was one of the
criticisms leveled against STP, so it's unfortunate to see it here.

> Their function related to forwarding

Yes; that's it.

> not consider these frame sizes to be "unusual"...

I'll use any word you like here.  At 18+6+18+1500 == 1542 octets,
they're simply bigger than some (likely many) older devices will
support.  Base-2 bias among us geeks makes 1536 a not uncommon limit.

It's actually worse than I'm suggesting.  The above narrow issue
occurs only when you have a lowish-MTU-limit device on the scene with
old-school Ethernet limits.  Things *really* start to fall apart if
any of your TRILL nodes are attempting to support jumbo frames,
because now you're pushing out to everyone's limit, and failure (even
with brand-new devices) is virtually guaranteed.

In other words, if TRILL thinks the limit is 9000, and you need 9024
to get your encapsulated packets through, then you've got a problem,
because you're all out of head room.

> 1504, and envelope frame = 1982.)

My guess is that your budget allows buying newfangled stuff more than
mine.  ;-}

> bridge forwarding respectively.

Agreed; that's just the trigger for the problem, when combined with
IS-IS's attempt to discover MTU restrictions.

> different Hellos sizes...

Yes.  :-<

> and, if STP turns off a port, it becomes unavailable for TRILL

That's not quite so.  We can arbitrarily define how STP and TRILL work
together, if they must work together at all.  As I said in my original
post:

We can simply say that when STP turns off a port, all of TRILL still
flows through the port, because we specifically exempt it.  Those
implementing TRILL already have to make _substantial_ changes to their
code to get even basic functions to work; this sort of exemption would
have trivial impact.

> they are now and get rid of all Hellos except on the Designated VLAN.

That's certainly a possibility.  It might have other bits of goodness
for other cases, such as the wiring closet case.  Interesting ...

> If you have run into it, it clearly has some reality :-)

;-}  As above, it seems like my reality has problems with other folks'
standards.

> such things as hubs any more"...

:-/

> us to the "DRB only if root bridge" position...

Yes.

> them but big enough to let some data through means your network is

Blocking the relatively tiny BPDUs due to MTU problems doesn't happen
in any real network I've seen.  Blocking the intentionally padded-out-
to-what-I-think-the-MTU-is IS-IS frames happens all the time.

There's another subtle issue here, which is that the change to IS-IS
to make it unencapsulated in TRILL has made MTU a much more complex
issue to handle in any sort of interoperable way.  IS-IS needs to know
the link MTU to communicate with its peers.  Unfortunately, the MTU it
uses is likely *NOT* the same as seen by traffic that's encapsulated
by TRILL.

Thus, I think the TRILL IS-IS implementations need to compute at least
MTU+6 in order to match the MTU seen by TRILL.

Dinesh G Dutt writes:
> occurring.

Yes, it's the problem prevention that I'm concerned about.

Whether the deployment is realistic or not probably depends on what
product you make, exactly who your customers are, and what they're
doing with it.  It's hard to put limits on that in ways that seem
on-topic for this group.

> I'm not in favor of running STP because one of the reasons I 

I understand the philosophical objection to running STP, but I'm more
interested in looking at the technical parts first.  In the fix I
suggested, STP was *not* actually controlling the link usage (TRILL
still runs even if STP disables), so STP failure is of much less
consequence, but instead it's guarding against the bad things that
TRILL can apparently do.

> than this min MTU, TRILL forwarding is disabled (i.e. encap/decap).

Specifying things doesn't make them so.

The issue I was posing was with MTU restrictions along the path
between TRILL peers.  The only way you can discover this with the
protocols we're using (as far as I know) is that you just don't
receive Hello messages from some or all of your peers.  How can you
distinguish between that failure case (MTU too small; must disable)
and the case where no peers are present (normal leaf network; must
enable)?

> hand, or shortly there-after.

Exactly how do you know that this problem exists?

In the failure mode, both RBridges start forwarding L2 frames, and the
network becomes unusable.  For an administrator (who would have access
to that knob), it's quite unclear that there is a restriction at all,
or that it is specifically the problem, or exactly what action should
be taken.  (Do I set it to 1500?  1480?  Some other number?)

We have some very painful history here with the FDDI fiasco.  It's
nice that the civilized world is back to 1500 again (no, that world
doesn't include PPPoE), because it limits the scope of the harm
greatly.  But now we're adding more overhead, and raising the problem
again.

> different VLAN topology options.

MTU restrictions along the path are simply not knowable to the
protocols involved, except when larger packets are dropped.

I think you're supposing that the administrator knows about all of the
magic limits built into the equipment his packets must traverse, even
if some of that equipment is locked in closets away from his prying
eyes, but, frankly, I don't see how that's a realistic position.
People are going to _expect_ that TRILL is just plug-and-play.  If
it's anything less than that, it almost certainly won't be acceptable
to any of the customers I know about.

> And that is what I was getting at.

I think the ease with which this can happen is a distinguishing
factor.  This isn't a case where I've asked for something foolish, and
the system has provided me with my foolish answer.  This is a case
where I set up the system using all of the expected defaults, and it
trashed my network.

>    of finding L2 forwarding problems - is unnecessary and

I don't understand "unnecessary" here.  MTU restrictions are a fact of
life.  That's why IS-IS has this feature.  The only alternative I see
is to whistle past the graveyard and "hope" that there's nothing in
the middle that might have restrictions.

If you said "ineffective," I'd be in agreement.  What IS-IS is doing
is in fact ineffective when used for L2 forwarding.

>    the effective maximum MTU at the Data-Link Layer.

Agreed; including the TRILL overhead.

> serve as a proof of concept that the issue you point out can be

I don't think I understand the answer.  Are you saying that all
RBridges must have MTU manually configured on all ports?  That would
certainly put a big dent in the zero configuration story that TRILL is
supposed to have.

It'd work, though, assuming administrators have perfect knowledge of
all parts of the infrastructure they're using, even the parts that are
under someone else's (e.g., a provider's) control.

---

From: "Anoop Ghanwani" <[email-masked]>
Date: Fri, 13 Feb 2009 10:34:52 -0800

> doing plain L2-to-L2 forwarding.

That makes me a little uncomfortable.  STP makes
assumptins about the size of the network for things
like TCNs to propagate in a reasonable amount of time.  
When run on very large topologies this would affect
convergence times quite significantly.

Anoop

---

From: James Carlson <[email-masked]>
Date: Fri, 13 Feb 2009 17:10:02 -0500

> convergence times quite significantly.

I'd wonder whether those large topologies are workable in general
(won't you just drown in NetBIOS noise?), but, yes, that sounds like a
reasonable concern with simply running them in parallel.

---

From: "Eric Gray" <[email-masked]>
Date: Sat, 14 Feb 2009 09:27:30 -0600

James,

	You're conflating the notion of "large topologies" with "large 
networks" where what's important in the former is the number of nodes
and links and the latter the number of noise makers.

	With the proliferation of so-called work group switches, it is
not uncommon to have bridges at the edge of the network numbering ~
half the number of end-stations.  Add in the bridges required to 
connect all of the edges, and you have quite a large bridge topology
- with the number of end-stations possibly barely out-numbering the
number of bridges.

	If you had to run STP on all of these bridges, you could be
looking at exactly the scenario Anoop mentions - and without adding
VLANs to the mix.

	Add VLANs, and the number of noise-makers can get quite large 
without causing the network to "drown in NetBIOS noise" (or other
noise sources) - thus compounding the problems that could exist in
large "simple" topologies using STP.

--
Eric

-----Original Message-----
From: James Carlson [mailto:[email-masked]] 
Sent: Friday, February 13, 2009 5:10 PM
To: Anoop Ghanwani
Cc: Donald Eastlake; Dinesh G Dutt; Eric Gray; TRILL/RBridge Working
Group

---

From: Donald Eastlake <[email-masked]>
Date: Sun, 15 Feb 2009 00:35:18 -0500

Hi,

See below

> doing plain L2-to-L2 forwarding.

I'm not sure of that. Assume a long chaing of RBridges, RB1, RB2, RB3,
... RBn, each connected to the next. Then assume RB1 is connected to a
restricted MTU hub which in turn is connected to RBn. Don't we have
the same problem if Hellos can't get through the hub? Admittedly, if
the hub was actually an 802.1 bridge and you implemented the options
root bridge collision check, you would be safe...

So I don't see why its restricted to "plain L2-to-L2 forwarding".

There are other disadvantages to a giant spanning tree and mentioned
further below.

> because you're all out of head room.

Your arguments above convince me more than ever that we need two
different frames for the two different functions currently being
performed by Hellos. For IS-IS topology adjacency, you want Hellos
that are more or less as long as the longest frames you will normally
be sending. If all of your campus handles 9K Jumbo frames, then you
want 9K+ Hellos so your topology will reflect adjacencies over which
the jumbo frames can flow.

On the other hand, for native frame loop protection, in theory, you
would want a "Hello" as small as the smallest native frame you would
encounter. That's probably impractically small as you need some
miminal data in these loop-protection-Hellos. But they could still be
pretty small which should be safe enough.

> My guess is that your budget allows buying newfangled stuff more than

? I was just quoting the standard. Most modern equipment seems to
either allow for a reasonably large amount of tag stuff (but not as
much as the new "envelope frame" standard) or goes all the way and
allows 9K jumbo frames.

>> different Hellos sizes...

However, maybe we need something like two Hello sizes. Of course, it
could actually be adjacency Hellos and native frame loop safety
Hellos; or it could be adjacency Hellos and some new IS-IS PDU type
for native loop safety frames; or it could be adjacency Hellos and
BPDUs (although people don't seem to want that).

> code to get even basic functions to work; this sort of exemption would

First of all, there is no positive benefit to having the entire
RBridge act as an STP node. It requires interaction between all the
ports. Easier to just run STP on each RBridge port as if it was a
stub.

Secondly, there are negative effects: Although we can say that the STP
blocked state has no effect on TRILL frames being received or
transmitted at an RBridge port, we can not change how *bridge* ports
work. Making the whole mixed RBridge / bridge campus one spanning tree
will, for most topologies, turn off lots of bridge ports and make then
unavailable for TRILL traffic. For example, assume a string of bridges
B1, B2, B3, ... Bn, and a string of RBridges, RB1, RB2, RB3, ... RBn.
Assume each bridge is connected to the next higher numbered bridge and
each RBridge is connected to the next higher numbered RBridge and each
bridge and RBridge having the same number are connected to each other.
With spanning tree running just in the bridges or just in the bridges
and the ports of the RBridges connected to them, all the
bridge-RBridge connections are available for TRILL traffic. With
spanning tree running in and through all bridges and RBridges, you
could get a spanning tree such that all the bridge-RBridge connections
but one was blocked at the bridge end and unavailable for TRILL
traffic...

>> > never appears in this document ... ?)

I now agree with you and think there should be text about MTU and this
problem in the specification.

> MTU+6 in order to match the MTU seen by TRILL.

Seems reasonably to me. (I wanted the IS-IS PDUs to remain
encapsulated, which also solves this, but the WG was of the other
opinion.)

> and the case where no peers are present (normal leaf network; must

Well, you want to disable topological adjacency if the MTU is too
small. But, unless the ports are configures as trunk ports, you always
want exactly one appointed forwarder per VLAN on the link...

As I say above, there are two different Hello usages. For IS-IS
adjacency purposes, padding the Hellos so they are the same size as a
maximum TRILL data frame for your network is probably the right things
to do. But you can't do that for the native frame loop avoidance
purpose where you want a smaller Hello, if anything, not a bigger one.

> be taken.  (Do I set it to 1500?  1480?  Some other number?)

With reasonably small native frame loop safety "Hellos", your network
shouldn't melt down. If big frames are being consistently dropped,
then you can suspect that your adjacency Hellos aren't big enough and
are establishing adjacencies that these large frames can't make it
through.

>> different VLAN topology options.

I don't see that configuring per port MTUs is such good idea... Are
you going to have different topologies for every different MTU value?
Are RBridge supposed to be able to fragment frames?

> to any of the customers I know about.

We need to consider both the zero configuration case and the heavily
configured case...

> is in fact ineffective when used for L2 forwarding.

? Hellos that detect adjacencies over which data can be send should to
be padded to the maximum size of the data frames or their detection of
adjacencies will be erroneous. This is a principle in the IS-IS
specification.

But frames sent to protect against native frame looping have to be
short so they will get through...

> certainly put a big dent in the zero configuration story that TRILL is

Sees like we need to allow some MTU configuration but specify the zero
configuration default for 802.3 links...

Thanks,
Donald

---

From: Radia Perlman <[email-masked]>
Date: Mon, 16 Feb 2009 21:17:09 -0800

Yup. Nasty problem.

So...it seems to be that there are two purposes to a Hello:
a) ensuring that you can hear your neighbors
b) testing to see how big a frame you can really send.

So these are kind of competing purposes.

I'd suggest having RBridges send two types of Hellos:
a) a minimal-sized Hello that includes the RBridge's ID and priority, 
and the designated VLAN -- perhaps
some other stuff like what it thinks the MTU size actually is
b) fully padded Hello with all the other information included.

The second type of Hello only needs to be sent on the designated VLAN, 
and perhaps could
be sent less frequently than the minimal sized Hello.

If R2 hears little Hellos from R1, and then doesn't hear any of the big 
ones, R2 could complain
somehow.

Radia

---

From: James Carlson <[email-masked]>
Date: Tue, 17 Feb 2009 09:03:00 -0500

> b) fully padded Hello with all the other information included.

Yes, that's essentially what I had in mind for my option (A).

---

From: James Carlson <[email-masked]>
Date: Tue, 17 Feb 2009 09:37:52 -0500

> root bridge collision check, you would be safe...

Yes, that's what I was hoping for in that case.  Since I had been
considering running STP in parallel, that would have guaranteed that
there'd be an be a proper root bridge ID available (even if that hub
itself wasn't a bridge at all, but a repeater), because *we* could
always be the root.

> There are other disadvantages to a giant spanning tree and mentioned

Agreed.

> pretty small which should be safe enough.

Yes, that's what Radia and I have both suggested.  I'm not wild about
it, but there's a case to be made.

> much as the new "envelope frame" standard) or goes all the way and

One of the unresolved issues is whether those who build such devices
consider TRILL overhead to be "tag stuff" or just payload.  Being
overly strict about what you consider to be "payload" seems to be par
for this course.

> ports. Easier to just run STP on each RBridge port as if it was a

Agreed.  The more I think about this option, the less I like it.
Consider me convinced.  ;-}

> encapsulated, which also solves this, but the WG was of the other

Rewacking our implementation has at least turned up some interesting
issues.

> want exactly one appointed forwarder per VLAN on the link...

There are secondary issues here as well.  If you detect an MTU
restriction, it means that the network is misconfigured (at best) or
that it (more likely) contains a "broken" device.  Thus, not only is
your adjacency in trouble, but the configuration of that interface and
perhaps of the larger network is in question, and you need to report
that there's a problem.  Even without adjacencies, there's the issue
of AF behavior to consider -- no fair having an AF that only handles
"some" of the packets.

(Obviously, there are MTU problems that _can't_ be detected this way
-- if a restriction has no TRILL neighbors behind it, you're sunk.  I
guess LLDP might be an out ... if it were commonly used.)

> adjacencies will be erroneous. This is a principle in the IS-IS

What makes that mechanism *effective* is that with normal IS-IS (for
L3 forwarding), the forwarding for non-local destinations stops if a
problem is encountered.  With the current TRILL scheme, forwarding
continues if a problem is encountered, and thus I was offering up
"ineffective" in contrast to Eric's suggested "unnecessary."

I think it's necessary.  I just don't think it works without the
changes you've suggested.

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 19 Feb 2009 14:43:29 -0500

Below is a slightly more specific suggestion for "two Hellos"...

> be sent less frequently than the minimal sized Hello.

More detailed proposal on two Hellos (this would be changes in Section
4.2.3.1 and possibly other sections of the protocol specifciation):

The current Hello contents discussion in 4.2.3.1.2 needs to be
augmented to discuss and distinguish adjacency Hellos and native frame
loop safety Hellos and discuss MTU considerations.

Adjacency Hellos: Only send on the Designated VLAN. Have all the
additional data elements listed in the current section 4.2.3.1.2
including the IS Neighbor TLV. Have the usual IS-IS padding to the
expected MTU tweaked as needed.

Native frame loop safety Hellos: Sent as in the current Draft on the
Designated VLAN and possibly many others. Have the same IS PDU type
and fixed header fields as adjacency Hello but only have the
additional data needed for loop safety. In particular, only items 1,
2, and 4 from the current section 4.2.3.1.2 and do not have an IS
Neighbor TLV and do not have padding.

Does the above seem reasonable as a starting point?

Thanks,
Donald

---

From: James Carlson <[email-masked]>
Date: Thu, 19 Feb 2009 14:55:20 -0500

> Does the above seem reasonable as a starting point?

You won't be able to avoid including (5) from the current 4.2.3.1.2,
because it's in the same sub-TLV with the rest, but, yes, that sounds
reasonable to me.

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 19 Feb 2009 15:07:55 -0500

OK. Item 5 is just a one bit flag so it doesn't seem like a big deal either way.

Thanks,
Donald

---

From: James Carlson <[email-masked]>
Date: Thu, 19 Feb 2009 15:13:06 -0500

> OK. Item 5 is just a one bit flag so it doesn't seem like a big deal either way.

Yep.  Just showing off that I've actually read this stuff.  ;-}