Subject: [rbridge] Auto-discard notification

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 11:04:54 -0700

Alia Atlas <akatlas at gmail.com> wrote:

> 
> Joe,
> 
> What about peeking into the L3 header in order to do load-balancing? 
> This is common for doing load-balancing for ECMP.  As I understand it,
> an rbridge can have several equal-cost paths to the egress rbridge
> indicated by the frame.  Presumably, the same concerns about
> re-ordering micro-flows apply here as for IP.
> 
> Alia

Agreed; we could use alternate addresses for a single egress, where each
used a different path, to achieve this. I.e., the 'peeking' would be at
the ingress only, at least that's my impression.

We want to avoid peeking elsewhere if possible, especially because it
won't occur on conventional bridges inside the campus due to the
inclusion of the shim header. The rbridges inside are intended to have
enough information from just the shim header to operate, but may also
peek ahead, though.

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 11:20:20 -0700

> enough information from just the shim header to operate, but may also

Would a convential bridge have multiple paths to select between?  
I would be concerned about using alternate addresses; this implies
knowledge on the number of different equal-cost paths available at
each point in the campus.

Possibly the ingress rbridge could compute a hash value that is then
forwarded as part of the shim header & is then used as input along the
way.  For instance, with MPLS, if the packet underneath the MPLS shim
header is not IP, then some equipment can do a hash on the label stack
for load-balancing (draft-ietf-mpls-ecmp-bcp-01.txt).

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 13:46:05 -0700

> Would a convential bridge have multiple paths to select between?  

I would expect not, since it should be using the sole spanning tree.

> each point in the campus.

We can - and probably need to - assign different addresses for each
interface on an egress. We could certainly assign different ones if
desired, i.e., as aliases. When multiple equal-cost paths exist, the
entries can be split out in the forwarding table. The ingress already
needs to know which egress rbridge to pick; if there are aliases, it can
select them on a per-flow basis.

This doesn't depend on equal cost paths at every decision point in the
campus, at least AFAICT.

> Alia-

The hard part is what the hash would depend on - IP address, transport
port, etc.?  We might consider that, though, if it's supported for MPLS.
Or we can have the rbridges peek at the headers beyond the shim; as you
note, it won't matter for conventional bridges inside the rbridge (which
wouldn't look even inside the shim) anyway.

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 14:09:18 -0700

> I would expect not, since it should be using the sole spanning tree.

So then the question is confined to how does an rbridge do
load-balancing.  It seems like there are two possibilities that we're
discussing.  Either the mid-point rbridge needs to be able to peek
past the shim header, the encapsulated MAC addresses, and find the IP
header, determine if it is an IP header, and then compute some hash on
that, or the ingress rbridge needs to do the same and somehow indicate
its hash downstream.
 
> select them on a per-flow basis.

Why do we need to assign different addresses for each interface on an
egress??  Wouldn't the egress rbridge receive the frame, remove the
encapsulation, and then decide which local interface to forward the
frame on based upon the remaining frame?

I'm confused by your suggestions for the per-path aliases; doesn't
that require the the midpoint rbridge know what path is intended for a
particular alias?   The midpoint rbridge will still receive a frame
and need to decide which of the multiple equal-cost paths it needs to
be sent upon.  Could you clarify with an example?

> wouldn't look even inside the shim) anyway.

Well, what the hash depends on doesn't necessarily have to be
specified.  I believe there's a BCP about this for IP forwarding; I
believe that recommends using the IP src/dest and UDP/TCP ports (if
available).  This is generally supported (at least to the IP src/dest
level) for IP and for MPLS.

I think that isolating the complexity of doing the hash to the ingress
rbridge might be preferable.  It depends on the forwarding path & how
much knowing the ethertype=rbridge & therefore being able to avoid
this helps.

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 14:20:29 -0700

> that, or the ingress rbridge needs to do the same and somehow indicate

Yes, those are the two alternatives IMO.

> frame on based upon the remaining frame?

I'm speaking of multiple interfaces on the campus-side of the egress, or
aliases to the single interface on the campus-side. The addresses can
then used for path differentiation in the routing algorithm.

> particular alias?

Yes - that would mean that different aliases would have different paths.
That's one version of the solution that could work.

> The midpoint rbridge will still receive a frame

I'm expecting that it could have different forwarding entries for the
different aliases for the egress. The net effect is to use the egress
address as a both the egress and the flow ID.

> much knowing the ethertype=rbridge & therefore being able to avoid

And the complexity of the routing algorithm that can separate the
aliases at rbridge forwarding tables inside the campus as well. It may
be cheaper to just have them peek and keep flows separate themselves.

Joe

---

From: spencer at mcsr-labs.org (Spencer Dawkins)
Date: Wed, 10 Aug 2005 16:48:33 -0500

>much knowing the ethertype=rbridge & therefore being able to avoid

I'm still catching up on RBRIDGE, but one point here is that it's a 
sender's decision which path gets used (based on using one path, 
selecting based on hashes, or phases of the moon), and the receiver has 
to be able to accept packets on any path - did I get this right?

Thanks,

Spencer

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 15:03:47 -0700

> to be able to accept packets on any path - did I get this right?

IMO, yes.

The only refinement I'd make to your summary is your use of 'the sender':

	the sender = source switching node

i.e., not the E2E source

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 15:18:09 -0700

> That's one version of the solution that could work.

I guess I am picturing a relatively standard SPF algorithm.  All
aliases advertised by an rbridge would have the same destination
sink-tree.  As a midpoint rbridge, how would I decide which path to
use for which alias?  Do I randomly sprinkle the aliases among the
paths?
 
> address as a both the egress and the flow ID.

Right.  What I'm missing is the logic in the SPF algorithm that
detemines what the forwarding entries should be.  Also, this increases
the forwarding entry space necessary as a consequence of the
complexity of the campus and the number of aliases...

> be cheaper to just have them peek and keep flows separate themselves.

I agree.  I think I'd prefer something based on the hashing; then the
question is whether one needs independent hashing or whether there's
an advantage to doing it at the ingress.  Doing it at the ingress
could give a bit of control (maybe some flows don't care about
re-ordering) and might limit the required logic to the non-campus
facing ports.

Alia

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 15:20:38 -0700

> to be able to accept packets on any path - did I get this right?

Certainly the receiver has to be able to accept packets on any path. 
As to the sender determining the path, well, for some flexibility in
determining, yes.  The sender at least gets to determine the flow
(either explicitly by a hash or such or implicitly by including the
relevant packet).

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 15:29:56 -0700

Alia Atlas wrote:
> sink-tree. 

But they ought to have different next-hops at some point inside the
campus, or there wouldn't be alternate paths. The different next-hops
show up as different forwarding entries.

> As a midpoint rbridge, how would I decide which path to

If you have only one outgoing nexthop, both alises would forward there
and there would be no multipath. If you have multiple next-hops, you
would assign one to each of the aliases and enter both into your
forwarding table.

> complexity of the campus and the number of aliases...

Yes, but the complexity is there anyway. Either way there needs to be
one entry per egress/path pair, i.e., for multiple nexthops to an
egress, there need to be multiple entries anyway.

> re-ordering) and might limit the required logic to the non-campus

It also makes some of the interior rbridge logic simpler; note that the
ingress/egress complexity is different than the interior rbridge
complexity already.

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 15:43:34 -0700

> show up as different forwarding entries.

The sink-tree looks the same for all aliases.  It's not really quite a
tree, because an rbridge can be connected by multiple links (those
being the ecmp points).
 
> would assign one to each of the aliases and enter both into your

This seems to me to require storing some state about the assignment of
next-hops to aliases for when new aliases are learned and when the
topology changes.  I agree that it is possible, but it seems somewhat
messy to me.

> egress, there need to be multiple entries anyway.

The number of aliases isn't tuned to the different number of next-hops
at a particular midpoint rbridge.  Also, there's a difference between
increasing the addresses known and the size of the resulting
forwarding result.  One grows your look-up structure while the other
is merely more memory.  Also, the former requires the resources from
the whole campus.  In the latter case, each rbridge can make decisions
about what degree and number of ecmp next-hops to support.
 
> ingress/egress complexity is different than the interior rbridge

Ok.  Did we just converge?  :-)  

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 15:50:13 -0700

Alia Atlas wrote:
...
> Ok.  Did we just converge?  :-)  

Subject to "this is an optimization", I think so. But since it's an
optimization, I haven't given it very deep thought.

I was hoping the interior of the system would do 'multipath routing to
an egress' the same way that IP networks can/do. Which is why peeking
inside the IP header at interior rbridges might be more appropriate,
come to think of it. But I don't have enough experience with that - it
might be useful for others with multipath routing experience to chime in.

IMO, the goal is to reuse existing solutions as much as possible here...

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 16:01:54 -0700

On 8/10/05, Joe Touch <touch at isi.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> Alia Atlas wrote:
> ...

Sure :-) 
 
> might be useful for others with multipath routing experience to chime in.

I've got a fair amount of router forwarding path experience; I don't
have as much on the ethernet bridging side.  More comments are always
useful :-)

> IMO, the goal is to reuse existing solutions as much as possible here...

Yes, I agree.  The problem is that no hardware is used to digging that
far into the packet, as far as I know.

In the back of my mind is the following.  First, assume that the shim
header has the structure of an MPLS shim header.  Second, assume that
the egress rbridge is represented by an MPLS label.  (For now, let's
not worry about the label distribution pieces yet or the
multicast/broadcast where the ingress rbridge is needed.)

Third, what is underneath the shim header is not an IP packet & will
not be identified as one.  So, for hardware that supports, say, MPLS
pseudo-wires, the frame would be forwarded based on a hash of the MPLS
label stack.  This implies that the ingress rbridge could put a hash
as the inner label (under the egress rbridge label) & the good
load-balancing could happen.

Of course, this assumes that the egress rbridge can pop off the 2
labels and that the inner label isn't actually ever used for
forwarding, but that seems somewhat reasonable to me.

Thoughts?

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Wed, 10 Aug 2005 16:10:51 -0700

Alia Atlas wrote:
> not be identified as one.

But anything we can't parse we can't assign to a specific flow, unless
we send all such things to a single default path. Otherwise those
packets will be reordered.

> So, for hardware that supports, say, MPLS

The only flows you will have a hash for are those that the ingress can
parse, i.e., those with an inner IP/transport header. If the ingress can
parse them sufficiently to create a hash, then so can the interior
rbridge nodes. The only difference is that the interior nodes need to go
past the shim header, but that's just an offset.

> forwarding, but that seems somewhat reasonable to me.

I'm not sure what it buys to do the hash only at the ingress vs. at the
interior nodes. Both have the same amount of info. It doesn't seem to
hurt anything; it trades edge complexity for interior (which is good)
but costs bandwidth by making the shim larger (which is bad). Seems like
a toss-up to me...

Joe

---

From: akatlas at gmail.com (Alia Atlas)
Date: Wed, 10 Aug 2005 16:39:11 -0700

> packets will be reordered.

I am assuming that the ingress is capable of doing more parsing.  For
instance, the ingress gets the frame before encapsulation & can use
current peeking technology to look inside it.

> past the shim header, but that's just an offset.

First, the question is whether one is trading off the interior
complexity for the edge.  It's certainly possible for the interior to
have to dig further down, and maybe that's necessary for the
broadcast/multicast, but if it is avoidable, that might be good.

Getting past the shim header may or may not be completely trivial.  It
depends on the forwarding path architecture.  For instance, once could
store most of the frame & only pass the beginning to the hardware that
makes a forwarding decision.

> but costs bandwidth by making the shim larger (which is bad). Seems like

For bw, we're talking an extra 32 bits.  I think of it as more a
question of complexity in implementation for the interior nodes & how
much functionality can be preserved.

Alia

---

From: Radia.Perlman at Sun.COM (Radia Perlman)
Date: Wed, 10 Aug 2005 22:43:17 -0700

I'm travelling, so don't have too much time for
email, and notice this very long thread.
I suspect more people than just me would appreciate if it one of you
would summarize the conclusions of this thread so far.

In skimming...some comments...
traditional bridges and traditional IP routers forward based solely
on destination address. However, routers can keep multiple paths to
the destination, and choose between them based on some other criteria.

One story I sometimes tell is that, fresh out of school, I could
throw around queing theory and prove that you'd get better network
utilization if you did path-splitting (presumably, round-robin).
Then Dave Clark pointed out that this makes life hard for layer 4...
packets get arbitrarily out of order, it's hard to measure round
trip delay, it's hard to figure out MTU size, etc. So then I
switched to advocating, if you keep multiple paths, to deterministically
choose the path so that the same flow will always go the same way.
I think that's how routers tend to be implemented today...each
router can independently do some sort of hash based on source address,
perhaps ports, perhaps TOS information, and use the hash to make
a selection from the equal (or almost-equal) cost paths.

Bridges really can't load-split, unless you
do something (like what has been suggested and possibly deployed)
of having different spanning trees for different types of packets, where
all the bridges know somehow which type of packet, and therefore, which
spanning tree, the packet belongs on.

MPLS though allows multiple paths to be set up between A and B (where
"A" is presumably a router, or something that is MPLS-aware), and
allows A to decide which path to send it on. A has complete control. Once
it selects the MPLS path, the packet's journey is completely determined.

So, I'm not sure whether I'm being helpful here because I'm having trouble
understanding this thread (as I said, I need Joe or Alia to summarize),
but what I'm saying is that there are two ways of doing path splitting;
one where someone close to the source determines the remaining path
(which requires something like MPLS), and the other where each
router along the way independently chooses among different paths.
You *might* be able to have an upstream router add advice to the header
for downstream routers to choose which path to use, but if you're
going to do that, it seems better to just use MPLS.

I might be completely off-base on this thread, in which case ignore my
comment, but please, if one of you would be willing to summarize the
thread, I and others will be quite grateful.

Oh, and if you do, can you make the subject line "Summary 1 of
load-balancing bridges"
(then if there is another flurry of messages, someone can add a "summary
2" note, with the theory that if a thread has a "summary" note, you can
catch up by starting at the most recent "summary" note.

Thanks,

Radia

---

From: riw at cisco.com (Russ White)
Date: Thu, 11 Aug 2005 11:22:36 -0400

> that, or the ingress rbridge needs to do the same and somehow indicate

Okay, I hate to do this, but I think we need to back up a bit, and make 
certain we understand the problems and complexities here. :-) Suppose we 
have this:

A---B---C---D---(destination)
     +---E---+

Here, we have two different paths from the entry rbridge to the exit 
rbridge. It's easy enough for B to compute two different paths to D, and 
load share between those two paths--in fact, it doesn't much matter how 
B goes about doing this, in terms of flow, etc, unless we want to worry 
about out of order packets (which we might). This is a completely 
different problem from this:

A---B---C---+
         |   +--(destination
         D---+

Which is that there are two paths to the destination, but B isn't aware 
of this. It seems like the solution proposed is for A to build a hash 
and include it in the shim header, so that B will forward some amount of 
traffic over the link to C, and some amount to D, load sharing the 
traffic. This seems like it's not really needed (?), since A is actually 
going to be able to encap the traffic to C or D directly, so A could 
just change the destination shim destination in the header on every 
other packet (or based on flows) so the traffic takes different paths 
through B to reach the destination.

Does this make sense, or did I lose it (it's been known to happen, I 
know....)? :-)

I'm not certain we need any additional information in the shim header to 
handle this (?). We do need to be careful about out of order packets, 
though, I think. I know most layer 3 switching mechanisms rely on per 
flow information to channel all information in a single flow along the 
same path, while channeling other flows along other paths. That's also 
possible here, if you allow the entry rbridge to peek at the layer 3 
information, or even by just sorting out based on destination layer 2 
address beyond the exit point, as long as you assume some sort of 
"correct" or "well ordered" destination address spread on the 
destination network.

:-)

Russ

- -- 
riw at cisco.com CCIE <>< Grace Alone

---

From: akatlas at gmail.com (Alia Atlas)
Date: Thu, 11 Aug 2005 11:12:50 -0700

As per Radia's request, here's my best effort at a summarization.

The introduction of rbridges implies that rbridges will need to do
load-balancing among equal-cost paths.  To avoid reordering at Layer
4, it is very desirable (even necessary) to have the rbridges able to
identify the micro-flows.  There are 3 possible ways of doing this
that were discussed.

(1) Each rbridge peeks under the rbridge shim header & encapsulated
MAC frame to the (potentially) IP packet underneath.  If there is an
IP packet, then an identification as to the micro-flow is made and the
frame is forwarded based on the micro-flow.  Behavior if there isn't
an IP packet hasn't been discussed.

(2) The ingress rbridge determines the micro-flow and marks the frame
with a micro-flow tag in some way.  Interior (or midpoint) rbridges
can use this tag to identify the micro-flow instead of having to peek
underneath the rbridge shim header.  This is an optimization of (1)
that reduces the complexity of interior rbridges at the cost of
additional header size.

(3)Each egress rbridge could have a number of aliases.  The ingress
rbridge identifies a micro-flow & forwards the frame to a particular
alias, indicating a path.  There are some issues here with number of
paths,  handling addresses connected to multiple egress rbridges, etc.

Option (2) seems promising, but whether it is the correct trade-off
versus (1) needs more discussion and input.  Please comment.  :-)

Alia

---

From: akatlas at gmail.com (Alia Atlas)
Date: Thu, 11 Aug 2005 11:16:14 -0700

> it selects the MPLS path, the packet's journey is completely determined.

MPLS actually has 2 models.  In the first LDP model, MPLS behaves just
like IP; each router does the load-balancing among the equal-cost
paths itself.  In the second TE model, the ingress explicitly creates
paths (LSPs) and then can load-balance among those paths.
 
> going to do that, it seems better to just use MPLS.

The trick with MPLS TE is that the head-end explicitly sets up the
paths ahead of time.  Do we want to go there?  I didn't think so.

Alia

---

From: riw at cisco.com (Russ White)
Date: Thu, 11 Aug 2005 14:47:12 -0400

> identify the micro-flows.  There are 3 possible ways of doing this

There are actually two different issues with load sharing:

A) There are two equal cost paths through the rbridges backbone to any 
given exit point along the edge).

B) A specific destination MAC address is reachable through two exit 
points, both with equal cost, through the rbridge cloud.

A needs to be addressed. B doesn't, as it's an issue the ingress rbridge 
should be hanlding.

> that reduces the complexity of interior rbridges at the cost of

1 is the correct solution, I think. In fact, I would say this is 
"implementation dependant," and the only language the draft actually 
needs is something like:

If an rbridge loads shares over two different paths to reach a single 
exit rbridge, it MUST do so in a way that prevents redordering of 
packets within a flow. This MAY be done by examining a larger part of 
the header of the packet within the shim header to correctly segregate 
multiple flows.

I think that would about cover it, really. It doesn't matter if every 
rbridge in the network does the separation in a different way, as long 
as each one uses the same mechanism consistently.

:-)

Russ

> _______________________________________________
> rbridge mailing list
> rbridge at postel.org
> http://www.postel.org/mailman/listinfo/rbridge

- --

---

From: akatlas at gmail.com (Alia Atlas)
Date: Thu, 11 Aug 2005 11:50:59 -0700

Hi Russ,

> different problem from this:

I think that we do care about out of order packets.
 
> through B to reach the destination.

Whether A computes a hash and includes it in the shim header is
orthogonal to the two cases that you've described.

In the latter case, A must be the ingress rbridge.  A will have 2
next-hops; the first will be given an egress rbridge of C and the
second will be given an egress rbridge of D.  A will load-balance
among them.  To B, these will look like frames to separate
destinations.

Incidentally, this second scenario appears to assume that the same MAC
address can be connected to multiple rbridges.  Is this realistic?  I
think I can see it for a bridge that is connected to both & has hosts
behind the bridge.

> "correct" or "well ordered" destination address spread on the

It's worrying about the out of order packets that is the issue.  The
question is whether every rbridge in the path has to peek at the layer
3 information - or whether it can be done once at the ingress & the
results noted.  I believe that there are three advantages with the
latter.

First, it reduces the complexity of the behavior requried by the
interior.  Second, I believe that there is existing hardware that
could possibly properly load-balance based on the shim header (if it
looked like an MPLS shim), without forwarding path modifications. 
(This isn't to claim that no hw changes would be necessary for
multicast/broadcast...)

Third, having the ingress compute the hash opens the possibilities for
intelligent hash generation and interaction with applications.  For
instance, imagine that some flows care about in-order & some don't -
or not as much.  An ingress rbridge could be configured to recognize
the latter and use different hash values, to improve the throughput. 
Even if the flow does care about mis-ordering, the ingress rbridge
could be configured to explicitly ensure that certain flows don't have
hash collisions.

In other words, one could put more intelligence at the edge to enable
smarter behaviors and simplify the interior complexity.

Alia

---

From: zinin at psg.com (Alex Zinin)
Date: Thu, 11 Aug 2005 11:57:06 -0700

Alia,

 When I looked at it a couple of months ago, it seemed possible to hash
 based on the original src+dst MAC addresses. This makes it L3/4-agnostic
 and still gives sufficient traffic dispersion.

-- 
Alex
http://www.psg.com/~zinin

---

From: spencer at mcsr-labs.org (Spencer Dawkins)
Date: Thu, 11 Aug 2005 14:38:19 -0500

An HTML attachment was scrubbed...
URL: http://www.postel.org/pipermail/rbridge/attachments/20050811/3552a689/attachment.html

---

From: riw at cisco.com (Russ White)
Date: Thu, 11 Aug 2005 15:46:43 -0400

> I think that we do care about out of order packets.

Yes, I do, too.

> think I can see it for a bridge that is connected to both & has hosts

I don't know, I just threw the case up to clarify (I hope).

> smarter behaviors and simplify the interior complexity.

Certainly--but if you put the hash at the ingress side, then you've 
essentially limited the ability of a specific implementation to hash 
differently for different load sharing characteristics. For instance, 
suppose that in one environment, you want to load share based on just 
the destination mac address, but in others, you might need to go deeper. 
Do you specify to always go deeper, and always eat the cost at the 
ingress rbridge, even though different devices might or might not need 
these capabilities?

To give a specific example--if you say you should always load share 
based on the destination ip address, then what if all the packets are 
destinted to a single server, from all sorts of different hosts? So, you 
add the source ip address into the hash, making the hash more expensive 
at the ingress edge. But now, add a NAT on the destination end, and you 
have to add in the destination port. Add a NAT on the source end, and 
you have to add in the source port. At what point does this stop?

Or do we just say: "The ingress edge rbridge should compute a 32 bit 
hash across some locally determined set of fields, and send that in the 
shim header?" I suppose it's possible to put the field in, and mark it 
for a specific purpose, but not actually specify how to choose which 
link to use based on it, or what specific hash to use to fill it in (?).

Now, on the other hand, we're concerned about the forwarding asics in 
these bridges, and whether or not they could handle doiong the hashes 
locally. I think that depends on the type of equipment we're thinking 
will actually be playing the part of an rbridge. Most routers today can 
look at least back to the port numbers to do hashes for load sharing, so 
the current asics used in routers can do this sort of thing. Does that 
make it any less of an issue?

:-)

Russ

- --

---

From: akatlas at gmail.com (Alia Atlas)
Date: Thu, 11 Aug 2005 13:20:15 -0700

On 8/11/05, Spencer Dawkins <spencer at mcsr-labs.org> wrote:
>  Sorry, still synchronizing - what we're talking about is introducing
> additional misordering, right?
>  
>  I wouldn't ask, but I have gotten sucked into some link layers that try to
> deliver in-order, which ALSO tortures TCP because of apparent round-trip
> variation while the receiver waits for a retransmission to forward in
> order...

Yes, just concerned about introducing additional misordering!  The
other hadn't even occurred to me.

Alia

---

From: akatlas at gmail.com (Alia Atlas)
Date: Thu, 11 Aug 2005 13:27:25 -0700

> link to use based on it, or what specific hash to use to fill it in (?).

This is what I was picturing.  I.e. the ingress does the intelligent
packet-crawling & puts a hash (of whatever size agreed) in the header.
 Then rbridges could use that hash instead of crawling through the
packet.

> make it any less of an issue?

Certainly current routers can do it - and, if Alex's suggestion that
just considering the original src/dest MAC is enough, they could be
enhanced look at that as well.  But we're not talking routers - but
bridges and they may have different cost points.

If we're willing to require new hardware at every hop for processing
incoming frames and outgoing frames (within the campus as well as
to/from hosts), then it's not an issue.

Alia

---

From: touch at ISI.EDU (Joe Touch)
Date: Thu, 11 Aug 2005 14:36:12 -0700

> the header of the packet within the shim header to correctly segregate 

I actually prefer this solution. It leaves the shim shorter, and I don't
see a real need to simplify this part of the operation of the internal
rbridges.

---

From: touch at ISI.EDU (Joe Touch)
Date: Thu, 11 Aug 2005 14:40:33 -0700

>  and still gives sufficient traffic dispersion.

That won't help allow internal dispersion of flows between two hosts,
which may be useful to allow higher cross-section bandwidth.

I.e., consider a 100mbps flow that goes

                   a---b
                  /     \
   src--ingres---c       d---egress---dst
                  \     /
                   e---f

If the links a---b and e---f are needed by other cross-campus traffic,
the sum of a---b/e---f may be the only way that the src--dst pair can
achieve 100mbps.

However, all flows between them will be between the same src/dst MAC.

That's when it would be useful to peek into not only the IP, but also
transport headers to mux across the pair of paths without reordering
within a transport connection.

Joe

---

From: touch at ISI.EDU (Joe Touch)
Date: Thu, 11 Aug 2005 14:43:29 -0700

Alia Atlas wrote:
...
...
> think I can see it for a bridge that is connected to both & has hosts

This doesn't make sense to me either. IMO, the spanning tree on the
right side (outside the campus) would dictate that only one of C or D
would be considered the egress for the destination.

Joe

---

From: touch at ISI.EDU (Joe Touch)
Date: Thu, 11 Aug 2005 14:44:41 -0700

> other hadn't even occurred to me.

There are many reasons not to multipath TCP inside a link, if possible:

	- reordering
	- inconsistent RTTs
	- incosistent loss/collision rates
	- inconsistent BW

Joe

---

From: sommerfeld at sun.com (Bill Sommerfeld)
Date: Thu, 11 Aug 2005 18:07:45 -0400

> would be considered the egress for the destination.

Maybe I'm misunderstanding the topology diagrams but I'd hope that there
would be some way for a host to be "multihomed" into an rbridge-based
network in a way which provided for load-balancing.
(perhaps the host would need to implement a "stub" rbridge?)

						- Bill

---

From: touch at ISI.EDU (Joe Touch)
Date: Thu, 11 Aug 2005 15:19:54 -0700

> (perhaps the host would need to implement a "stub" rbridge?)

Rbridges provide additional internal bandwidth via multipath routing,
but outside the campus they're limited to existing ethernet bridging
capabilities.

AFAIK, there's no way to multihome a single ethernet MAC on a single L2
subnet to use two different bridges as part of the path - that violates
spanning tree (presuming spanning tree, of course).

Joe

---

From: jtk at northwestern.edu (John Kristoff)
Date: Fri, 12 Aug 2005 09:43:55 -0500

On Thu, 11 Aug 2005 14:40:33 -0700

>From an operational perspective I'd prefer Alex's suggestion to use
MAC addresses as the flow hash for load balancing.  It seems much
simpler and seems to solve most of the problem.  Separate tools or
other protocols can be used to achieve more perfect load balancing
if it is needed.

John

---

From: mike at linx.net (Mike Hughes)
Date: Fri, 12 Aug 2005 16:29:09 +0100

--On 12 August 2005 09:43 -0500 John Kristoff <jtk at northwestern.edu> wrote:

> MAC addresses as the flow hash for load balancing. 

>From my operational perspective, MAC-based load-balancing just *isn't* good
enough, especially if you're dealing with a reasonable volume of
pre-aggregated traffic. (I've already sent my gory details, and I know I'm
not alone - though willing to accept being in a minority.)

> Separate tools or other protocols can be used to achieve more perfect load
> balancing if it is needed.

If rbridge will be doing MAC-based load-balancing across equal cost paths,
and nothing more intelligent than that, then an "on/off" control knob needs
to be provided for in the specification.

Otherwise, don't do it at all, and leave it to other protocols - if a job
is worth doing, let's do it well.

Cheers,
Mike
-- 
Mike Hughes     Chief Technical Officer  London Internet Exchange
mike at linx.net   http://www.linx.net/
     "Only one thing in life is certain: init is Process #1"

---

From: jtk at northwestern.edu (John Kristoff)
Date: Fri, 12 Aug 2005 12:21:32 -0500

On Fri, 12 Aug 2005 16:29:09 +0100
Mike Hughes <mike at linx.net> wrote:

> pre-aggregated traffic. (I've already sent my gory details, and I know
> I'm not alone - though willing to accept being in a minority.)

Fair enough.  It just seems to me the added complexity of peeking into
upper layers may be problematic, but we live in an increasingly complex
network environment these days.  I think I could envision similar
problems that Joe described in the diagram with MAC-based flow load
sharing when having to peek at the upper layers.  For instance, call
his dst a VPN box.

John

---

From: touch at ISI.EDU (Joe Touch)
Date: Fri, 12 Aug 2005 14:56:09 -0700

> other protocols can be used to achieve more perfect load balancing

Load balancing by looking at IP and TCP headers is already common and
solves the problem at hand (avoiding reordering within a transport
protocol).

MAC-based load balancing, besides not helping with the case I cited,
also fails for systems with multiple NICs that stripe a single transport
across multiple interfaces (e.g., channel bonding). Granted, they're
sort of 'asking for it', but it seems OK to use a fairly common
technique if it solves this problem too.

Joe

---

From: touch at ISI.EDU (Joe Touch)
Date: Fri, 12 Aug 2005 14:59:09 -0700

> technique if it solves this problem too.

PS - one final thought. We don't really need to spec any of this per se,
IMO. We can mention that per-MAC or per-transportflow is fine, and that
we leave it up to the rbridge implementation to decide which to support.
Again, so long as it's consistent within a single rbridge device it need
not be coordinated at all.

Joe

---

From: touch at ISI.EDU (Joe Touch)
Date: Fri, 12 Aug 2005 15:42:17 -0700

John Kristoff wrote:
>>>From my operational perspective, MAC-based load-balancing just
>>
>>>*isn't* good
>>

Yeah - at some level, the aggregation is obfuscated by other layers. The
goal is to avoid reordering transport, but if there's no transport
visible there's (by definition) nothing you can do.

Joe

---

From: shane at castlepoint.net (Shane Amante)
Date: Sat, 13 Aug 2005 04:10:59 -0600

Joe,

On Aug 12, 2005, at 3:56 PM, Joe Touch wrote:

[ snip ]

 > technique if it solves this problem too.

I was just curious if you could be more specific as to what you're  
referring to in the above?  I'm aware of MLPPP that is used to stripe  
packets across multiple, underlying component circuits in the  
bundle.  However, I'm not aware of (Ethernet) NIC's and/or protocols  
that can also provide the same capability on media other than TDM  
circuit.  Could you elaborate on vendors/NIC's, (if such a thing  
exists and I'm not misunderstanding you)?

Thanks,

-shane

> <mime-attachment>

---

From: touch at ISI.EDU (Joe Touch)
Date: Sat, 13 Aug 2005 11:14:25 -0700

> exists and I'm not misunderstanding you)?

I'm not aware of vendor-provided solutions that do this, but rather
installed configurations that do. Some are in the grid community.

Joe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://www.postel.org/pipermail/rbridge/attachments/20050813/9a5d82ee/signature.bin