Subject: [rbridge] load balancing rbridges

From: mike at linx.net (Mike Hughes)
Date: Thu, 11 Aug 2005 10:28:55 +0100

Hi folks,

Still catching up on what's been going on with rbridge, and I missed Paris
because of Summer hols. As someone who runs a network for where rbridges
would be an ideal solution to a raft of architectural nags, I'm starting to
follow things with interest.

Thinking about load balancing, I think we've already asserted the need to
prevent evils like packet re-ordering for a traffic flow across the
topology.

When running my plain old ethernet campus network (around Docklands in
London, for those who don't know), I came across issues (5 years ago)
trying to do even the most basic loadsharing across a point-to-point GE
link agg. Circuits in a link-agg group would become over-subscribed, while
others remained heavily under utilised. 

This was mostly because vendors were not peeking deeply enough into the
frame/packet to create a worthwhile hash. They were often just doing L2
dest, ingress port (aagh!) or L2 src/dest hashing - and variants of this
such as partial L2 src/dest or ingress port/L2 dest.

In my network, it's a relatively small number of end stations sourcing a
large amount of traffic - around 200 stations sourcing over 30G of traffic
today - there was a need to look deeper in the packet to make a workable
hash, because of the restricted "gene pool" at L2.

After enough beating up from myself (and some other IX operators, and
probably others elsewhere that I never met), all the larger 10GE switch
vendors are now doing better hashing, usually based on the L3/4 header
information as well. This often works for them, in their architectures,
because it's still only one CAM cycle for them - because of the (unused, in
our case) L3 functionality built into the boxes, they are reading the L3
info, but usually doing nothing with it if the box is working L2-only.

In short, I'm agreeing with those who say that you need to look inside the
payload header to generate the hash for ECMP sharing. This should give you
a reasonable balance of traffic, and keep flows running along the same path
through the network. That hash should be done by the ingress rbridge, and
not touched by the intermediate nodes in the topology, unless there is a
topology change while the frame is transiting the network.

I think you've got to "peek" down into the payload, because of the
likelihood of a limited gene pool at the L2 MAC and rbridge MAC/shim layers.

Thanks,
Mike
-- 
Mike Hughes     Chief Technical Officer  London Internet Exchange
mike at linx.net   http://www.linx.net/
     "Only one thing in life is certain: init is Process #1"