Subject: [rbridge] LIDs

From: kreeger at cisco.com (Larry Kreeger (kreeger))
Date: Thu, 2 Nov 2006 22:21:52 -0800

Hi,

I was catching up on old RBridge threads and saw this one about LIDs.

LIDs can be useful in reducing costs in large switches where the entire
(potentially large) local MAC table would not need to be present in
every module of the system.  In fact it could probably eliminate the
need to keep locally learned MACs in hardware at all.  Assuming a switch
with 16 slots and 48 ports per linecard, and servers running 16 virtual
machines each connected to the ports, the number of local MACs would be
16*48*16= 12,288.  With 48 bits per address, that is ~59kbits.

With LIDs, an entry is need only per port, not per local MAC.  So given
a 16 bit LID, only 16 cards * 48 ports * 16 bits = ~12kbits, a savings
of ~47kbits.

The more locally attached MACs, the bigger the savings.

  - Larry
 
Caitlin Bestler wrote on Wednesday, October 18, 2006 1:19 PM:

> Erik Nordmark wrote:
>> Radia Perlman wrote:
>> 
>>> We haven't discussed the LID fields. Perhaps that should be a
>>> different thread. Since that doesn't affect forwarding, and is only
>>> a convenience to the egress RBridge, that doesn't seem too
>>> compelling. The ingress RBridge has to do the lookup per endnode,
>>> so it's only at most twice as much computation for a Designated
>>> RBridge to have to do the lookup both when a packet leaves its link
>>> and when it enters. 
>> 
>> If we are trying to squeeze bits and see a need for a LID capability,
>> it might make sense to allow different egress rbridges have different
>> number of LID bits - they only need to number their ports so in some
>> cases few bits would be sufficient.
>> 
>> Assuming we have a 19 bit space for RBridge alias plus LID, then an
>> rbridge with 16 ports would ask for an alias that is 19-4=15 bits.
>> 
>> The downside of such an approach is that the alias setup would need
>> to be able to handle variable length aliases, and the transit
>> rbridges would either need variable length matching on the RBridge
>> alias, or expand out to all the matching 19 bit patterns and have
>> the hardware do full 19 bit matches. For instance, the above 15 bit
>> alias would expand to 16 different matches on 19 bits.
>> 
>>     Erik
> 
> Bitmask aliasing an RBridge ID has the same potential uses as LID
> aliasing for InfiniBand, and the same complexities. 
> 
> My generate comment on the LID is that it is only worth considering
> if it can be multiplexed with the RBridge ID. Otherwise you are
> replacing an egress RBridge table [local mac-->local port] by having
> having more data in the ingress RBridge table [remote mac-->remote
> rbridge + LID]. I'm not certain that it really reduces the amount of
> memory, and given the likely size of the [local mac-->local port]
> table I don't think it improves latency.      
> 
> 
> 
> 
> _______________________________________________
> rbridge mailing list
> rbridge at postel.org
> http://mailman.postel.org/mailman/listinfo/rbridge

---

From: caitlinb at broadcom.com (Caitlin Bestler)
Date: Fri, 3 Nov 2006 09:07:25 -0800

Larry Kreeger (kreeger) wrote:
> LIDs can be useful in reducing costs in large switches where
> the entire (potentially large) local MAC table would not need
> to be present in every module of the system.  In fact it
> could probably eliminate the need to keep locally learned
> MACs in hardware at all.  Assuming a switch with 16 slots and
> 48 ports per linecard, and servers running 16 virtual
> machines each connected to the ports, the number of local
> MACs would be 16*48*16= 12,288.  With 48 bits per address,
> that is ~59kbits.
> 
> With LIDs, an entry is need only per port, not per local MAC.
>  So given a 16 bit LID, only 16 cards * 48 ports * 16 bits =
> ~12kbits, a savings of ~47kbits.
> 

Instead of keeping the MAC to Port mapping in the egress RBridge
you instead keep it in each ingress rbridge that has learned
of this rbridge. I'm not seeing a net savings in memory costs
anywhere. There is certainly a potential latency benefit, but
is there really one that is valid over the desired lifespan
of the protocol?

---

From: touch at ISI.EDU (Joe Touch)
Date: Mon, 06 Nov 2006 15:22:51 -0800

Caitlin Bestler wrote:
...
> is there really one that is valid over the desired lifespan

Catching up, and based on discussions today:

LIDs don't seem to reduce memory; they increase it. The egress needs the
MAC/LID table to tell the ingress anyway.

Pro:	LIDs allow one existing ingress lookup to determine egress name
	and egress port, saving the port lookup step at the egress.

Con:	Larger shim header.

	Need to coordinate LIDs to the ingress whenever egress port
	changes (even if egress name doesn't), i.e., when a node moves
	ports on an egress, the rbridge is impacted.

	Packets lost/misdirected between when LID changes and when
	ingress is updated.

	Larger table -- O(R*P), vs O(R), where R=# routers, P=# ports.

Given the fact that the lookup required is basically what bridges
already do, I'm not sure why this is a useful reduction of effort at the
egress. Further, a large egress should already have the resources to
handle a large number of ports; shifting the work to the ingress could
potentially redistribute the load in ways that make small ingresses
responsible for very large egress tables.

This seems like complexity and scalability risk that should be avoided.

Joe

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 250 bytes
Desc: OpenPGP digital signature
Url : http://mailman.postel.org/pipermail/rbridge/attachments/20061106/2c59f391/signature.bin

---

From: akatlas at gmail.com (Alia Atlas)
Date: Mon, 6 Nov 2006 15:36:47 -0800

I agree that LIDs are not a good idea.  For instance, if the ingress
has a separate next-hop table, then the size of the next-hop table
must grow to be O(rbridge interfaces) instead of O(rbridges) to store
the different encapsulation information.

This is avoiding a lookup on the egress rbridge to distribute more
change through the network and increase substantially the table
requirements on all ingress rbridges.

Alia