Subject: [rbridge] Break 4096 VLAN limit

From: Emmanuel Hocdet <[email-masked]>
Date: Fri, 10 Sep 2010 12:45:25 +0200

Hi,

On heavy virtualisation, with Rbridge, VLAN limit is very low.
There is a way to break the 4096 limit with Rbridge/Trill ?

my candid approach  (modify 802.1q, but only in Rbridge)
Replace the inner vlan with a inner extended vlan, only used by Rbridge
On endnode, use a dynamic mapping  extended vlan <=> (local) vlan

Is it theoretically possible?

Emmanuel

_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Donald Eastlake <[email-masked]>
Date: Sat, 11 Sep 2010 08:58:13 -0400

Hi Emmanuel,

> There is a way to break the 4096 limit with Rbridge/Trill ?

Yes, there are cases where the number of VLANs would be a problem. It
is not clear how to get around this within the constraints of the
current TRILL standard; however, there are number of different ways
that standard could be extended.

> On endnode, use a dynamic mapping  extended vlan <=> (local) vlan

Perhaps people should go off and write up some alternatives which
could then later be discussed on the list and/or at the Beijing
meeting.

Thanks,
Donald

---

From: Benson Schliesser <[email-masked]>
Date: Sat, 11 Sep 2010 11:22:13 -0500

> that standard could be extended.

I agree that VLAN scale is a serious problem, and I'd like to see it addressed.  The mechanism described by Emmanuel sounds reasonable, especially the edge-translation to unique .1q tags (to avoid host impact).

However I'd prefer to see the network use an approach with larger namespace than Q-in-Q provides.  From my perspective, the L2 segment / VLAN namespace in a datacenter should be similar to a PBB (or whatever) namespace.  This would enable bridging datacenter segments into other datacenters, metro Ethernet, etc, without requiring translation (i.e. configuration) at the datacenter-WAN edge.

Cheers,
-Benson

---

From: Xu Xiaohu <[email-masked]>
Date: Mon, 13 Sep 2010 10:39:28 +0800

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代
表
> Benson Schliesser
> 发送时间: 2010年9月12日 0:22
> 收件人: [email-masked]
> 抄送: Donald Eastlake; Emmanuel Hocdet
> 主题: Re: [rbridge] Break 4096 VLAN limit
> 
> 
> I agree that VLAN scale is a serious problem, and I'd like to see it
addressed.
> The mechanism described by Emmanuel sounds reasonable, especially the
> edge-translation to unique .1q tags (to avoid host impact).

I understand that 4096 VLANs are not enough for large data centers.

> However I'd prefer to see the network use an approach with larger
namespace
> than Q-in-Q provides.  From my perspective, the L2 segment / VLAN
namespace
> in a datacenter should be similar to a PBB (or whatever) namespace.  This
would
> enable bridging datacenter segments into other datacenters, metro
Ethernet,
> etc, without requiring translation (i.e. configuration) at the
datacenter-WAN
> edge.

Is it possible to use other namespace (e.g., route-target used in L2VPN or
L3VPN) than VLAN ID or Q-in-Q value to identify the VPN membership? In this
way, VLAN IDs can be assigned in a more flexible way. That's to say, VLAN
IDs or Q-in-Q values just need to be locally unique on a given edge RB, and
the same VLAN IDs or Q-in-Q values could be used on different edge RBs to
represent different service instances. In addition, it is much convenient to
merge two Rbridge domains which had different VLAN plannings. In contrast,
even using VLAN stack, it is possible to encounter VLAN conflict when
merging two Rbrdige domains. 

Best wishes,
Xiaohu

---

From: Joe Touch <[email-masked]>
Date: Sun, 12 Sep 2010 21:08:00 -0700

On 9/11/2010 5:58 AM, Donald Eastlake wrote:
> Hi Emmanuel,
>
> On Fri, Sep 10, 2010 at 6:45 AM, Emmanuel Hocdet<[email-masked]>  wrote:
>> Hi,
>>
> Yes, there are cases where the number of VLANs would be a problem.

See RFC 5556 - this is outside the scope of this WG:

    Solutions to TRILL need not support deployment of larger scales of
    Ethernet link subnets than current broadcast domains can support
    (e.g., around 1,000 end-hosts in a single bridged LAN of 100 bridges,
    or 100,000 end-hosts inside 1,000 VLANs served by 10,000 bridges).

I.e., this can't be a problem if the base network supports the number of 
VLANs. If it is, TRILL isn't intended to fix it.

Joe

---

From: Benson Schliesser <[email-masked]>
Date: Sun, 12 Sep 2010 23:40:15 -0500

Hi, Joe.

> VLANs. If it is, TRILL isn't intended to fix it.

Regardless, scale in datacenter networks needs improving.  The problem could reasonably be divided into multiple aspects, one of which is the size of broadcast segment namespace.  Do you object to the TRILL WG addressing this topic, if re-chartered to do so?  Where would you suggest is a more appropriate forum, and how would that work eventually integrate with a TRILL deployment?

Cheers,
-Benson

---

From: Joe Touch <[email-masked]>
Date: Mon, 13 Sep 2010 09:07:32 -0700

Hi, Benson,

On 9/12/2010 9:40 PM, Benson Schliesser wrote:
> Regardless, scale in datacenter networks needs improving. The
> problem could reasonably be divided into multiple aspects, one of
> which is the size of broadcast segment namespace. Do you object to
> the TRILL WG addressing this topic, if re-chartered to do so? Where
> would you suggest is a more appropriate forum, and how would that
> work eventually integrate with a TRILL deployment?

That would need to start with an understanding of whether rbridges help 
scale, esp. of the broadcast domain. For reasons why they would probably 
not, see the LISP WG.

I don't think we need to different WGs trying to solve this (IMO) 
unsolvable problem. At best, I'd say take this over to LISP.

Joe

---

From: Donald Eastlake <[email-masked]>
Date: Mon, 13 Sep 2010 13:46:14 -0400

> VLANs. If it is, TRILL isn't intended to fix it.

First, there are no network size restriction in any version of the
TRILL Charter that I am aware of.

Second, "need not" in the words above means only that a solution which
does not address larger networks, in some dimension, still meets the
requirements of the RFC 5556 Problem Statement, not that features
which make TRILL more capable and happen to to be particularly
beneficial for larger networks are somehow prohibited.

Third, RFC 5556 was mostly put together in 2008. I think the view then
was to try to get to a decent base TRILL standard and people didn't
want to try to take on too much. Now that has been accomplished, in
the TRILL base protocol standard. It does not seem unreasonable to
think now about extensions. In fact, some extensions already exist in
draft form.

What seems important to me for some work to be taken on is (1) that it
be allowed by the Charter, (2) the WG wants to take it on, and (3)
there are people willing to do the work. There seems to be
considerable interest in the topic in question. One reason I suggested
people write drafts is to see if there were people willing to do the
work

I have no idea where the idea has come from that if some improvement
to the TRILL base protocol standard happens to be particularly helpful
for large networks, the TRILL WG is somehow prohibited from working on
it.

Thanks,
Donald

PS: I am currently in England and have somewhat spotted Internet connectivity.

---

From: Joe Touch <[email-masked]>
Date: Mon, 13 Sep 2010 12:33:51 -0700

On 9/13/2010 10:46 AM, Donald Eastlake wrote:
> On Mon, Sep 13, 2010 at 12:08 AM, Joe Touch<[email-masked]>  wrote:
>>

Not in the charter, but we agreed on limiting the scope to avoid issues 
of scale during the early work of the WG, as documented in RFC5556. This 
is at least partly because there's nothing about the TRILL architecture 
that supports increased number of devices per se; it's intended to 
support increased node-node capacity and better routing protocols, but 
not scale.

> beneficial for larger networks are somehow prohibited.

"happen to be" is different from "focused on". This has come up many 
times in the WG, and we've basically said that such enhancements would 
be considered only if optional (e.g., as with cached ARP responses).

> people write drafts is to see if there were people willing to do the

It's also important that the work in question be engineering, not 
research. There's simply no evidence that rbridges support increased 
number of edge devices where existing ethernet devices would fail. That 
would suggest a scalable solution to map-encap systems in general, and, 
as the LISP WG is discovering, no such solution exists yet.

Joe

---

From: Donald Eastlake <[email-masked]>
Date: Mon, 13 Sep 2010 19:13:27 -0400

Hi Joe,

On Mon, Sep 13, 2010 at 3:33 PM, Joe Touch <[email-masked]> wrote:
>>> ...
> Not in the charter, but we agreed on limiting the scope to avoid issues of
> scale during the early work of the WG, as documented in RFC5556. This is at
> least partly because there's nothing about the TRILL architecture that
> supports increased number of devices per se; it's intended to support
> increased node-node capacity and better routing protocols, but not scale.

I don't think "to avoid issues of scale" is accurate. Lots of things
have some influence on scale and we would certainly have avoided
anything that caused bad scaling that would have limited TRILL to
small networks. We just didn't want to tackle increased scaling at the
same time we were getting the basic TRILL protocol under control.

Of course TRILL scales somewhat better than bridges do. Just the
reduction in the forwarding tables at the transit RBridges does that.
(And if you are going to complain that SPB has now copied enough from
TRILL that it has that same advantage, I would point out that routing
calculations in TRILL scale as n*(log n) while in SPB they scale as
n*n*(log n).)

> "happen to be" is different from "focused on". This has come up many times
> in the WG, and we've basically said that such enhancements would be
> considered only if optional (e.g., as with cached ARP responses).

"Many times"? The mailing list is on line. I'll believe you as soon as
you provide me with pointers to three different cases where there was
WG decision as you have described. (Note: postings by you do not
constitute a WG decision.)

But as for enhancements being options, I would say, of course. Once
you have specified a protocol and lots of people are implementing it,
you want to be careful about mandatory changes and generally make them
only with consensus of the implementors. Where did you get the
impression that extensions would be some sort of mandatory upgrade?

> It's also important that the work in question be engineering, not research.
> There's simply no evidence that rbridges support increased number of edge
> devices where existing ethernet devices would fail. That would suggest a
> scalable solution to map-encap systems in general, and, as the LISP WG is
> discovering, no such solution exists yet.

Well Joe, perhaps the future will surprise you on some points. In the
mean time, I do not agree with you that thinking about greater scaling
should be classified as thought-crime.

Thanks,
Donald

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 14 Sep 2010 06:20:50 -0500

On 13 Sep 10, at 11:07 AM, Joe Touch wrote:
> That would need to start with an understanding of whether rbridges help scale, esp. of the broadcast domain. For reasons why they would probably not, see the LISP WG.
> 
> I don't think we need to different WGs trying to solve this (IMO) unsolvable problem. At best, I'd say take this over to LISP.

In fairness, the problem of scaling a datacenter network includes a number of issues.  Taken in aggregate, it's a big challenge.  LISP and other interconnect technologies have (in my view) a place in solving the overall scale problem.  Likewise, technologies that address optimized broadcast distribution are relevant.  As are improvements to inter-layer resolution / discovery protocols (i.e. ARMD).

But in this instance, all I'm referring to is the broadcast segment namespace.  If I wish to deploy 10,000 hosts in a datacenter, each with their own VLAN, I'm limited by the 802.1q namespace.  If I wish to interconnect a dozen datacenter facilities transparently, without manually configuring translation policy for each broadcast domain individually, yet also without namespace conflict, I'm limited by the 802.1q namespace.

If TRILL would support a larger namespace, ideally something on the scale of PBB and/or using modern IEEE 802 namespaces, then the way is paved for better scale in combination with the aspects described above.  Otherwise, TRILL (and 802.1d bridging) is relegated to smaller islands that require more complex and burdensome interconnect configuration.  As a service provider, I find the latter situation painful.

Cheers,
-Benson

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 14 Sep 2010 07:51:32 -0500

Hi, Xiaohu.

> merging two Rbrdige domains. 

I gather that you're describing a scenario where the number of "islands" are increased.  So, if there are 100 datacenters each with 10,000 VLANs, one might build 3 islands of 802.1q namespace in each physical datacenter for a total of 300 islands.  Interconnect between the islands would be facilitated by a translation policy mapping the local namespace to a global namespace.  This could work, of course, but the management overhead would be undesirable in some cases.

>From my perspective, the solution should avoid disparate namespaces across a number of islands requiring translation policy and configuration to be interconnected.  We should discuss what number of islands is appropriate, but I think that at minimum the namespace should allow unique IDs across all datacenters managed by a single service provider.  Inter-provider connectivity could reasonably be supported with translation and/or other mechanisms.

In short, I think we need both mechanisms: a larger namespace supporting a broader administrative domain, and a translation mechanism for interconnect at administrative boundaries.

Cheers,
-Benson

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 Sep 2010 07:17:10 -0700

Hi, Donald,

On 9/13/2010 4:13 PM, Donald Eastlake wrote:
> Hi Joe,
>
> On Mon, Sep 13, 2010 at 3:33 PM, Joe Touch<[email-masked]>  wrote:
>>

Routing isn't the problem. Please review LISP's problems developing a 
scalable discovery protocol. The issue is that the reductions for the 
core nodes comes at the expense of the size of the map tables at the 
edge (O(m^2), for m ethernet origins/sinks), and the cost of maintaining 
those tables ('global' in nature, i.e., requires either broadcast ala 
ARP, centralization ala LANE, or arbitarily far multihop ala DHT). LISP 
doesn't have a scalable solution here.

This is not an issue for TRILL precisely *because* the PAS says that 
TRILL focuses on solutions that are not intended to scale beyond that of 
existing ethernets. The point of that statement is to keep discovery 
tenable.

> constitute a WG decision.)

I said:

a) it has come up many times in the WG

b) the WG decided, during the PAS discussion, that the solution would 
not address scale

Neither of these asserts that the WG made multiple decisions. Yes, I'm 
the person who tends to remind the WG of the decision, as I was an 
author of the PAS and I helped point out the rationale for this issue at 
the time.

I appreciate that datacenters need to solve this problem, but their need 
doesn't change the fact that neither TRILL nor LISP has a solution here. 
Working on that isn't a though crime, but it's outside the scope of what 
IETF WGs should work on. I objected to making LISP a WG for this reason 
- it's research, not engineering, and unsolved research at that.

*if* a solution exists, we can consider updating the charter. But just 
saying 'let's work on it' replicates the mistake of LISP, and creates 
another WG looking for the *same* solution that (at least currently) 
simply doesn't exist.

Joe

---

From: Xu Xiaohu <[email-masked]>
Date: Sun, 19 Sep 2010 16:51:16 +0800

Hi Benson,

表
> Benson Schliesser
> 发送时间: 2010年9月14日 20:52
> 收件人: Xu Xiaohu
> 抄送: [email-masked]; 'Donald Eastlake'; 'Emmanuel Hocdet'
> > represent different service instances. In addition, it is much
convenient
> to
> I gather that you're describing a scenario where the number of "islands"
are
> increased.  So, if there are 100 datacenters each with 10,000 VLANs, one
might
> build 3 islands of 802.1q namespace in each physical datacenter for a
total
> of 300 islands.  Interconnect between the islands would be facilitated by
a
> translation policy mapping the local namespace to a global namespace.
This
> could work, of course, but the management overhead would be undesirable in
some
> cases.

Yes, it breaks the 4096 VLAN limit and eases the merging of different
Rbridge domains, at the cost of introducing some management overhead.
However, isn't such management overhead sustainable for a network operator
that runs such a super large cloud data center network? 

> >From my perspective, the solution should avoid disparate namespaces
across
> a number of islands requiring translation policy and configuration to be
> interconnected.  We should discuss what number of islands is appropriate,
but
> I think that at minimum the namespace should allow unique IDs across all
> datacenters managed by a single service provider.  Inter-provider
> connectivity could reasonably be supported with translation and/or other
> mechanisms.
> 
> In short, I think we need both mechanisms: a larger namespace supporting a
> broader administrative domain, and a translation mechanism for
interconnect
> at administrative boundaries.

Agree.

Best wishes,
Xiaohu

---

From: Emmanuel Hocdet <[email-masked]>
Date: Wed, 29 Sep 2010 18:56:34 +0200

Hi,

My exact approach:
no management overhead
no host impact

One big namespace (for one or more datacenter)
On EndNode:
       Attach VNIC on RBridge with a namespace id
       RBridge create a local dynamic mapping  VLAN id<=>namespace id
       VNIC realy create with the (local) VLAN id

Emmanuel