Subject: Need your action to review draft-ietf-armd-problem-statement-00.txt

From: Linda Dunbar <[email-masked]>
Date: Wed, 09 Nov 2011 19:31:33 +0000

For those of you who touch upon Data Center networking or server virtualization, be it Overlay, VPN4DC, or Application/Software driven Controller for Data Center, Address Resolution is something Data Center Gateways  will encounter because hosts/applications in data centers do communicate with peers which are not in the same Overlay, outside the VPN domain, or from general public. ARMD has published its WG problem statement draft. Your comments and suggestions are appreciated (send to [email-masked]<mailto:[email-masked]>).

Thanks, Linda Dunbar

-----------------------------------------------------------------------------------

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Address Resolution for Massive numbers of hosts in the Data center Working Group of the IETF.

        Title           : Problem Statement for ARMD

        Author(s)       : Thomas Narten

        Filename        : draft-ietf-armd-problem-statement-00.txt

        Pages           : 11

        Date            : 2011-10-17

   This document examines issues related to the massive scaling of data

   centers.  Our initial scope is relatively narrow.  Specifically, we

   focus on address resolution (ARP and ND) within the data center.

A URL for this Internet-Draft is:

http://www.ietf.org/internet-drafts/draft-ietf-armd-problem-statement-00.txt

Internet-Drafts are also available by anonymous FTP at:

ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:

ftp://ftp.ietf.org/internet-drafts/draft-ietf-armd-problem-statement-00.txt

---

From: Pedro Marques <[email-masked]>
Date: Wed, 9 Nov 2011 11:52:00 -0800

Linda,
I'd like to point out that the problem does not need to be solved if
only IP connectivity is required. This I-D:
http://tools.ietf.org/html/draft-marques-l3vpn-end-system-02 describes
a solution in which address resolution is not required irrespective of
the location of the traffic destination. Simply put, traffic can be
routed at each hop.

As per previous threads in this list, the concept of subnet which is
of no practical usefulness in this space. Since VMs can be randomly
distributed across the network the subnet does not provide any
information aggregation value. Thus there is no need to do address
resolution.

  Pedro.

On Wed, Nov 9, 2011 at 11:31 AM, Linda Dunbar <[email-masked]> wrote:
> For those of you who touch upon Data Center networking or server
> virtualization, be it Overlay, VPN4DC, or Application/Software driven
> Controller for Data Center, Address Resolution is something Data Center
> Gateways  will encounter because hosts/applications in data centers do
> communicate with peers which are not in the same Overlay, outside the VPN
> domain, or from general public. ARMD has published its WG problem statement
> draft. Your comments and suggestions are appreciated (send to
> [email-masked]).
>
>
>
> Thanks, Linda Dunbar
>
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories. This draft is a work item of the Address Resolution for Massive
> numbers of hosts in the Data center Working Group of the IETF.
>
>
>

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 09 Nov 2011 20:12:08 +0000

Pedro, 

You are absolutely right that there is no ARP/ND scaling issue if IP can be extended to hosts. Even if IP can be extended to Access switches, there is not much ARP/ND issues. 
The draft lists down several common data center network designs, and states that some of the designs don't have ARP/ND scaling issues (but may have other issues for data centers). 
That is why I encourage people to read the draft and provides comments. 

Linda 

> -----Original Message-----
> From: Pedro Marques [mailto:[email-masked]]
> Sent: Wednesday, November 09, 2011 1:52 PM
> To: Linda Dunbar
> Cc: [email-masked]; L3VPN WG mailing list; [email-masked];
> [email-masked]
> Subject: Re: Need your action to review draft-ietf-armd-problem-
> statement-00.txt
> 
> Linda,

---

From: "Susan Hares" <[email-masked]>
Date: Wed, 9 Nov 2011 15:27:03 -0500

Pedro:

Your review of the draft would be especially helpful in the L3 only design. 

Sue 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Wednesday, November 09, 2011 3:12 PM
To: Pedro Marques
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list;
[email-masked]
Subject: Re: [armd] Need your action to review
draft-ietf-armd-problem-statement-00.txt

Pedro, 

You are absolutely right that there is no ARP/ND scaling issue if IP can be
extended to hosts. Even if IP can be extended to Access switches, there is
not much ARP/ND issues. 
The draft lists down several common data center network designs, and states
that some of the designs don't have ARP/ND scaling issues (but may have
other issues for data centers). 
That is why I encourage people to read the draft and provides comments. 

Linda 

> > draft. Your comments and suggestions are appreciated (send to
_______________________________________________
armd mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/armd

---

From: Pedro Marques <[email-masked]>
Date: Wed, 9 Nov 2011 14:20:53 -0800

Linda,
Please see inline.

> You are absolutely right that there is no ARP/ND scaling issue if IP can be extended to hosts. Even if IP can be extended to Access switches, there is not much ARP/ND issues.

The draft is a bit unclear in terms of its assumptions regarding the
connectivity of the VMs. It does state that the VMs typically have
their own IP and mac addresses but doesn't mention that these are
typically also associated with different VLANs.

There are multiple proposals (e.g. VXLAN) in which these VLANs are
data-center wide. Thus if the VM sees a LAN adjacency to its "subnet"
there is an ARP/ND issue to solve. VXLAN for instance assumes that ARP
requests will be broadcasted using IP multicast groups per VLAN.

You may want to characterize the several 'categories' of proposals
that satisfy the requirements of VM mobility and closed user group
isolation.

For example:

L2 approaches: (e.g. VXLAN)
  These tend towards creating a data-center wide broadcast domain. The
broadcast domain can then be used to send ARP/ND requests.

Routed L2/L3 approaches (e.g. EVPN):
  Present an L2 interface to the VM but transform it to a routed
approach in a network element and advertise mac (optionally) + ip host
prefixes from the network elements. An important sub category here is
whether these solutions then assume multicast capability or not in the
underlying network.

Host based approaches.
  These can present either an L2 or L3 "service" and assume or not the
presence of an underlying multicast service on the part of the
underlying "fabric" network.

Map service based approaches:
  Use a query based mechanism rather than a "push" based mechanism
like routing to distributed the overlay network information.

Each of these has its own way to address ARP/ND related issues. If you
your intent is to document the possibilities there is plenty of
options to choose from. :-)

  Pedro.

---

From: Thomas Narten <[email-masked]>
Date: Thu, 10 Nov 2011 10:49:36 -0500

Hi Pedro.

> typically also associated with different VLANs.

Fair enough. I'll look at clarifying this in the next version (though
I'm not sure right off how it changes anything).

> requests will be broadcasted using IP multicast groups per VLAN.

I don't think it is quite right to say that VXLAN uses "data center
wide VLANs".  VXLAN (and overlay approaches in general) use their own
instance identifier space. It is conceptually like a VLAN (in that it
groups together related servers). But it is not the same as a VLAN,
and using that terminology in the context of VXLAN is probably going
to confuse folk.

> broadcast domain can then be used to send ARP/ND requests.

Correct. But there are also optimizations that can be done (like Proxy
ARP) to limit the need to actually broadcast.

> whether these solutions then assume multicast capability or not in the

And how is VXLAN not a routed approach? It presents an L2 interface to
the VM, but everything is routed at L3 from the switch on out.

The main differences I see between EVPN and overlay approachs like
VXLAN are:

1) VXLAN goes all the way to the hypervisor. It's only the
VM->Hypervisor "link" that is L2.

2) VXLAN uses a different control plane (e.g., IEEE 802 approach)
whereas EVPN appears to be BGP based. 

> underlying "fabric" network.

Not sure what you mean here. Is this where the Host (VM) itself is
aware of the service and interacts directly with it? And wouldn't that
require host changes, which IMO is pretty much a non-starter?

> options to choose from. :-)

Indeed!

Thomas

---

From: Pedro Marques <[email-masked]>
Date: Thu, 10 Nov 2011 08:59:37 -0800

Thomas,

> and using that terminology in the context of VXLAN is probably going

The proposal uses a data-center wide broadcast domain. In common
terminology people often refer to a broadcast domain as a VLAN.

> ARP) to limit the need to actually broadcast.

It depends on what action is taken when a VM moves. In what i'm
labeling as an "L2 based" approach a VM move is handled by generating
a gratuitous ARP reply to update all ARP caches and learning tables.
If that is the VM move update mechanism proxy ARP cannot be used.

> the VM, but everything is routed at L3 from the switch on out.

The addresses of the overlay network are learned by the egress
switches and it depends on a broadcast domain. It sees the underlying
"fabric" network as a LAN.
This is in contrast with approach in which the information regarding
the overlay network is advertised via routing protocols (your get your
choice of ISIS variants and BGP).

> whereas EVPN appears to be BGP based.

VXLAN treats the underlying DC fabric network as a LAN. And uses
learning on the data plane.

EVPN only "learns" on the edge facing interface.

This has consequences on VM moves for instance. In the second case the
move can be done as a "route advertisement". It also has consequences
on the operational state. With a learning based approach the actual
state of the network is just valid at that precise moment. Which makes
it very difficult to understand whether the state is correct or not.
That is common with all data-driven approaches.

> require host changes, which IMO is pretty much a non-starter?

The Host OS / hypervisor. Not the Guest VM. It is a common practice
that the network stack of the host OS is a special purpose stack (e.g.
vSwitch style approaches). The network stack is not the only aspect of
the VM hosting software that is modified to support this class of
applications.

  Pedro.

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 14 Nov 2011 12:42:01 -0800

Linda,

> Pedro, 
> 
> You are absolutely right that there is no ARP/ND scaling issue if
> IP can be extended to hosts. Even if IP can be extended to Access
> switches, there is not much ARP/ND issues.
> 
> The draft lists down several common data center network designs,
> and states that some of the designs don't have ARP/ND scaling
> issues (but may have other issues for data centers).
> 

From 4.2 of the draft:

 4.2.  Scenario 2: L3 Terminates at the Aggregation Switch

   In Scenario 2, the L3 network extends only to the aggregation
   switches (or perhaps to routers that connect to the aggregation
   switches).  The aggregation switches (or the routers that connect to
   multiple aggregation switches) could terminate multiple distinct IP
   subnets (e.g., one per VLAN) or one large IP subnet.  In order to let
   hosts belonging to different IP subnets be placed under any access
   switches, it is necessary for access switches to enable multiple
   VLANs and aggregation switches to enable some VLANs (or subnets) over
   many physical ports.  This configuration breaks the confinement of
   the VLAN's broadcast domain and makes it equivalent to all the access
   switches being part of the same L2 broadcast domain (and IP subnet).
   Thus, this configuration allows VMs to be moved to servers connected
   to other access switches, but increases the size of the L2 broadcast
   domain, which can lead to difficulties outlined below.

This is the scenario that is used to justify the need for large L2
domain. 

The draft assumes that in order to move a given VM from one server
to another, where these two servers are connected to different
access switches, *all* the access switches must be configured with
the VM's IP subnet, and thus *all* the access switches must be part
of the VM's L2 broadcast domain. This is even if at any particular
point in time the servers hosting all the VMs of that IP subnet are
connected to only a subset of these switches. But is it truly
necessary, or could the L2 broadcast domain span only the switches
connected to the servers that presently host VMs of the IP subnet ?

Moreover, even if all the access switches need to be part of the
same L2 broadcast domain, the increase in the size of the L2 broadcast
domain in the above scenario is caused by the access switches, not
by the number of VMs per L2 domain. The difficulies mentioned in
the last sentence of the quoted paragraph are due to ARP, but ARP
is generated by VMs, not by the access switches. Given that, how
the increase in the size of the L2 broadcast domain *due solely to
the access switches*, could increase the amount of ARP traffic that
routers have to process ?

Yakov.

---

From: "Susan Hares" <[email-masked]>
Date: Tue, 15 Nov 2011 01:20:11 -0500

Yakov: 

We are not trying to justify any definition we have, but to report what DC
operators have given as the problem in an OPS draft. 

In many ways, the text has been successful by having you comment on this
point so that operators can consider in depth this scenario. 

With all due respect to you as a grand master of word-smithing, let me
restate some of the argument so that I can attempt to inspire discussion. 

The "all" in this discussion indicates "all" within this domain which
crosses the VLANs in the text. The 802.1q VLAN mechanism provides some
scoping of broadcast and unicast for an L2 grouping. The IP Subset provides
a scoping for scoping for IP (L3 grouping). ARP maps the L2 grouping to the
l3 grouping. The growth in the number of nodes (VM or access) in the
combined groupings of mapped nodes are the problem. 

If you want to restrict nodes within this at L2 or L3, you must find a means
to determine the L2 grouping. The mapping of the IP to MAC Address is scoped
by the combination of L2/L3 grouping. 

So.. at some point the VM or the real nodes reach an "aggregation grouping"
at the access switch or the VMs forming an access switch. This grouping
point is stressed by the underlying growth of L2/L3 node combination. On the
ARMD BOF/List, Manish Karir (Merit) published a study in looking at the ARP
problem using VMs groupings.  

Some large scale operators stated the VM aggregation I described above did
not represent a case widely used. Therefore the discussion focused on the
deployed access switches, servers, and VMs as described in the text.  Please
refer to past discussion from Igor G. on this list or his drafts on
solutions/problems. Also, you can refer to NANOG session on this.  

Yakov - let me know of I am unclear or just misunderstood your discussion
point. Linda, Manish, and Igor let me know if I was incorrect in anything I
said.

Hopefully, I've given the discussion a push. If nothing else, it will 

Sue 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Yakov Rekhter
Sent: Monday, November 14, 2011 3:42 PM
To: Linda Dunbar
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list;
[email-masked]; [email-masked]; Pedro Marques
Subject: Re: [armd] Need your action to review
draft-ietf-armd-problem-statement-00.txt

Linda,

> Pedro,
> 
> You are absolutely right that there is no ARP/ND scaling issue if IP 
> can be extended to hosts. Even if IP can be extended to Access 
> The draft lists down several common data center network designs, and 
> states that some of the designs don't have ARP/ND scaling issues (but 
> may have other issues for data centers).
> 

This is the scenario that is used to justify the need for large L2 domain. 

The draft assumes that in order to move a given VM from one server to
another, where these two servers are connected to different access switches,
*all* the access switches must be configured with the VM's IP subnet, and
thus *all* the access switches must be part of the VM's L2 broadcast domain.
This is even if at any particular point in time the servers hosting all the
VMs of that IP subnet are connected to only a subset of these switches. But
is it truly necessary, or could the L2 broadcast domain span only the
switches connected to the servers that presently host VMs of the IP subnet ?

Moreover, even if all the access switches need to be part of the same L2
broadcast domain, the increase in the size of the L2 broadcast domain in the
above scenario is caused by the access switches, not by the number of VMs
per L2 domain. The difficulies mentioned in the last sentence of the quoted
paragraph are due to ARP, but ARP is generated by VMs, not by the access
switches. Given that, how the increase in the size of the L2 broadcast
domain *due solely to the access switches*, could increase the amount of ARP
traffic that routers have to process ?

Yakov.

> > http://tools.ietf.org/html/draft-marques-l3vpn-end-system-02 
> > describes a solution in which address resolution is not required 
> > irrespective of the location of the traffic destination. Simply put, 
> > traffic can be routed at each hop.
> > 
> > On Wed, Nov 9, 2011 at 11:31 AM, Linda Dunbar 
> > <[email-masked]>
> > wrote:
> > > virtualization, be it Overlay, VPN4DC, or Application/Software 
> > > driven Controller for Data Center, Address Resolution is something 
> > > Data
> > Center
> > > Gateways  will encounter because hosts/applications in data 
> > > centers
> > do
> > > communicate with peers which are not in the same Overlay, outside 
> > > the
> > VPN
> > > ------------------------------------------------------------------
> > > ---
> > --------------
> > >
> > >
> > >
> > >
> > >
> > > directories. This draft is a work item of the Address Resolution 
> > > for
> > Massive
> > >    centers.  Our initial scope is relatively narrow.  
> > > Specifically,
> > we
> > >
> > > ftp://ftp.ietf.org/internet-drafts/draft-ietf-armd-problem-stateme
> > > nt-
> > 00.txt
> > >
> > >
>

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 15 Nov 2011 11:19:19 -0800

Linda,

> Data Center Gateways  will encounter because hosts/applications
> in data centers do communicate with peers which are not in the same
> Overlay, outside the VPN domain, or from general public. ARMD has
> published its WG problem statement draft. Your comments and suggestions
> are app reciated (send to [email-masked]<mailto:[email-masked]>).

Should the overhead due to the broadcast unknown be considered as yet
another "pain point" associated with large L2 domains ?

Yakov.
 
> A New Internet-Draft is available from the on-line Internet-Drafts directorie
s. This draft is a work item of the Address Resolution for Massive numbers of h
osts in the Data center Working Group of the IETF.
> --Boundary_(ID_2xYEQabSJUJeXxiogMJ0fQ)
> Content-type: text/html; charset=us-ascii
> Content-transfer-encoding: quoted-printable
> 
> <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
> osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
> xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schemas-m=
> icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:office=
> :access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s=3D"=
> uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-microsof=
> t-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-microsoft-co=
> m:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadshee=
> t" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns=
> :odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:schemas-micro=
> soft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
> xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=3D"http://m=
> icrosoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Repl=3D"http://=
> schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.microsoft.com/share=
> point/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.com/office/excel=
> /2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" xmlns:ois=
> =3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir=3D"http://=
> schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D"http://www.w3=
> .org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint=
> /dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" xmlns:xsd=3D"http=
> ://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.microsoft.com/sha=
> repoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"=
>  xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" xmlns:sps=3D"http://=
> schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"http://www.w3.org/2001=
> /XMLSchema-instance" xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so=
> ap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udc=
> p2p=3D"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf=3D"http:/=
> /schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss=3D"http://sche=
> mas.microsoft.com/office/2006/digsig-setup" xmlns:dssi=3D"http://schemas.mi=
> crosoft.com/office/2006/digsig" xmlns:mdssi=3D"http://schemas.openxmlformat=
> s.org/package/2006/digital-signature" xmlns:mver=3D"http://schemas.openxmlf=
> ormats.org/markup-compatibility/2006" xmlns:m=3D"http://schemas.microsoft.c=
> om/office/2004/12/omml" xmlns:mrels=3D"http://schemas.openxmlformats.org/pa=
> ckage/2006/relationships" xmlns:spwp=3D"http://microsoft.com/sharepoint/web=
> partpages" xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/20=
> 06/types" xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/200=
> 6/messages" xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/Sli=
> deLibrary/" xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortal=
> Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" xmlns:=
> st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40">
> <head>
> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
> >
> <meta name=3D"Generator" content=3D"Microsoft Word 12 (filtered medium)">
> <style><!--
> /* Font Definitions */
> @font-face
> 	{font-family:SimSun;
> 	panose-1:2 1 6 0 3 1 1 1 1 1;}
> @font-face
> 	{font-family:"Cambria Math";
> 	panose-1:2 4 5 3 5 4 6 3 2 4;}
> @font-face
> 	{font-family:Calibri;
> 	panose-1:2 15 5 2 2 2 4 3 2 4;}
> @font-face
> 	{font-family:SimSun;
> 	panose-1:2 1 6 0 3 1 1 1 1 1;}
> /* Style Definitions */
> p.MsoNormal, li.MsoNormal, div.MsoNormal
> 	{margin:0in;
> 	margin-bottom:.0001pt;
> 	font-size:11.0pt;
> 	font-family:"Calibri","sans-serif";}
> a:link, span.MsoHyperlink
> 	{mso-style-priority:99;
> 	color:blue;
> 	text-decoration:underline;}
> a:visited, span.MsoHyperlinkFollowed
> 	{mso-style-priority:99;
> 	color:purple;
> 	text-decoration:underline;}
> pre
> 	{mso-style-priority:99;
> 	mso-style-link:"HTML Preformatted Char";
> 	margin:0in;
> 	margin-bottom:.0001pt;
> 	font-size:10.0pt;
> 	font-family:"Courier New";}
> span.HTMLPreformattedChar
> 	{mso-style-name:"HTML Preformatted Char";
> 	mso-style-priority:99;
> 	mso-style-link:"HTML Preformatted";
> 	font-family:"Courier New";}
> span.EmailStyle19
> 	{mso-style-type:personal;
> 	font-family:"Calibri","sans-serif";
> 	color:windowtext;}
> span.EmailStyle20
> 	{mso-style-type:personal-reply;
> 	font-family:"Calibri","sans-serif";
> 	color:#1F497D;}
> .MsoChpDefault
> 	{mso-style-type:export-only;
> 	font-size:10.0pt;}
> @page WordSection1
> 	{size:8.5in 11.0in;
> 	margin:1.0in 1.25in 1.0in 1.25in;}
> div.WordSection1
> 	{page:WordSection1;}
> --></style><!--[if gte mso 9]><xml>
> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
> </xml><![endif]--><!--[if gte mso 9]><xml>
> <o:shapelayout v:ext=3D"edit">
> <o:idmap v:ext=3D"edit" data=3D"1" />
> </o:shapelayout></xml><![endif]-->
> </head>
> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
> <div class=3D"WordSection1">
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">For those of you who t=
> ouch upon Data Center networking or server virtualization, be it Overlay, V=
> PN4DC, or Application/Software driven Controller for Data Center, Address R=
> esolution is something Data Center Gateways
>  &nbsp;will encounter because hosts/applications in data centers do communi=
> cate with peers which are not in the same Overlay, outside the VPN domain, =
> or from general public. ARMD has published its WG problem statement draft. =
> Your comments and suggestions are appreciated
>  (send to <a href=3D"mailto:[email-masked]">[email-masked]</a>). <o:p></o:p><=
> /span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Thanks, Linda Dunbar<o=
> :p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">----------------------=
> -------------------------------------------------------------<o:p></o:p></s=
> pan></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <pre>A New Internet-Draft is available from the on-line Internet-Drafts dir=
> ectories. This draft is a work item of the Address Resolution for Massive n=
> umbers of hosts in the Data center Working Group of the IETF.<o:p></o:p></p=
> re>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Title&nbsp;&nbsp;&nbsp;&nbs=
> p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Problem Statement for ARMD<o:p></o=
> :p></pre>
> <pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Author(s)&nbsp;&nbsp;&nbsp;=
> &nbsp;&nbsp;&nbsp; : Thomas Narten<o:p></o:p></pre>
> <pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Filename&nbsp;&nbsp;&nbsp;&=
> nbsp;&nbsp;&nbsp;&nbsp; : draft-ietf-armd-problem-statement-00.txt<o:p></o:=
> p></pre>
> <pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Pages&nbsp;&nbsp;&nbsp;&nbs=
> p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 11<o:p></o:p></pre>
> <pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Date&nbsp;&nbsp;&nbsp;&nbsp=
> ;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2011-10-17<o:p></o:p></pre>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre>&nbsp;&nbsp; This document examines issues related to the massive scal=
> ing of data<o:p></o:p></pre>
> <pre>&nbsp;&nbsp; centers.&nbsp; Our initial scope is relatively narrow.&nb=
> sp; Specifically, we<o:p></o:p></pre>
> <pre>&nbsp;&nbsp; focus on address resolution (ARP and ND) within the data =
> center.<o:p></o:p></pre>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre>A URL for this Internet-Draft is:<o:p></o:p></pre>
> <pre><a href=3D"http://www.ietf.org/internet-drafts/draft-ietf-armd-problem=
> -statement-00.txt">http://www.ietf.org/internet-drafts/draft-ietf-armd-prob=
> lem-statement-00.txt</a><o:p></o:p></pre>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre>Internet-Drafts are also available by anonymous FTP at:<o:p></o:p></pr=
> e>
> <pre><a href=3D"ftp://ftp.ietf.org/internet-drafts/">ftp://ftp.ietf.org/int=
> ernet-drafts/</a><o:p></o:p></pre>
> <pre><o:p>&nbsp;</o:p></pre>
> <pre>This Internet-Draft can be retrieved at:<o:p></o:p></pre>
> <pre><a href=3D"ftp://ftp.ietf.org/internet-drafts/draft-ietf-armd-problem-=
> statement-00.txt">ftp://ftp.ietf.org/internet-drafts/draft-ietf-armd-proble=
> m-statement-00.txt</a><o:p></o:p></pre>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> </body>
> </html>
> 
> --Boundary_(ID_2xYEQabSJUJeXxiogMJ0fQ)--
> 
> --Boundary_(ID_pqxZHOFXGA69gJuhLyEG+A)
> Content-id: <[email-masked]>
> Content-type: text/plain; name=ATT00001.txt
> Content-transfer-encoding: 7BIT
> Content-disposition: attachment; filename=ATT00001.txt; size=127;
>  creation-date="Fri, 04 Nov 2011 15:30:31 GMT";
>  modification-date="Fri, 04 Nov 2011 15:30:31 GMT"
> Content-description: ATT00001.txt
> 
> --Boundary_(ID_pqxZHOFXGA69gJuhLyEG+A)--

---

From: Pedro Marques <[email-masked]>
Date: Tue, 15 Nov 2011 09:46:51 -0800

Sue,

I think the following statements capture a common configuration for
small / low-bandwidth (1g) / private DCs:
  - VLANs are used for isolation.
  - These vlans are locally scoped to a pair of aggregation switches.
  - They are statically configured at the access switches.
  - Traffic exchange between vlans is done at the aggregation switches
(often subject to service modules).

The above is not contentious, i presume.

I think that everyone would also agree that:
  - It is highly desirable to be able to have an application (a
process running on a server) to be placed in an arbitrary location of
the DC. (which conflicts with vlan locality).
  - It is expected that the number of application processes that
participate in the same isolation domain will grow significantly (due
to external factors related to application design).
  - It is expected that the bandwidth requirements for severs will
increase by an order of magnitude (along with higher server
utilization).

I hope we all agree so far.

Now from the 2 sets of premises above it does not directly follow that
the answer is to build larger and larger L2 domains. I do understand
that to be a common perception but that is actually not necessary and
imho not useful.

The fundamental misconception is that the application processes (in a
VM or other wise) do need VLANs. They do not. They need connectivity;
both within an inner level of isolation (today's vlan) and to other
services in the data center (which today is routed often subject to
security policies). An example of the later would be a set of database
servers; database servers are often expensive and shared among
different applications.

In order to be able to place any VM anywhere, the VM needs to have
direct connectivity to multiple isolation domains. isolation domains
which are today modeled as L2 and isolation domains that are today
modeled as L3.

The concept of an L2/L3 boundary is not useful if you assume that a VM
can be placed in an arbitrary location. Anywhere you choose to put
that "boundary": top-of-rack, aggregation, etc, you are likely to find
that the boundary gets in the way of the VMs being able to be located
anywhere. At least without causing traffic to cross the core twice
which is a non-starter.

So to actually solve the problem according to requirements one needs
to get rid of the boundary and move to a solution that is all L2 or
all L3. And ARP/ND is really not a relevant problem.

With an "L2" solution tends to come the assumption that there is a
single "isolation" domain... and a single subnet, which doesn't really
cut it. "L3" and specially BGP-based IP/VPNs natively support the
concept of direct traffic exchanges between different isolation
domains, subject to policy.

Even if you do decide to select an L2 approach that limits an
application to a single isolation domain, the L2 mac information will
need to be present at every switch where that L2 VLAN is present. If
you follow an approach where there is core-interface to core-interface
mac-learning (VXLAN for instance) then indeed you do have ARP
broadcasts being used to update forwarding tables. But approaches such
as EVPN can perform learning on the edge interface only eliminating
the need to generate ARP broadcasts.

It is entirely likely that the approaches that attempt to provide for
large broadcast domains already contain a solution for L2 address
discovery. Conversely it is rather unclear that you can address the
ARP/ND scaling issue independently of the design of the L2 solution
itself.

regards,
  Pedro.

---

From: "Susan Hares" <[email-masked]>
Date: Tue, 15 Nov 2011 13:57:52 -0500

Pedro: 

In all politeness, let me acknowledge your message. I will respond after I
see if Igor (Yahoo) or other DC/Carrier operators will comment on their
viewpoints.

Best wishes, 

Sue 

-----Original Message-----
From: Pedro Marques [mailto:[email-masked]] 
Sent: Tuesday, November 15, 2011 12:47 PM
To: Susan Hares
Cc: Yakov Rekhter; Linda Dunbar; [email-masked]; [email-masked]; L3VPN
WG mailing list; [email-masked]
Subject: Re: [armd] Need your action to review
draft-ietf-armd-problem-statement-00.txt

Sue,

regards,
  Pedro.

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 16 Nov 2011 01:30:22 +0000

Hi Yakov,

I believe unknown unicast flood should be considered as another pain point associated with large L2 domains which are extended across multiple data centers. The reason is the WAN bandwidth is much limited and expensive.

Best regards,
Xiaohu 
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Yakov Rekhter [[email-masked]]
发送时间: 2011年11月16日 3:19
到: Linda Dunbar
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list; [email-masked]
主题: Re: Need your action to review      draft-ietf-armd-problem-statement-00.txt

Linda,

Yakov.

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 16 Nov 2011 02:02:45 +0000

<XXH> Agree.

<XXH> Reasonable. However, the ARP cache mechanism is suitble for most MAC-forwarding based DCN solutions (e.g., VPLS) and much helpful for ARP broadcast reduction unless the L2VPN PE function is pushed to the physical server.

Best regards,
Xiaohu

regards,
  Pedro.

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 16 Nov 2011 02:16:02 +0000

Yakov, 

You are absolutely right. We will make sure it is properly captured in the document. 

Thanks, Linda

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 16 Nov 2011 03:22:13 +0000

By the way, I also noticed that the MAC forwarding table scalability issue is mentioned in the ARMD problem statement doc. Should we extend the charter scope of ARMD WG or re-form the DCOPS WG to discuss so many challenges and requirements in data centers?

Best regards,
Xiaohu
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Linda Dunbar [[email-masked]]
发送时间: 2011年11月16日 10:16
到: Yakov Rekhter
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list; [email-masked]
主题: RE: Need your action to review      draft-ietf-armd-problem-statement-00.txt

Yakov,

Thanks, Linda

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 16 Nov 2011 03:27:33 +0000

XiaoHu, 

MAC forwarding table size issues are addressed by IEEE802.1. There are several proposals there targeting at the issues. 

Linda

> From: Xuxiaohu
> Sent: Wednesday, November 16, 2011 11:22 AM
> To: Linda Dunbar; Yakov Rekhter
> Subject: re: Need your action to review draft-ietf-armd-problem-
> statement-00.txt
> 
> 
> By the way, I also noticed that the MAC forwarding table scalability
> issue is mentioned in the ARMD problem statement doc. Should we extend
> the charter scope of ARMD WG or re-form the DCOPS WG to discuss so many
> challenges and requirements in data centers?
> 
> Best regards,
> Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 16 Nov 2011 03:35:37 +0000

So, did you mean the description text for the MAC table scalability issue should be removed from the ARMD problem statement doc?

Best regards,
Xiaohu
________________________________________
发件人: Linda Dunbar
发送时间: 2011年11月16日 11:27
到: Xuxiaohu; Yakov Rekhter
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list; [email-masked]
主题: RE: Need your action to review      draft-ietf-armd-problem-statement-00.txt

XiaoHu,

Linda

---

From: David Allan I <[email-masked]>
Date: Wed, 16 Nov 2011 00:00:07 -0500

IMO the issue is always in the eyes of the beholder. IF we were only able to ever deploy ONE box in scenario 2 there would be an issue. Fortunately we do not have such an irrational limitation. So there are obvious and trivial solutions to addressing MAC scalability on a single platform, deploy more than one box!

So if we want to find things to worry about, perhaps we should cast our nets elsewhere...

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Wednesday, November 16, 2011 11:36 AM
To: Linda Dunbar; Yakov Rekhter
Cc: [email-masked]; [email-masked]; L3VPN WG mailing list; [email-masked]
Subject: re: Need your action to review draft-ietf-armd-problem-statement-00.txt

Best regards,
Xiaohu

XiaoHu,

Linda

> the charter scope of ARMD WG or re-form the DCOPS WG to discuss so 
> many challenges and requirements in data centers?
>
> Best regards,
> Xiaohu
> > Should the overhead due to the broadcast unknown be considered as 
> > yet another "pain point" associated with large L2 domains ?
> >
> > Yakov.
> >

---

From: Warren Kumari <[email-masked]>
Date: Sat, 19 Nov 2011 20:32:19 +0800

> see if Igor (Yahoo) or other DC/Carrier operators will comment on their

Commenting (onna plane back from NRT->SFO so expect some decoherence.)

> The above is not contentious, i presume.

Nope, not contentious (I personally would have worded bullets 2 and 3 as one ("configured on the switching infrastructure") to better cover the folk who collapse the aggregation / access into one device, etc...
> I hope we all agree so far.

Mostly….

I question bullet 2 and the relevance to the argument, but not important...

> the answer is to build larger and larger L2 domains.

Fully, 100% agree.

> I do understand

Not quite sure where the "crossing the core twice" pops up, but sure...

> all L3.

Have to disagree here -- a hybrid approach with small L2 domains also solves the issue. Not really pushing this, just a sayin'...

> And ARP/ND is really not a relevant problem.

In *this* design, with this set of problems and solutions, sure.
If you have (e.g) already built your datacenter with 4 stinking huge L3 (or L2) core switches and O(1000) ToR switches you may (do?) have ARP/ND issues. Waving this away with the "Doctor, doctor" argument, while tempting, doesn't actually solve the issue….

> cut it.

But, again, some folk have built to this design, and are familiar with this design, and like this design, and have shown the ability to actually operate this design at scale. It works for them and declaring their design invalid because it doesn't match your (and, frankly my) views of how to build things also doesn't cut it...

> "L3" and specially BGP-based IP/VPNs natively support the

As do many other overlay / tunneling solutions, and better design of end applications (why exactly to customers purchasing VMs / machines in a datacenter want all of their machines in the same "network"? Largely because A: this is how they view the world and have the $$$ and B: some of their apps want the things they talk to in the same subnet….)

> ARP/ND scaling issue independently of the design of the L2 solution

Hmmm. "unclear" is a squishy word -- I think that the ARP/ND scaling issue *can* be addressed independently of the design of the L2 solution -- for example, I can statically configure all ARP on all hosts in a arbitrary large L2 domain...

I guess in summary I largely agree with your mail, but feel that the ARP scaling (and overlay ideas (and datacenter architecture)) questions are worth addressing.
While simply waving a magic wand and telling folk who have architectured their datacenter as L2 that they are "doing it wrong" is fun (and Lord knows I've done this and had some good fights^W debates with at least one of the other active folk on this list) they may have valid reasons for needing their design…

W