Subject: [armd] another revision for ARMD charter statement

From: Linda Dunbar <[email-masked]>
Date: Sun, 12 Sep 2010 08:53:31 -0500

After separate discussions with Force10's Sridhar and Brocade's Anoop, we
have decided to make some changes to the ARMD charter statements. Here is
the revised one. Please provide your comments. We plan to submit this
charter statement to Internet AD Ralph and Jari next Tuesday.  

ARMD: Address Resolution for Massive amount of hosts in one broadcast Domain

Description of Working Group:

As server virtualization is introduced to data centers, the number of hosts
in a data center can grow dramatically.  Each physical server, which used to
host one end-station, now can host hundreds of end-stations or virtual
machines (VMs). Virtual machines (VMs) can be easily added and/or deleted
and moved between servers.  The VM's flexible addition/deletion and moving
around provide foundation for virtual hosts and virtual desktops offered by
Cloud Computing services and make it possible to combine pool of servers
into one single entity for ease of management and better utilization.

This rapid growth of virtual hosts could tremendously impact networks and
servers. One major issue is frequent ARP (IPv4) or neighbor discovery (IPv6)
requests from hosts. All hosts send out those requests frequently due to
host ARP caches being aged out in minutes. With tens of thousands of hosts
(each with a distinct MAC address) in one Data Center, the number of ARP (or
ND) packets per second from all hosts could potentially be as high as 1,000
to 10,000/second. This rate imposes tremendous computational burden on many
hosts.

Traditional subnet (VLAN) partitioning no longer works well when servers are
virtualized. First of all, servers need to be on the same subnet for VM
mobility. Secondly, when each server only has one host, there is only one
VLAN enabled for the port towards the server. The switch can block all
broadcast messages from other subnets (VLANs). When one physical server is
supporting >100 Virtual Machines, i.e. >100 hosts, most likely the virtual
hosts on one server are on different subnets (VLANs). If there are 50
subnets (VLANs) enabled on the switch port to the server, the server has to
handle all the ARP broadcast messages on all 50 subnets (VLANs). The amount
of ARP to be processed by each server is still too much. 

One characteristic of a next-generation Data Center is the possibility of
one subnet spanning  multiple sites. For example, a server administrator can
combine servers in multiple sites into one huge resource pool for better
resource management and ease of allocation.  Each of the servers in this
pool might be configured with one or more VMs. If mobility is required, like
VMware's vMotion, all the physical servers  (and associated VMs) need to be
on the same IP subnet. Under those circumstances, one Layer 2 network (or
subnet) could span across multiple sites. The multiple sites could be
connected by any types of network, like L2VPN, L3VPN, Ethernet network, or
simple IP networks. Efficient address resolution across multiple sites
should be considered by this working group.

The goal of this working group is to develop interoperable solutions to
solve the above problems within the Data Center and multi Data Center
environments.  This could involve multi-domain (multi-company), multi-site
and multi-vendor environments. 

The design should consider the following properties:

*	All solutions developed by ARMD WG should not require any behavior
changes on hosts, applications, or Virtual Machines being deployed in the
market. 
*	All solutions developed should not break DHCP, or any other
broadcast/multicast mechanism used by applications.
*	Evaluating the impact to IPv6 ND, and develop solutions accordingly
if needed. 
*	Should consider variety of solutions, including directory based,
proxy based, or cache based solutions. 
*	Include analysis of security concerns of IPv4 ARP requests from
malicious users. Evaluating potential security solutions and conclude if the
security threat can justify solutions. 
*	ARMD assumes the direct links to individual hosts and virtual
machines are IEEE802.3 Ethernet links.  
*	Should consider scenarios of one Ethernet network being
interconnected by another network, which can be L2VPN, pure IP, Ethernet, or
others. 
*	Should consider isolation of address resolution solutions across
different client controlled virtual subnets, i.e. hosts from one client
should be limited to getting MAC addresses from virtual hosts within its own
controlled subnet(s). Should consider a performance analysis of proposed
solutions.   

Here are the items which should not be in the scope of the working group: 

*	Re-define DHCP behavior
*	Re-define IPv6 ND security model
*	Solution developed should not expect any behavior changes for the
guest OS/applications running on the VMs
*	Direct links from hosts and virtual hosts being non-Ethernet links.

Goals and Milestones:

*	Problem statement 
*	Gap Analysis Statement within Data Center

*	Gaps within ARP
*	Gaps within IPv6 ND or autoconfiguration

*	Survey of Existing Solutions 

*	Survey of NHRP (RFC2332) & SCSP,  and their applicability to
Ethernet networks;
*	Survey of TRILL work as a potential solution; 
*	Survey of  other potential solutions, like MOOSE or SEATTLE; and
*	Other proposals

*	Security Analysis of Existing Solutions

*	Study existing solutions, like Dynamic ARP Inspection, Egress ARP
Inspection, etc, and evaluate if any of them or others should be included in
the solutions specified by this working group.  

*	Architectural Design for Next-Generation Data Center 
*	Protocol Documents

Time line for BOF presentations

Initial Problem Statement                            August   

GAP analysis                                              October 

Survey of Existing Solutions                        October

Security Analysis of Existing Solutions         October 

Architectural Design for NG                        October 

Final Problem statement for BOF                October

      Protocol Documents                                   October

Regards, Linda Dunbar

---

From: Anoop Ghanwani <[email-masked]>
Date: Sun, 12 Sep 2010 11:27:08 -0700

*   Should consider isolation of address resolution solutions across different client controlled virtual subnets, i.e. hosts from one client should be limited to getting MAC addresses from virtual hosts within its own controlled subnet(s). Should consider a performance analysis of proposed solutions.

I still have a problem with the first sentence.  There is not a concrete,
well-accepted definition of what a "client controlled virtual subnet" is,
and how forwarding takes place when something like that is defined.

Unrelated to the above, I believe the second sentence is supposed to
be a bullet on its own?

Anoop

From: Linda Dunbar [mailto:[email-masked]]
Sent: Sunday, September 12, 2010 6:54 AM
To: [email-masked]
Cc: Anoop Ghanwani; 'T Sridhar'
Subject: another revision for ARMD charter statement

After separate discussions with Force10's Sridhar and Brocade's Anoop, we have decided to make some changes to the ARMD charter statements. Here is the revised one. Please provide your comments. We plan to submit this charter statement to Internet AD Ralph and Jari next Tuesday.

As server virtualization is introduced to data centers, the number of hosts in a data center can grow dramatically.  Each physical server, which used to host one end-station, now can host hundreds of end-stations or virtual machines (VMs). Virtual machines (VMs) can be easily added and/or deleted and moved between servers.  The VM's flexible addition/deletion and moving around provide foundation for virtual hosts and virtual desktops offered by Cloud Computing services and make it possible to combine pool of servers into one single entity for ease of management and better utilization.

This rapid growth of virtual hosts could tremendously impact networks and servers. One major issue is frequent ARP (IPv4) or neighbor discovery (IPv6) requests from hosts. All hosts send out those requests frequently due to host ARP caches being aged out in minutes. With tens of thousands of hosts (each with a distinct MAC address) in one Data Center, the number of ARP (or ND) packets per second from all hosts could potentially be as high as 1,000 to 10,000/second. This rate imposes tremendous computational burden on many hosts.

Traditional subnet (VLAN) partitioning no longer works well when servers are virtualized. First of all, servers need to be on the same subnet for VM mobility. Secondly, when each server only has one host, there is only one VLAN enabled for the port towards the server. The switch can block all broadcast messages from other subnets (VLANs). When one physical server is supporting >100 Virtual Machines, i.e. >100 hosts, most likely the virtual hosts on one server are on different subnets (VLANs). If there are 50 subnets (VLANs) enabled on the switch port to the server, the server has to handle all the ARP broadcast messages on all 50 subnets (VLANs). The amount of ARP to be processed by each server is still too much.

One characteristic of a next-generation Data Center is the possibility of one subnet spanning  multiple sites. For example, a server administrator can combine servers in multiple sites into one huge resource pool for better resource management and ease of allocation.  Each of the servers in this pool might be configured with one or more VMs. If mobility is required, like VMware's vMotion, all the physical servers  (and associated VMs) need to be on the same IP subnet. Under those circumstances, one Layer 2 network (or subnet) could span across multiple sites. The multiple sites could be connected by any types of network, like L2VPN, L3VPN, Ethernet network, or simple IP networks. Efficient address resolution across multiple sites should be considered by this working group.

The goal of this working group is to develop interoperable solutions to solve the above problems within the Data Center and multi Data Center environments.  This could involve multi-domain (multi-company), multi-site and multi-vendor environments.
The design should consider the following properties:

 *   All solutions developed by ARMD WG should not require any behavior changes on hosts, applications, or Virtual Machines being deployed in the market.
 *   All solutions developed should not break DHCP, or any other broadcast/multicast mechanism used by applications.
 *   Evaluating the impact to IPv6 ND, and develop solutions accordingly if needed.
 *   Should consider variety of solutions, including directory based, proxy based, or cache based solutions.
 *   Include analysis of security concerns of IPv4 ARP requests from malicious users. Evaluating potential security solutions and conclude if the security threat can justify solutions.
 *   ARMD assumes the direct links to individual hosts and virtual machines are IEEE802.3 Ethernet links.
 *   Should consider scenarios of one Ethernet network being interconnected by another network, which can be L2VPN, pure IP, Ethernet, or others.
 *   Should consider isolation of address resolution solutions across different client controlled virtual subnets, i.e. hosts from one client should be limited to getting MAC addresses from virtual hosts within its own controlled subnet(s). Should consider a performance analysis of proposed solutions.

 *   Re-define DHCP behavior
 *   Re-define IPv6 ND security model
 *   Solution developed should not expect any behavior changes for the guest OS/applications running on the VMs
 *   Direct links from hosts and virtual hosts being non-Ethernet links.

Goals and Milestones:

 *   Problem statement

    *   Gaps within ARP

    *   Survey of NHRP (RFC2332) & SCSP,  and their applicability to Ethernet networks;
    *   Survey of TRILL work as a potential solution;
    *   Survey of  other potential solutions, like MOOSE or SEATTLE; and
    *   Other proposals

    *   Study existing solutions, like Dynamic ARP Inspection, Egress ARP Inspection, etc, and evaluate if any of them or others should be included in the solutions specified by this working group.

Regards, Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 13 Sep 2010 10:03:30 -0500

Hi Anoop, 

Thanks for pointing this out. The bullet on isolation should be removed as
the text for client controlled virtual subnet has been removed. 

The second sentence on performance will stay. 

Regards, Linda Dunbar

  _____  

From: Anoop Ghanwani [mailto:[email-masked]] 
Sent: Sunday, September 12, 2010 1:27 PM
To: Linda Dunbar; [email-masked]
Cc: 'T Sridhar'
Subject: RE: another revision for ARMD charter statement

be a bullet on its own?

Anoop

 

 

*     Evaluating the impact to IPv6 ND, and develop solutions accordingly if
needed. 

*     Should consider variety of solutions, including directory based, proxy
based, or cache based solutions. 

*     ARMD assumes the direct links to individual hosts and virtual machines
are IEEE802.3 Ethernet links.  

*     Should consider scenarios of one Ethernet network being interconnected
by another network, which can be L2VPN, pure IP, Ethernet, or others. 

Goals and Milestones:

*     Problem statement 

o    Gaps within ARP

o    Gaps within IPv6 ND or autoconfiguration

o    Survey of NHRP (RFC2332) & SCSP,  and their applicability to Ethernet
networks;

o    Survey of TRILL work as a potential solution; 

o    Survey of  other potential solutions, like MOOSE or SEATTLE; and

o    Other proposals

o    Study existing solutions, like Dynamic ARP Inspection, Egress ARP
Inspection, etc, and evaluate if any of them or others should be included in
the solutions specified by this working group.  

*     Protocol Documents

GAP analysis                                              October 

 

Regards, Linda Dunbar