Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 26 Sep 2013 07:50:27 +0200 (CEST)

On Tue, 20 Aug 2013, Suresh Krishnan wrote:

> Hi all,
>  This draft has been presented at several intarea face to face meetings
> and has received quite a bit of discussion. It has been difficult to
> gauge whether the wg is interested in this work or not. This call is
> being initiated to determine whether there is WG consensus towards
> adoption of draft-nachum-sarp-06 as an intarea WG draft. Please state
> whether or not you're in favor of the adoption by replying to this
> email. If you are not in favor, please also state your objections in
> your response. This adoption call will complete on 2013-09-04.

I would be much more interested in this work if it also solved other 
problems, such as packets sourced or destined from/to outside the network, 
to make sure it gets to a router for closest exit into a default gw 
router.

So for instance, if a VM is moved from east to west, packets destined for 
default gw should now be sent to default gw located in west. The L3 
network should also know that this IP address has moved and inject a /32 
route from the west router to make sure packets traverse the L3 network as 
far as possible before it goes into the DC network.

But I still wonder if this shouldn't be solved on L3 totally. Why can't 
datacenter standards properly use L3 for routing instead of using L2 to 
create huge L2 domains? It's inefficient if geographical distance is 
substantial due to packets taking inefficient paths due to the L3 network 
not knowing where the end IP address actually is.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 26 Sep 2013 09:21:25 +0000

(cc'd this email to L3VPN and NVo3 since this discussion may be interesting to them)

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Mikael Abrahamsson
> 发送时间: 2013年9月26日 13:50
> 收件人: Suresh Krishnan
> 抄送: Internet Area
> 主题: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt
> 

Hi Mikael,

The approach described in this draft (http://tools.ietf.org/html/draft-xu-l3vpn-virtual-subnet-01) may meet your above desire. BTW, such approach has already been implemented by some data center network vendors. 

Best regards,
Xiaohu 

> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Zhenghewen <[email-masked]>
Date: Thu, 26 Sep 2013 20:31:16 +0800

Hi, Mikael,

   The application mentioned is related to the consistence of VM IP address, exactly, it is about service consistence. In theory, absolute service consistence seems impossible because even live VM motion also introduce some milliseconds service disruption. Let's ignore the temporarily service disruption.
   There are 2 types of applications for service consistence, one for internet data centers (IDC) served for public internet users, for example the service provided by www.google.com; the other for (virtual) private data centers (vPDC) served for a tenant(e.g., a company) intranet users:

   1), For IDC, in the current real deployment, especially for IDC, operator just uses DNS to keep service consistence after VM motion, for example, before moving a VM who owns 10.10.10.10 response to www.example.com into another site where the VM will have new IP address 20.20.20.20, a new DNS entry (20.20.20.20 : www.example.com) will be added into DSN server, and then move the VM to the other site; after the completion of VM, the original DNS entry (10.10.10.10 : www.example.com) will be deleted . Maybe some users or connections will be impacted, but the impaction is very limited whatever at scope or duration. Many web sites adopt the method. Maybe that is why we do not see the live VM motion without any service disruption in real application.
   Certainly, we have other choices including host routing (e.g. virtual subnet, LISP), MAC routing.

   2), For vPDC, if VM#1 is moved from east to west through a layer 2 network (for example, IP fabric via NVO3), a gratuitous ARP will be released from the VM#1 after its motion is completed from east to west, then all other VMs in east and all other VMs in west know the new ARP entry for VM#1 (i.e., VMs in east data center know new ARP [west gateway's MAC address : VM#1's IP address] for VM#1, VMs in west data center know new ARP [VM#1's MAC address : VM#1's IP address]) according to the draft, the communications with VM#1 in the intranet will not be impacted, we get the consistence of VM IP address.
   Certainly, after the gratuitous ARP, the gateway in east and the gateway in west for the tenant also know the new ARP entry for VM#1. All packets destined to VM#1 will correctly reach VM#1.

Best regards,
Hewen Zheng

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 26 Sep 2013 14:44:32 +0200 (CEST)

On Thu, 26 Sep 2013, Zhenghewen wrote:

> Hi, Mikael,
>
>   The application mentioned is related to the consistence of VM IP 
> address, exactly, it is about service consistence. In theory, absolute 
> service consistence seems impossible because even live VM motion also 
> introduce some milliseconds service disruption. Let's ignore the 
> temporarily service disruption.

Check, but I wasn't concerned with service continuity, I was concerned 
with optimizing packet flight path.

If VM was moved from east to west and east and west are 10ms RTT away from 
each other (very far, sure), then you want to make sure that all VMs in 
east talk to the routers at east, not use a default gw that is located in 
west. You also want to make sure that packets sitting in another subnet at 
east don't have to traverse to the router at west and then back to east 
again.

---

From: Brian Haberman <[email-masked]>
Date: Fri, 27 Sep 2013 15:59:41 -0400

Suresh,
     Sorry for the late feedback, but there are some issues that should
be discussed.  Some of the following are my views of the document and
others are summarized based on feedback from others.

     To be clear, the following comments are being made as an individual
participant.

1. The idea of creating a Layer-2 NAT is rather unappealing.  Many folks
in the IETF understand the rash of issues that arise with this type of
approach.  It appears that is what happens with SARP (at least in some
instance).

2. It is unclear *who* would want to build (or has built) a layer-2
network at this size and sees the application of proxies/NATs as the
solution to scaling issues.  Are there operators who have built networks
in this way who can clearly explain the problem space?  This comes about
from the outcome of ARMD.

3. Given the length of time that this draft has been around, are there
implementations?

4. How does this approach deal with non-IP traffic?

Now with my AD hat on...

     I believe there are some issues raised by Dave Allan that have been
left unanswered, though Dave can correct me if I am wrong.  Consensus
does not require unanimity, but it does require that all concerns raised
be addressed.

Regards,
Brian

---

From: David Allan I <[email-masked]>
Date: Fri, 27 Sep 2013 20:40:44 +0000

HI Brian:

You offer a pretty good summary, to which I would add as such a solution would be highly likely to be limited to 4K VLANs (as it is implemented as a flat Ethernet technology layer). As soon as I add an overlay to address this limitation, a chunk of the surrounding rationale for the approach IMO evaporates....and if the solution was defined flat as advertised, given the amount of ranting about the 4K limit in numerous WGs, progressing a solution that embodies such a limitation would appear to be less than useful.

Although the list discussion around my comments was an interesting and informative one, I did not consider my concerns addressed. An L3 ARP proxy driving a 1:N MAC-NAT breaks a lot of stuff. IMO that is rather fundamental and more discussion would not change the facts. In that regard I cannot see how my concerns can be addressed by SARP as it stands...

I hope this helps...
cheers
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Haberman
Sent: Friday, September 27, 2013 1:00 PM
To: [email-masked]
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Suresh,
     Sorry for the late feedback, but there are some issues that should be discussed.  Some of the following are my views of the document and others are summarized based on feedback from others.

     To be clear, the following comments are being made as an individual participant.

1. The idea of creating a Layer-2 NAT is rather unappealing.  Many folks in the IETF understand the rash of issues that arise with this type of approach.  It appears that is what happens with SARP (at least in some instance).

2. It is unclear *who* would want to build (or has built) a layer-2 network at this size and sees the application of proxies/NATs as the solution to scaling issues.  Are there operators who have built networks in this way who can clearly explain the problem space?  This comes about from the outcome of ARMD.

3. Given the length of time that this draft has been around, are there implementations?

Now with my AD hat on...

     I believe there are some issues raised by Dave Allan that have been left unanswered, though Dave can correct me if I am wrong.  Consensus does not require unanimity, but it does require that all concerns raised be addressed.

Regards,
Brian

---

From: Ted Lemon <[email-masked]>
Date: Fri, 27 Sep 2013 16:45:24 -0400

> Although the list discussion around my comments was an interesting and informative one, I did not consider my concerns addressed. An L3 ARP proxy driving a 1:N MAC-NAT breaks a lot of stuff. IMO th…

I realize that for the people presenting these points it is obvious why a L2 nat is a bad idea, and I think I know at least one reason why they feel this way, but it would help to actually walk the working group through it rather than just pointing out that it is so, because I suspect that there are plenty of working group participants who are not experts on the topic, but would understand readily if the concerns were explained in more detail.

---

From: David Allan I <[email-masked]>
Date: Fri, 27 Sep 2013 20:58:58 +0000

Hi Ted:

The "scaling benefit" offered by the approach is that the set of subtending end system MAC addresses is summarized as the MAC address of the ARP proxy which overwrites the ES address in the frame. The result being that MAC tables in the core tend to be smaller.  The key thing is that this is a 1:N translation so the ARP proxy needs IP information in the frame and local ARP cache information in order to successfully translate the MAC address in frames addressed to itself to the subtending end system's MAC addresses. 

What this means is any non-IP E2E protocol (CFM, QCN etc.) is broken and how one would instrument and fault sectionalize this network is unclear. It looks like an Ethernet subnet but does not behave like one. 

Again hope this helps
Dave

---

From: Thomas Narten <[email-masked]>
Date: Fri, 27 Sep 2013 17:28:25 -0400

Given the flurry of mail, I went back and reviewed my notes on this
document.

I believe I understand the problem (reduce size of FDB), but am
skeptical that solving it via the approach in this document is
worthwhile.

1) As DaveA points out, FDB tables are large on chips these
days. However, at least initially, SARP would be implemented in the
service processor (consuming CPU cycles). It would be years (if ever)
before silicon would implement this. But in parallel, silicon will
just get better and have bigger FDB tables... Moreover, the SARP
motivation section specifically cites a shortage of processor cycles
as a problem ... but doing SARP in software will increase the demands
on the CPU...  its unclear to me that the increased CPU burdens SARP
implies would be offset by the reduced amount of ARP processing that
the solution is hoping will result... I.e., at least in the short
term, it's unclear that SARP would actually help in practice.

2) Doing L2 NAT at line rates (an absolute requirement for edge
devices) will only happen if this is done in silicon. I don't see that
happening unless there is strong support from vendors/operators/chip
makers... Software based solutions simply will not have sufficient
performance. I think the IEEE would be a better place to have the
discussion about what L2 chip makers are willing to implement...

3) Only works for IP. Doesn't work for non-IP L2. Doesn't that only
solve part of the problem?

4) High availability is complex (but presumably also necessary). It
smells a lot like multi-chassis LAG (with the need to synchronize
state between tightly-coupled peers). Although IEEE is working in this
general area now, there are tons of vendor-specific solutions for
doing this at L2. Do we really want to tackle standardizing this in
the IETF?  Isn't the relevant expertise for this general area in IEEE?

5) This solution touchs on both L2 (NATing L2 addresses) and L3
(ARPing for IP addresses). We absolutely would need coordinate with
IEEE before deciding to take on this work.

6) ARP caching will be a tradeoff. You want to cache responses for
better performance, but long cache timeouts will result in black holes
after VMs move. There is no great answer here. I expect long timeouts
to be unacceptable operationally, which means that the benefits of
caching will be limited (and these benefits are the key value
proposition of this approach). It is really hard to estimate whether
the benefits will be sufficient in practice. Gratuitous ARPs can help,
but they are not reliable, so they will help, but have limitations...

7) I haven't fully worked this out, but I wonder if loops can form
between proxies. There is the notion that when a VM moves, proxies
will need to update their tables. But careful analysis will be needed
to be sure that one can't ever have loops where proxies end up
pointing to each other. And since the packets are L2 (with no TTL),
such loops would be disasterous. (Note point 6: we are using
heuristics (like gratuitous ARP) to get tables to converge after a
change. Heuristics tend to have transient inconsistencies.. i.e.,
possibly leading to loops.)

Again, overall, I understand the generic problem and that it would be
nice to have a solution. However, I don't see a simple solution
here. I see a fair amount of complexity, and I'm skeptical that it's
worth it (e.g., when the next gen of silicon will just have a larger
FDB).

What I'd really like to see (before having the IETF commit to this
work) is:

1) operators who are feeling the pain described in the document
stepping forward and saying they think the solution being proposed is
something they would be willing to deploy and is better than other
approaches they have in their toolkit.

2) Vendors (including silicon) saying they see the need for this and
think the would implement it.

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Sep 2013 21:32:48 +0000

Ted, 

Thanks for the comments and reply. 

Again, the draft is not about building one large Layer 2.

The proposed proxy gateway is to enable subnets not to be restricted physically together, i.e. allowing the gateway nodes in one access domain to be the "proxy gateway" for all the hosts attached regardless those hosts' subnets. 

The draft is allow all the hosts attached to one "gateway" to be represented by this gateway, very much like today's router represent all the hosts attached.    

As for L2 NAT mentioned in the draft, there are issues and the benefits. The benefits have been documented in the draft and many of our emails. The issues need to be addressed. That is why we would like the WG to contribute. 

There are a lot of issues with L3 NAT too, but L3 NAT is one of the key components that make today's internet scale. 

Linda

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Ted Lemon
> Sent: Friday, September 27, 2013 3:45 PM
> To: David Allan I
> Cc: [email-masked]
> > Although the list discussion around my comments was an interesting
> and informative one, I did not consider my concerns addressed. An L3
> ARP proxy driving a 1:N MAC-NAT breaks a lot of stuff. IMO that is
> rather fundamental and more discussion would not change the facts. In
> that regard I cannot see how my concerns can be addressed by SARP as it
> stands...
> 
> I realize that for the people presenting these points it is obvious why
> a L2 nat is a bad idea, and I think I know at least one reason why they
> feel this way, but it would help to actually walk the working group
> through it rather than just pointing out that it is so, because I
> suspect that there are plenty of working group participants who are not
> experts on the topic, but would understand readily if the concerns were
> explained in more detail.
>

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Sep 2013 21:36:06 +0000

David, 

> What this means is any non-IP E2E protocol (CFM, QCN etc.) is broken
> and how one would instrument and fault sectionalize this network is
> unclear. 

[Linda] The proposed "proxy gateway" only represents the IP applications' end hosts address by the "gateway address" (i.e. only proxy them when the hosts issues ARP/ND). 

For CFM, QCN and non-IP E2E, no proxy address is used. 

Linda

It looks like an Ethernet subnet but does not behave like one.

---

From: David Allan I <[email-masked]>
Date: Fri, 27 Sep 2013 21:40:51 +0000

HI Linda:

You wrote:

Then I still need a complete MAC table for  these things to work as I need both end system AND ARP proxy addresses in all FDBs, so the ARP proxy has only added to the FDB size, not diminished it. While rendering my OAM protocols as less effective as they are not instrumenting the path taken by IP traffic. They bypass the ARP proxy logic.

Make sense?
Dave

-----Original Message-----
From: Linda Dunbar [mailto:[email-masked]] 
Sent: Friday, September 27, 2013 2:36 PM
To: David Allan I; Ted Lemon
Cc: [email-masked]
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

David, 

Linda

> On Sep 27, 2013, at 4:40 PM, David Allan I 
> <[email-masked]>
> wrote:
> that regard I cannot see how my concerns can be addressed by SARP as 
> it stands...
> 
> I realize that for the people presenting these points it is obvious 
> why a L2 nat is a bad idea, and I think I know at least one reason why 
> they feel this way, but it would help to actually walk the working 
> group through it rather than just pointing out that it is so, because 
> I suspect that there are plenty of working group participants who are 
> not experts on the topic, but would understand readily if the concerns 
> were explained in more detail.
>

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Sep 2013 21:53:52 +0000

David, 

The CFM, OAM are exchanged among switches. The number of switches are limited. 

The proxy gateway is to enable edge nodes not to have individual addresses for end hosts in remote domains. The edge nodes only need to have the remote gateway addresses and remote switches' addresses. Those addresses are limited no matter how subnet is spread and how diverse servers are virtualized. 

Linda

> From: David Allan I [mailto:[email-masked]]
> Sent: Friday, September 27, 2013 4:41 PM
> To: Linda Dunbar; Ted Lemon
> Cc: [email-masked]
> [Linda] The proposed "proxy gateway" only represents the IP
> applications' end hosts address by the "gateway address" (i.e. only
> proxy them when the hosts issues ARP/ND).
> Then I still need a complete MAC table for  these things to work as I
> need both end system AND ARP proxy addresses in all FDBs, so the ARP
> proxy has only added to the FDB size, not diminished it. While
> rendering my OAM protocols as less effective as they are not
> instrumenting the path taken by IP traffic. They bypass the ARP proxy
> logic.
> 
> Make sense?
> Dave
>

---

From: David Allan I <[email-masked]>
Date: Fri, 27 Sep 2013 22:27:24 +0000

Hi Linda:

This sounds like the location of a proxy could be arbitrary in an Ethernet network, is this what you are suggesting?

Thanks
Dave

-----Original Message-----
From: Linda Dunbar [mailto:[email-masked]] 
Sent: Friday, September 27, 2013 2:54 PM
To: David Allan I; Ted Lemon
Cc: [email-masked]
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

David, 

Linda

> > rather fundamental and more discussion would not change the facts. 
> > In that regard I cannot see how my concerns can be addressed by SARP 
> > as it stands...
> >
> > group through it rather than just pointing out that it is so, 
> > because I suspect that there are plenty of working group 
> > participants who are not experts on the topic, but would understand 
> > readily if the
> concerns

---

From: Tal Mizrahi <[email-masked]>
Date: Tue, 1 Oct 2013 08:26:19 +0200

Hi Brian,

Thanks for your inputs.

> Given the length of time that this draft has been around, are there implementations?

Speaking from a chip vendor's perspective: yes, the data plane aspects of SARP are implemented in some of our silicons. Obviously SARP also requires functionality in the control plane, but since this functionality would typically be implemented in software, as a chip vendor it is less relevant for us.

Tal Mizrahi.
Marvell.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Haberman
Sent: Friday, September 27, 2013 11:00 PM
To: [email-masked]
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Suresh,

Now with my AD hat on...

Regards,
Brian

---

From: Tal Mizrahi <[email-masked]>
Date: Tue, 1 Oct 2013 09:01:59 +0200

Hi Thomas,

Thanks for taking the time to review the draft.

> However, at least initially, SARP would be implemented in the service processor (consuming CPU cycles). It would be years (if ever) before silicon would implement this.

Speaking as a chip vendor: our existing silicons support the data plane functionality of SARP. In fact, I am taking a wild guess (but I may be wrong) that some of our competitors' silicons that support IP/TRILL routing also support the data plane functionality of SARP, simply because from a data plane perspective SARP performs a somewhat similar functionality: address lookup + MAC address replacement.
In general, I don't think anyone intended for the data plane of SARP to be implemented in software.

> As DaveA points out, FDB tables are large on chips these days.
> silicon will just get better and have bigger FDB tables...

Right, but the *demand* for bigger FDB tables is also increasing, even more rapidly. Moreover, the FDB table is typically the biggest table in a switch silicon, which means that a large FDB table significantly increases the cost of the silicon.
BTW, I took a look at RFC6820 (I am sure you are familiar with this document :) ), and it certainly addresses MAC table size as a real issue: " When L3 extends only to aggregation switches (see Section 6.4.2), however, MAC table size limitations can be a real issue."

Thanks,
Tal.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 12:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Given the flurry of mail, I went back and reviewed my notes on this document.

I believe I understand the problem (reduce size of FDB), but am skeptical that solving it via the approach in this document is worthwhile.

1) As DaveA points out, FDB tables are large on chips these days. However, at least initially, SARP would be implemented in the service processor (consuming CPU cycles). It would be years (if ever) before silicon would implement this. But in parallel, silicon will just get better and have bigger FDB tables... Moreover, the SARP motivation section specifically cites a shortage of processor cycles as a problem ... but doing SARP in software will increase the demands on the CPU...  its unclear to me that the increased CPU burdens SARP implies would be offset by the reduced amount of ARP processing that the solution is hoping will result... I.e., at least in the short term, it's unclear that SARP would actually help in practice.

2) Doing L2 NAT at line rates (an absolute requirement for edge
devices) will only happen if this is done in silicon. I don't see that happening unless there is strong support from vendors/operators/chip makers... Software based solutions simply will not have sufficient performance. I think the IEEE would be a better place to have the discussion about what L2 chip makers are willing to implement...

3) Only works for IP. Doesn't work for non-IP L2. Doesn't that only solve part of the problem?

4) High availability is complex (but presumably also necessary). It smells a lot like multi-chassis LAG (with the need to synchronize state between tightly-coupled peers). Although IEEE is working in this general area now, there are tons of vendor-specific solutions for doing this at L2. Do we really want to tackle standardizing this in the IETF?  Isn't the relevant expertise for this general area in IEEE?

5) This solution touchs on both L2 (NATing L2 addresses) and L3 (ARPing for IP addresses). We absolutely would need coordinate with IEEE before deciding to take on this work.

6) ARP caching will be a tradeoff. You want to cache responses for better performance, but long cache timeouts will result in black holes after VMs move. There is no great answer here. I expect long timeouts to be unacceptable operationally, which means that the benefits of caching will be limited (and these benefits are the key value proposition of this approach). It is really hard to estimate whether the benefits will be sufficient in practice. Gratuitous ARPs can help, but they are not reliable, so they will help, but have limitations...

7) I haven't fully worked this out, but I wonder if loops can form between proxies. There is the notion that when a VM moves, proxies will need to update their tables. But careful analysis will be needed to be sure that one can't ever have loops where proxies end up pointing to each other. And since the packets are L2 (with no TTL), such loops would be disasterous. (Note point 6: we are using heuristics (like gratuitous ARP) to get tables to converge after a change. Heuristics tend to have transient inconsistencies.. i.e., possibly leading to loops.)

Again, overall, I understand the generic problem and that it would be nice to have a solution. However, I don't see a simple solution here. I see a fair amount of complexity, and I'm skeptical that it's worth it (e.g., when the next gen of silicon will just have a larger FDB).

1) operators who are feeling the pain described in the document stepping forward and saying they think the solution being proposed is something they would be willing to deploy and is better than other approaches they have in their toolkit.

2) Vendors (including silicon) saying they see the need for this and think the would implement it.

Thomas