Subject: [nvo3] TTL handling in an L3 service

From: Thomas Narten <[email-masked]>
Date: Tue, 26 Nov 2013 11:57:57 -0500

Hi.

In precisely defining L3 service, one question that comes up is how is
the TTL treated. That is, say NVO3 provides L3 VN service to a
TS. When TSes on the VN communicate with each other, they are always
using IP. What happens to the TTL in such packets?

I see several choices:

a) do not decrement the TTL at all. Treat the TSes as if they were directly
   attached to each other (i.e., on the same link)

b) Decrement by 1...

c) Decrement by some random amount.. :-)

Some protocols may care about TTL handling. IPv6 Neighbor Discovery,
for example, requires that ND packets be dropped if they are received
with a TTL other than 255 (i.e., they'd require choice a). I think
some other routing protocols do the same (as a way to ignore packets
from offlink "attackers").

What do tenants of an L3 service expect? Do they care (other than in
cases like ND)?

Can we just define L3 service as saying the TTL of tenant packets are
not modified by NVO3?

Any advice from L3 service providers that already provide such
services today?
 
Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 26 Nov 2013 09:28:52 -0800

Thomas, there is plenty of discussion on TTL handling for LISP in RFC 6830. And at the same time, you have to decide what traceroute behavior you want. You need to decide if tunnels are 1-hop or n hops. MPLS chose 1-hop tunnels but product configuration knobs could change the behavior. LISP choose n-hops because operators gave us requirements to see the entire path.

You will have to ask this question for QOS/TOS handling as well. David Black contributed text to the LISP working group, so you can see the resulting decisions in RFC 6830.

Dino

On Nov 26, 2013, at 8:57 AM, Thomas Narten <[email-masked]> wrote:

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Thomas Narten <[email-masked]>
Date: Tue, 26 Nov 2013 14:43:08 -0500

Hi Dino.

I'm not really asking about TTL processing for tunnels in
general... I'm asking specifically in the context of virtual network
service, where tenants are connecting to virtual networks expecting
either L2 or L3 service. What expectations  to the TSes have  when it
comes to TTL modification?

In the L2 case, tenatnts expect to be given service equivalent to an
L2 broadcast domain. I.e., nodes can talk directly to each other at
the L2 level using unicast/multicast. This is what we usually think of
as being attached to a shared "link".

But what if the service being provided is L3? In this case TSes are
only sending IP packets, but all TSes are directly reachable (i.e.,
without leaving the VN). What expectation do TSes in this type of
situation expect of the TTL processing?

Do they even care?

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 26 Nov 2013 13:45:14 -0800

> comes to TTL modification?

I think it is exactly the same case. 

> as being attached to a shared "link".

Right - but for a L2 service there is no TTL to decrement in the NVE since it forwards a packet based on a MAC header. 

> situation expect of the TTL processing?

"Directly" needs to be defined above. If the two TSes are in the same subnet then it would behave as it does today (when 2 hosts are connected to hub - I don't say switch because I don't want to complicate the discussion). 

> Do they even care?

If I am 10.0.0.1 in CA on subnet 10.0.0.0/8 and you are 10.0.0.2 in NY on subnet 10.0.0.0/8 and we are reachable via the overlay, I expect one line if output from traceroute. If you are 11.0.0.1 I expect at least 2 lines of output. 

Dino

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 26 Nov 2013 14:33:13 -0800

>> situation expect of the TTL processing?

Thomas,

I think there are two different ways to view an L3 service:
1. A closed user group which is larger than a single IP subnet, 
provisioned as part of setting up the VN.
2. A more efficient way to pass around IP packets; no need to have an 
(inner) Ethernet header if the TSs only care about IP.

I reality it is a combination of the two. The administrator that 
provisions the VN see the CUG aspect of #1. The TS and NVE 
implementations see the #2.

> If I am 10.0.0.1 in CA on subnet 10.0.0.0/8 and you are 10.0.0.2 in NY on subnet 10.0.0.0/8 and we are reachable via the overlay, I expect one line if output from traceroute. If you are 11.0.0.1 I…

Dino,

I think what you are saying is that from a host/TS perspective you'd 
expect an L3 service to have the same TTL behavior as when using a L2 
service with routers that are external to the VN.
That makes a lot of sense to me.

In that way the size and shape of the L3 VN/CUG doesn't impact the TTL 
handling.

    Erik

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 26 Nov 2013 16:03:59 -0800

> Dino,
> 
> I think what you are saying is that from a host/TS perspective you'd expect an L3 service to have the same TTL behavior as when using a L2 service with routers that are external to the VN.

Right, that is what I am saying.

> In that way the size and shape of the L3 VN/CUG doesn't impact the TTL handling.

Right.

Dino

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 27 Nov 2013 01:47:14 +0000

> -----邮件原件-----
> 发件人: nvo3 [mailto:[email-masked]] 代表 Thomas Narten
> 发送时间: 2013年11月27日 0:58
> 收件人: [email-masked]
> 主题: [nvo3] TTL handling in an L3 service
> 
> Hi.
> 
> In precisely defining L3 service, one question that comes up is how is the TTL
> treated. That is, say NVO3 provides L3 VN service to a TS. When TSes on the VN
> communicate with each other, they are always using IP. What happens to the
> TTL in such packets?
> 
> I see several choices:
> 
> Some protocols may care about TTL handling. IPv6 Neighbor Discovery, for
> example, requires that ND packets be dropped if they are received with a TTL
> other than 255 (i.e., they'd require choice a). I think some other routing
> protocols do the same (as a way to ignore packets from offlink "attackers").

Hi Thomas,

I think it's worthwhile to discuss the TTL handling issue associated with L3 service or L3 overlay. However, the above example (i.e., ND) is not accurate since it seems no need to transport the ND messages across a Layer3 overlay.

Best regards,
Xiaohu

> What do tenants of an L3 service expect? Do they care (other than in cases like
> ND)?
> 
> Can we just define L3 service as saying the TTL of tenant packets are not
> modified by NVO3?
> 
> Any advice from L3 service providers that already provide such services today?
> 
> Thomas
>

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 27 Nov 2013 01:52:51 +0000

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2013年11月27日 9:47
> 收件人: 'Thomas Narten'; [email-masked]
> 主题: 答复: [nvo3] TTL handling in an L3 service
> 
> 
> 
> > -----邮件原件-----
> > the TTL treated. That is, say NVO3 provides L3 VN service to a TS.
> > When TSes on the VN communicate with each other, they are always using
> > IP. What happens to the TTL in such packets?
> >
> > I see several choices:
> >
> > some other routing protocols do the same (as a way to ignore packets from
> offlink "attackers").
> 
> Hi Thomas,
> 
> I think it's worthwhile to discuss the TTL handling issue associated with L3
> service or L3 overlay. However, the above example (i.e., ND) is not accurate
> since it seems no need to transport the ND messages across a Layer3 overlay.

In addition, I can't imagine a scenario where routing protocol messages need to be transported across a Layer3 overlay as well in the NVO3 context.

Xiaohu

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 26 Nov 2013 18:08:48 -0800

> In addition, I can't imagine a scenario where routing protocol messages need to be transported across a Layer3 overlay as well in the NVO3 context.

I betcha many people would want to run multi-hop BGP on top of an overlay. :-)

Dino

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 27 Nov 2013 02:20:59 +0000

> -----邮件原件-----
> 发件人: nvo3 [mailto:[email-masked]] 代表 Dino Farinacci
> 发送时间: 2013年11月27日 10:09
> 收件人: Xuxiaohu
> 抄送: Thomas Narten; Xuxiaohu; [email-masked]
> 主题: Re: [nvo3] 答复: TTL handling in an L3 service
> 
> > In addition, I can't imagine a scenario where routing protocol messages need
> to be transported across a Layer3 overlay as well in the NVO3 context.
> 

IMHO, it makes sense to run BGP among NVEs. In this case, BGP messages would be transported over the underlay, rather than the overlay.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 26 Nov 2013 19:54:16 -0800

> IMHO, it makes sense to run BGP among NVEs. In this case, BGP messages would be transported over the underlay, rather than the overlay. 

What if you wanted the BGP peer to be mobile?  ;-)

Dino

---

From: Pankaj Garg <[email-masked]>
Date: Wed, 27 Nov 2013 05:19:41 +0000

The overlay networks should behave exactly as the underlay network (in terms of behavior). I think, a TS (or VM) running on overlay network shouldn't know whether it is running on top of some overlay or not. In such a network, if the tenant brings their own virtual appliance(s) to connect multiple VNs (where each VN is an L2 network), they should be able to exchange BGP or other routing protocol messages among those routers. Similarly, if NVE is doing the L3 routing and connecting multiple L2 VNs, it should be able to peer with a tenant owned router. A simple use case is a group of VNs connected via an L3 router and connected via VPN or other service to on-prem network. In this case, if the route changes happen on the on-prem network, using BGP or other routing protocol to propagate the changes to the L3 router on the cloud side would be needed.

On TTL, if we think of NVE as a single logical router, then as the traffic is forwarded between one L2 VN to another L2 VN, the TTL should decrement by 1. The behavior would be exactly the same, if these two VNs were not virtual and connected via a physical router. The behavior would also be the same if these two VNs were virtual but connected via a virtual router running in a VM. This gives consistent behavior to TS in all cases independent of where the router is running (or located).

Pankaj

> -----Original Message-----
> From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
> Sent: Wednesday, November 27, 2013 9:24 AM
> To: Xuxiaohu
> Cc: Thomas Narten; [email-masked]
> Subject: Re: [nvo3] 答复: TTL handling in an L3 service
> 
> > IMHO, it makes sense to run BGP among NVEs. In this case, BGP messages
> would be transported over the underlay, rather than the overlay.
>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Wed, 27 Nov 2013 11:06:05 +0000

TTL processing in the context of nvo3 is already described in the draft-ietf-nvo3-dataplane-requirements.
See section 3.2.2 for details.

Marc 

> From: nvo3 [mailto:[email-masked]] On Behalf Of Thomas Narten
> Sent: Tuesday, November 26, 2013 8:43 PM
> To: Dino Farinacci
> Cc: [email-masked]
> Subject: Re: [nvo3] TTL handling in an L3 service
> 
> Hi Dino.
> 
> general... I'm asking specifically in the context of virtual 
> network service, where tenants are connecting to virtual 
> networks expecting either L2 or L3 service. What expectations 
>  to the TSes have  when it comes to TTL modification?
> 
> L2 broadcast domain. I.e., nodes can talk directly to each 
> other at the L2 level using unicast/multicast. This is what 
> we usually think of as being attached to a shared "link".
> 
> But what if the service being provided is L3? In this case 
> TSes are only sending IP packets, but all TSes are directly 
> reachable (i.e., without leaving the VN). What expectation do 
> TSes in this type of situation expect of the TTL processing?
> 
> Do they even care?
> 
> Thomas
>

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 30 Nov 2013 09:08:07 +0000

> 发送时间: 2013年11月27日 5:45
> 收件人: Thomas Narten
> 抄送: [email-masked]
> 主题: Re: [nvo3] TTL handling in an L3 service
> 
> > Hi Dino.
> >
> > I'm not really asking about TTL processing for tunnels in general...
> > I'm asking specifically in the context of virtual network service,
> > where tenants are connecting to virtual networks expecting either L2
> > or L3 service. What expectations  to the TSes have  when it comes to
> > TTL modification?
> 
> "Directly" needs to be defined above. If the two TSes are in the same subnet
> then it would behave as it does today (when 2 hosts are connected to hub - I
> don't say switch because I don't want to complicate the discussion).
> 
> > Do they even care?

> If I am 10.0.0.1 in CA on subnet 10.0.0.0/8 and you are 10.0.0.2 in NY on subnet
> 10.0.0.0/8 and we are reachable via the overlay, I expect one line if output from
> traceroute. If you are 11.0.0.1 I expect at least 2 lines of output.

Thomas has asked a very worthwhile question: Do they even care? 

According to the TTL handling requirement for L3 VNI as specified in the NVo3 DP Requirement doc (see below), if you do a trace route for 11.0.0.1, the trace route output would be exactly the same as your expectation since L3 NVEs are usually acting as default GWs. That's to say, you can't see any difference when you do a trace route for an IP address outside your subnet. If you do a trace route for 10.0.0.2, the trace route output may list one or two more hops (i.e., the ingress and/or the egress NVE) than your above expectation. However, do you really care about the trace route output for an IP address which is within the same subnet as yours? What's your special purpose for doing a trace route, rather than a PING for an IP address which belongs to the same subnet as yours? 

To some extent, the L3 forwarding for intra-subnet traffic is just a special use case of the Proxy ARP mechanism [RFC1027]. That's to say, the traffic from 10.0.0.1 to 10.0.0.2 is intra-subnet traffic from the host's POV while it is also inter-subnet traffic from the router's POV. Therefore, the above "unexpected" trace route output demonstrates exactly the router hops that the intra-subnet traffic (from the tenant host POV) would traverse on the overlay path. 

*******************************
3.2.2. L3 VNI
    
       L3 VNIs MUST provide virtualized IP routing and forwarding. L3 VNIs
       MUST support per-tenant forwarding instance with IP addressing
       isolation and L3 tunneling for interconnecting instances of the same
       VNI on NVEs.
    
       In the case of L3 VNI, the inner TTL field MUST be decremented by
       (at least) 1 as if the NVO3 egress NVE was one (or more) hop(s)
       away. The TTL field in the outer IP header MUST be set to a value

    Lasserre, et al.        Expires May 12, 2014                  [Page 6]
     
    Internet-Draft        NVO3 Data Plane Requirements        November 2013

       appropriate for delivery of the encapsulated frame to the tunnel
       exit point. Thus, the default behavior MUST be the TTL pipe model
       where the overlay network looks like one hop to the sending NVE.
       Configuration of a "uniform" TTL model where the outer tunnel TTL is
       set equal to the inner TTL on ingress NVE and the inner TTL is set
       to the outer TTL value on egress MAY be supported.
********************************

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 30 Nov 2013 09:54:20 +0000

Hi Thomas,

By the way, you may also ask feedbacks from some vendors who have implemented the feature of forwarding intra-subnet traffic at L3 in their released data center network products.

From the following public information which can be googled, it seems at least the following three vendors have implemented it:

www.juniper.net/us/en/local/pdf/whitepapers/2000535-en.pdf
(quoted text: Fallback Switching
Contrail supports a hybrid mode where a virtual network is both a L2 and a L3 overlay simultaneously. In this case the
routing instances on the vRouters have both an IP FIB and a MAC FIB. For every packet, the vRouter first does a lookup
in the IP FIB. If the IP FIB contains a matching route, it is used for forwarding the packet. If the IP FIB does not contain a
matching route, the vRouter does a lookup in the MAC FIB—hence the name fallback switching.
Note that the “route first and then bridge” behavior of fallback switching is the opposite of the “bridge first and then
route” behavior of integrated routing and bridging (IRB).)

http://blogs.enterasys.com/dci-made-simple-with-onefabric/
(quoted text: Fabric Routing with IP mobility uses host routing techniques to dynamically distribute and inject host routes from the data center switch (that has fabric routing enabled) that a VM is closest connected to – and remove them from the previous closest fabric routing switch)

http://www.cisco.com/en/US/solutions/collateral/ns224/ns945/white_paper_c11-728337.pdf
(quoted text: Cisco DFA advancements include enhanced forwarding, in which IP addresses are used regardless of whether the
communication is within or between traditional Layer 2 subnets. This feature introduces several optimizations and
simplifications, including the elimination of a first-hop redundancy protocol, the use of small MAC address tables,
and optimal forwarding for all unicast frames.)

http://www.valleytalk.org/wp-content/uploads/2013/08/ciscoDFA.pdf

Best regards,
Xiaohu