Subject: [nvo3] draft-ietf-nvo3-encap-00 should add considerations of traversing NAPT

From: Linda Dunbar <[email-masked]>
Date: Wed, 12 Jul 2017 19:37:09 +0000

Sami, et al,

The draft-ietf-nvo3-encap-00 is written very clear.

However, the Section 6 (Common Encapsulation Considerations) should add a sub-section on the consideration of traversing NAPT.  Encapsulated traffic could go to different data centers or WAN, which could go through Network Address Port Translation devices

Using VxLAN as an example: VxLAN specification [RFC 7348] uses a set of Port numbers to achieve better traffic distribution among multiple paths, which is fine within one data center, but causing trouble when traversing NAPT. NAPT use Port number to map back the source address. With a set of port numbers, NAPT can't easily figure out the reverse direction traffic's final IP addresses.

In addition, since the IP of packets change through NAPT device, it can mess up the learning of the peer NVE used in encapsulation.

STUN can be used to get changed IP and port from NAPT device, but it requires NAPT device support STUN. That's not available in some scenarios. Furthermore, it can't solve the aforementioned five-tuple issue.

VXLAN over IPSec may be used to deal with the above problems, but IPSec brings up to 88 bytes of overhead plus the key distribution management, which can lower the efficiency.

Suggestion: Add Section 6.10 Traversing NAPT consideration.
I can help to provide the text if you all think the suggestion is acceptable.

We can discuss more in Prague.

Thanks, Linda Dunbar

Huawei USA IP Technology Lab
5340 Legacy Drive,
Plano, TX 75024
Tel: +1 469-277 - 5840
Fax: +1 469 -277 - 5900

---

From: Sami Boutros <[email-masked]>
Date: Wed, 12 Jul 2017 20:56:48 +0000

Thanks Linda for your comments,

Let’s discuss more in Prague to understand the use case.

Thanks,

Sami
From: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, July 12, 2017 at 12:37 PM
To: NVO3 <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: draft-ietf-nvo3-encap-00 should add considerations of traversing NAPT
Resent-From: <[email-masked]<mailto:[email-masked]>>
Resent-To: Sami Boutros <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>
Resent-Date: Wednesday, July 12, 2017 at 12:37 PM

Sami, et al,

Using VxLAN as an example: VxLAN specification [RFC 7348] uses a set of Port numbers to achieve better traffic distribution among multiple paths, which is fine within one data center, but causing trouble when traversing NAPT. NAPT use Port number to map back the source address. With a set of port numbers, NAPT can’t easily figure out the reverse direction traffic’s final IP addresses.

STUN can be used to get changed IP and port from NAPT device, but it requires NAPT device support STUN. That’s not available in some scenarios. Furthermore, it can’t solve the aforementioned five-tuple issue.

Thanks, Linda Dunbar

---

From: Dan Wing <[email-masked]>
Date: Wed, 12 Jul 2017 23:08:32 +0000

> On Jul 12, 2017, at 12:37 PM, Linda Dunbar <[email-masked]> wrote:
> 
> Sami, et al,
> 
>  
> 
> Using VxLAN as an example: VxLAN specification [RFC 7348] uses a set of Port numbers to achieve better traffic distribution among multiple paths, which is fine within one data center, but causing trouble when traversing NAPT.

You're describing a problem with Geneve, which mimics VXLAN in that both of them suggest using a wide range of UDP ports to help underlay ECMP and to help receiver CPU load balancing, specifically this text of https://tools.ietf.org/html/draft-ietf-nvo3-geneve:

   Source port:  A source port selected by the originating tunnel
      endpoint.  This source port SHOULD be the same for all packets
      belonging to a single encapsulated flow to prevent reordering due
      to the use of different paths.  To encourage an even distribution
      of flows across multiple links, the source port SHOULD be
      calculated using a hash of the encapsulated packet headers using,
      for example, a traditional 5-tuple.  Since the port represents a
      flow identifier rather than a true UDP connection, the entire
      16-bit range MAY be used to maximize entropy.

If a reduced set of source ports is used instead, as you propose, the ECMP and CPU load balancing benefits are lost.  That seems problematic.

> NAPT use Port number to map back the source address. With a set of port numbers, NAPT can’t easily figure out the reverse direction traffic’s final IP addresses.

The reverse traffic doesn't use the inverted 5-tuple.  The reverse traffic is sent to the Geneve destination port (6081), and a firewall or NAT or NAPT mapping is necessary for UDP/6081 traffic -- on both datacenters, which both probably have their own underlay NAPTs.  Those firewalls (or NATs or NAPTs) need to have appropriate pinholes for UDP/6081.

> In addition, since the IP of packets change through NAPT device, it can mess up the learning of the peer NVE used in encapsulation.

The underlay did the NAPT, so I don't see a problem with the NVE overlay getting confused.  Could you explain in more detail?

> STUN can be used to get changed IP and port from NAPT device, but it requires NAPT device support STUN.

NAPT devices are not expected to implement STUN.  STUN is expected to be implemented in the hosts behind the NAT and on a server on the other side of the NAT (usually on a server on the Internet).  See Figure 1 on page 6 of https://tools.ietf.org/html/rfc5389#page-6.

> That’s not available in some scenarios. Furthermore, it can’t solve the aforementioned five-tuple issue.
> 
>  
> 
> VXLAN over IPSec may be used to deal with the above problems,

Both Geneve and VXLAN run over UDP, and both use a fixed destination port (rather than inverted 5-tuple) for return traffic.  Not sure how VXLAN succeeds at dealing with the above problems, but I would love to learn.

> but IPSec brings up to 88 bytes of overhead plus the key distribution management, which can lower the efficiency.

Should be able to use IPsec transport mode, which is more around 40 bytes overhead.

-d

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_nvo3&d=DwICAg&c=uilaK90D4TOVoH58JNXRgQ&r=IMDU0f3LtPMQf5XkZ06fNg&m=60T3yKN2I7oCxe8OH9mfZix-2ykSSjL-P-RoXCkZGdg&s=q7A_LzZuDp-yZnlA7Xw_N7yuLk4HO7K07jgl3Z78Ixg&e=

---

From: Dan Wing <[email-masked]>
Date: Wed, 12 Jul 2017 23:08:33 +0000

-d

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Jul 2017 06:52:29 -0700

FWIW, why not just add the entropy in the IPv6 flow ID rather than
expecting it at the transport layer? Intermediate network devices should
be relying only on the flow ID for that entropy anyway.

(and yes, that doesn't solve the problem for IPv4, but perhaps that's a
good reason to encourage use of IPv6)

Joe

On 7/12/2017 4:08 PM, Dan Wing wrote:
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 13 Jul 2017 19:31:09 +0000

Dan,

Thanks for pointing out the Geneve's processing for traversing VxLAN.

First of all I wasn't proposing using reduced set of source ports. I think that the general document needs to add the consideration for traversing NAT because it is an issue for IPv4. Joe Touch suggested adding the entropy in the IPv6 flow ID as a solution for IPv6.

As for your suggested approach in Geneve:
      The reverse traffic is sent to the Geneve destination port (6081), and a firewall or NAT or NAPT mapping is necessary for UDP/6081 traffic -- on both datacenters, which both probably have their own underlay NAPTs.  Those firewalls (or NATs or NAPTs) need to have appropriate pinholes for UDP/6081.

Does it mean all reverse traffic only use UDP port 6081?
Or all the NAT device convert the NVE’s UDP port 6081 to multiple port numbers?

Thanks,
Linda

-----Original Message-----
From: Dan Wing [mailto:[email-masked]]
Sent: Wednesday, July 12, 2017 6:09 PM
To: Linda Dunbar <[email-masked]>
Cc: NVO3 <[email-masked]>; [email-masked]
Subject: Re: [nvo3] draft-ietf-nvo3-encap-00 should add considerations of traversing NAPT

> On Jul 12, 2017, at 12:37 PM, Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:
>
> Sami, et al,
>
>
>

-d

> [email-masked]<mailto:[email-masked]>

---

From: Dan Wing <[email-masked]>
Date: Thu, 13 Jul 2017 19:39:04 +0000

> On Jul 13, 2017, at 6:52 AM, Joe Touch <[email-masked]> wrote:
> 

I don't know if IPv6 routers do ECMP using flow ID, and I don't know if physical NICs do RSS CPU load balancing based on IPv6 flow ID.

> good reason to encourage use of IPv6)

Yeah, for v4 we're stuck with using source UDP port to provide the ECMP and receiver benefit.

What of NAT64?

-d

>> You're describing a problem with Geneve, which mimics VXLAN in that both of them suggest using a wide range of UDP ports to help underlay ECMP and to help receiver CPU load balancing, specifically this text of https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dnvo3-2Dgeneve-3A&d=DwIDaQ&c=uilaK90D4TOVoH58JNXRgQ&r=IMDU0f3LtPMQf5XkZ06fNg&m=uud2CHHWYVN8GxM5H4uIwsteqFFoakAhWdD8nCVDJKA&s=J0Hb0SpNIBjHmMAaoXKIM0W89HrWqzozPsq5n3qmD00&e= 
>> 
>> NAPT devices are not expected to implement STUN.  STUN is expected to be implemented in the hosts behind the NAT and on a server on the other side of the NAT (usually on a server on the Internet).  See Figure 1 on page 6 of https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc5389-23page-2D6&d=DwIDaQ&c=uilaK90D4TOVoH58JNXRgQ&r=IMDU0f3LtPMQf5XkZ06fNg&m=uud2CHHWYVN8GxM5H4uIwsteqFFoakAhWdD8nCVDJKA&s=TCHbljyS6SjK00_hvov-AWPmrDHVbyhVuV7ldIdWQoc&e= .
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_nvo3&d=DwIDaQ&c=uilaK90D4TOVoH58JNXRgQ&r=IMDU0f3LtPMQf5XkZ06fNg&m=uud2CHHWYVN8GxM5H4uIwsteqFFoakAhWdD8nCVDJKA&s=rLTq8CpFk2GwGniHCqktWbL0-O8O-nNXMIfE0k5iodw&e= 
>

---

From: Dan Wing <[email-masked]>
Date: Thu, 13 Jul 2017 19:51:40 +0000

> Does it mean all reverse traffic only use UDP port 6081?

Right, the reverse traffic is sent using destination UDP port 6081, and does not flip the 5-tuple.  If it flipped the 5-tuple, then one NVE would have to start transmitting first, or we would have to build ICE (RFC5245) and add a signaling protocol between the datacenters (which ICE requires).

> Or all the NAT device convert the NVE’s UDP port 6081 to multiple port numbers?

I don't understand 'convert ... to multiple port numbers'.

-d

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Jul 2017 12:59:19 -0700

> I don't know if IPv6 routers do ECMP using flow ID, and I don't know if physical NICs do RSS CPU load balancing based on IPv6 flow ID.
I don't either, but *if* they (either one) does, they really should.

RFC 6438 discusses this issue and implies that flow ID isn't used
because it's mostly zero, but I wouldn't be surprised if that were to
change over time.

> Yeah, for v4 we're stuck with using source UDP port to provide the ECMP and receiver benefit.

Sure, that'd help, except we'd have to expect that an IPv4 router would
know to jump into the IPv6 flow ID to do ECMP. That seems more of a
stretch than the above, IMO.

Joe

---

From: Dan Wing <[email-masked]>
Date: Thu, 13 Jul 2017 20:23:07 +0000

> On Jul 13, 2017, at 12:59 PM, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Sorry, I wasn't clear.  I'm not worried of the NAT64 device itself, but rather the IPv4 network beyond the NAT64, should we use IPv6 flow-id as proposed.

If there are different ways of expressing ECMP and RSS (receiver side scaling) for IPv6 versus IPv4, my worry is what a NAT64 is supposed to do with the IPv6 packet (containing a flow-id) it translates to IPv4.  If traffic from IPv6 is using Flow-ID to provide ECMP and RSS and it hits a NAT64, is the NAT64 now needing to do NAPT64 (port translation) in an attempt to provide some Flow-ID-like functionality to the source UDP port number?  If so, that's new for NAT64, and can't work with stateless translation (IVI and related techniques).  If not, the IPv4 network suffers (no ECMP) and receiver suffers (because no RSS) because the sender was on IPv6 and the sender used flow-id.  In the other direction, IPv4 to IPv6, it seems easy for the NAT64 (stateless or stateful) to copy the UDP source port number into the IPv6 flow-id field, or do a cute 20-bit hash of the IPv4 5-tuple, or whatever.  If we spec'd that, NAT64 in the IPv4->IPv6 direction would align with RFC6438 and provide more ergs to IPv6 flow-id. 

However, I remain troubled by the IPv6->IPv4 direction with NAT64, should we use only Flow-ID on the IPv6 side.

-d

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Jul 2017 13:29:44 -0700

On 7/13/2017 1:23 PM, Dan Wing wrote:
> If there are different ways of expressing ECMP and RSS (receiver side scaling) for IPv6 versus IPv4, my worry is what a NAT64 is supposed to do with the IPv6 packet (containing a flow-id) it translates to IPv4.  If traffic from IPv6 is using Flow-ID to provide ECMP and RSS and it hits a NAT64, is the NAT64 now needing to do NAPT64 (port translation) in an attempt to provide some Flow-ID-like functionality to the source UDP port number?  
Well, you're translating for a more capable protocol to a less-capable
one. Losing information is part of that game.

There's no way to do NAT64 translation without losing some of the info
in the IPv6 header, period.

It MIGHT try to use the flow ID to guide source UDP port generation, but
that assumes there's a UDP header to play with in addition to the IPv4
header. That's not NAT64 anymore.

> If so, that's new for NAT64, and can't work with stateless translation (IVI and related techniques).  If not, the IPv4 network suffers (no ECMP) and receiver suffers (because no RSS) because the sender was on IPv6 and the sender used flow-id.  In the other direction, IPv4 to IPv6, it seems easy for the NAT64 (stateless or stateful) to copy the UDP source port number into the IPv6 flow-id field, or do a cute 20-bit hash of the IPv4 5-tuple, or whatever.  If we spec'd that, NAT64 in the IPv4->IPv6 direction would align with RFC6438 and provide more ergs to IPv6 flow-id. 
>

I don't think we should try to lobotomize an elegant IPv6 solution "just
in case" it hits NAT64.

IMO, that's NAT64's problem to deal with (or not).

Joe

---

From: Dan Wing <[email-masked]>
Date: Thu, 13 Jul 2017 20:53:13 +0000

> On Jul 13, 2017, at 1:29 PM, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Yeah, well, then there is IPv6's UDP checksum versus IPv4's.  :-)

> I don't think we should try to lobotomize an elegant IPv6 solution "just

IPv6 flow-id is only elegant if, on a v6 network, it provides us same ECMP and RSS benefits as UDP source port.  We haven't concluded we get (or don't get) ECMP and RSS yet.

> IMO, that's NAT64's problem to deal with (or not).

I could envision a knob that if the encapsulated traffic is traversing a NAT64, then it could use the in-elegant solution (UDP source port number) *and* the IPv6 flow-ID.  An SDN controller could (or should) know if the traffic will traverse a NAT64.  If the encapsulated traffic remains on v6 (and flow-ID gives us ECMP and gives us RSS), then it could avoid putting flow entropy into the UDP source port.

-d

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Jul 2017 13:59:20 -0700

> I could envision a knob that if the encapsulated traffic is traversing a NAT64, then it could use the in-elegant solution (UDP source port number) *and* the IPv6 flow-ID.  An SDN controller could (…
Agreed.

Seems like all of this is worth including in the discussion.

Joe

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 13 Jul 2017 21:34:08 -0400

> good reason to encourage use of IPv6)

Given that we're talking about encapsulations, I see that Geneve has a
"Virtual Network Identifier" field that can be treated as part of the
entropy.  If it isn't needed to identifiy a virtual network, it can
loaded with entropy from the encapsulated packet.  In any case, packets
with different NVI values can be freely reordered.

And there is an 8-bit reserved field which could be defined to be
entropy.

These fields are at a fixed location relative to the containing headers,
so they're easy to find.

It might also be useful to define an option entirely for containing flow
identifiers or entropy -- the essential definition being that packet
routing is allowed to treat it as entropy, two packets with different
values for this field may be reordered freely.  And recommend that if
this option is used, it should be the first one, so routers could find
it easily if it is present.

Dale

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Jul 2017 20:40:51 -0700

> These fields are at a fixed location relative to the containing headers,
Yes, but I sincerely doubt router vendors will peek into GENEVE headers.
IMO, they'll either use IPv6 flow IDs or look for transport flow
context. Even a NVI isn't really flow information, so might not have any
bearing on whether a set of packets (with the same NVI) should maintain
their relative order.

Joe

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 14 Jul 2017 22:11:25 -0400

Joe Touch <[email-masked]> writes:
> Even a NVI isn't really flow information, so might not have any

Well, the definition of "flow information" is not in the positive
sense -- if the value is the same between two packets, their order must
be maintained -- but rather in the negative sense -- if the value is
*different* between two packets, their order *need not* be maintained.
That causes devices to behave conservatively, in that they are allowed
to reorder packets only if they have specific knowledge that the packets
are "in different flows".  From that point of view, if a device knows
that a certain field is an NVI, then it can treat that field as flow
information.

Dale

---

From: Dan Wing <[email-masked]>
Date: Sat, 15 Jul 2017 02:36:20 +0000

> On Jul 14, 2017, at 7:11 PM, Dale R. Worley <[email-masked]> wrote:
> 

Using Geneve VNI doesn't take us towards that goal, though.  On a simple network, there is only one VNI (one virtual network), and if we used solely VNI for underlay ECMP, all tunneled packets would go over the same ECMP path.  We want each tunneled TCP flow to take one path, and ideally the next tunneled TCP flow taking another ECMP path.

-d

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 20 Jul 2017 22:19:46 -0400

Dan Wing <[email-masked]> writes:
> Using Geneve VNI doesn't take us towards that goal, though.  On a
> simple network, there is only one VNI (one virtual network), and if we
> used solely VNI for underlay ECMP, all tunneled packets would go over
> the same ECMP path.  We want each tunneled TCP flow to take one path,
> and ideally the next tunneled TCP flow taking another ECMP path.

I'm mixing a number of points here.  In order from what I consider most
important to least:

- It would be desirable to have a Geneve option to contain entropy, so
  that in situations where outer headers don't provide a good place for
  entropy, devices don't have to look further than the Geneve header.

- There is an unused octet in the Geneve header.  It might be useful to
  define it to contain entropy for ECMP.  (Unfortunately, it's only one
  byte.(

- ECMP can use as entropy any field which is "flow identification", that
  is, where intermediate devices are allowed to disorder packets which
  have different values of the flow identification field.  In
  particular, any virtual network identifier serves as flow
  identification.  Thus, we can specify in Geneve that any device
  handling packets with Geneve headers can disorder packets with
  different values of the VNI field.  But a consequence of *that* is
  that in situations where the VNI field is not used to communicate
  useful information (presumably the additional options are the reason
  Geneve is being used), the VNI field can be used for entropy.

Dale

---

From: Dan Wing <[email-masked]>
Date: Fri, 21 Jul 2017 17:20:45 +0000

> On Jul 20, 2017, at 7:19 PM, Dale R. Worley <[email-masked]> wrote:
> 

For both points above:   are there cases where the outer header can't provide entropy of the inner headers?  With IPv4, Geneve is carried over UDP and the UDP source port works fine.  With IPv6, Geneve is carried over UDP and the UDP source port and IPv6 flow label can both be used.  Is Geneve carried over something that isn't UDP?  Said another way, do we want to burn Geneve option bits (and MTU) with this entropy information?

>  Geneve is being used), the VNI field can be used for entropy.

Feels like a lot of rules for a Geneve-aware router to follow for ECMP.  But it would be worth elaborating on this idea some more to see if it can become a simple rule for the Geneve-aware router.  I'm thinking:

  If packet is IPv6 and flow-label > 0 then EMCP hash of [ source IP || destination IP || flow-label ]
  elseif packet is UDP
      and packet is Geneve then ECMP using hash of [ source IP || Geneve VNI || Geneve-entropy option ]
      else ECMP using hash of [ source IP || destination IP || source UDP port || destination UDP port ]
  elseif packet is TCP or SCTP or DCCP
      ECMP hash of [ source IP || destination IP || source TCP/SCTP/DCCP port || destination TCP/SCTP/DCCP port ]
  else
      don't ECMP

(This is starting down the path of a Router Requirements spec.)

-d

---

From: Joe Touch <[email-masked]>
Date: Fri, 21 Jul 2017 11:37:53 -0700

Hi, all,

> For both points above:   are there cases where the outer header can't provide entropy of the inner headers?  With IPv4, Geneve is carried over UDP and the UDP source port works fine.  With IPv6, Ge…
This seems a reasonable way forward.

I.e., entropy is useful to ECMP only if routers actually use those bits.
Using existing header fields as they're already interpreted by ECMP
makes more sense than expecting routers to peek into Geneve.

Joe

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 21 Jul 2017 20:31:10 -0400

Dan Wing <[email-masked]> writes:
> For both points above:   are there cases where the outer header can't
> provide entropy of the inner headers?  With IPv4, Geneve is carried
> over UDP and the UDP source port works fine.  With IPv6, Geneve is
> carried over UDP and the UDP source port and IPv6 flow label can both
> be used.  Is Geneve carried over something that isn't UDP?  Said
> another way, do we want to burn Geneve option bits (and MTU) with this
> entropy information?

I envision Geneve -- as a *generalized* encapsulation technique -- being
used in a broader class of applications.  E.g., Geneve directly over IP,
or over 802.1, or possibly even Geneve directly over the physical medium
(presumably for point-to-point use).  I've always been in favor of
generalizing mechanisms, as long as nobody is *required* to support its
use in full generality and as long as it doesn't introduce overhead into
systems that don't need the generality.

Hence, in this case, having ways of permitting an operational
environment to put entropy in Geneve headers that don't constrain users
of Geneve in more prosaic circumstances.

Dale