Subject: [Rtg-dt-encap-considerations] Updated draft

From: Erik Nordmark <[email-masked]>
Date: Mon, 09 Mar 2015 11:59:40 -0700

I've pulled in the edits/additions from Tom.
(The encap.diff file shows what changed.)

I think this is good enough to submit. Any show stoppers?

I have a few minutes to handle those after lunch and before the deadline.

    Erik

--- encap-00a.txt	2015-03-09 11:51:08.000000000 -0700
+++ encap-00.txt	2015-03-09 11:51:22.000000000 -0700
@@ -30,7 +30,7 @@
    common issues for the different data plane encapsulations being
    discussed in the NVO3 and SFC working groups and also in the BIER
    BoF, and also to look at the relationship between such encapsulations
-   in the case when they might be used at the same time.  The purpose of
+   in the case that they might be used at the same time.  The purpose of
    this design team is to discover, discuss and document considerations
    across the different encapsulations in the different WGs/BoFs so that
    we can reduce the number of wheels that need to be reinvented in the
@@ -122,11 +122,11 @@
    Related Work section points at some prior work that relates to the
    encapsulation considerations in this document.
 
-   The encapsulation protocols typically have some unique information
-   that they need to carry.  In some cases that information might be
-   modified along the path and in other cases it is constant.  The in-
-   flight modifications has impacts on what it means to provide security
-   for the encapsulation headers.
+   Encapsulation protocols typically have some unique information that
+   they need to carry.  In some cases that information might be modified
+   along the path and in other cases it is constant.  The in-flight
+   modifications has impacts on what it means to provide security for
+   the encapsulation headers.
    o  NVO3 carries a VNI Identifier edge to edge which is not modified.
       There has been OAM discussions in the WG and it isn't clear
       whether some of the OAM information might be modified in flight.
@@ -151,8 +151,8 @@
       future schemes to scale up to larger network.
    The extensibility needs and constraints might be different when
    considering hardware vs. software implementations of the
-   encapsulation headers.  And NIC hardware might have different
-   constraints than switch hardware.
+   encapsulation headers.  NIC hardware might have different constraints
+   than switch hardware.
 
    As the IETF designs these encapsulations the different WGs solve the
    issues for their own encapsulation.  But there are likely to be
@@ -198,15 +198,14 @@
 4.  Scope
 
    It is important to keep in mind what we are trying to cover and not
-   cover in this document and effort.  It is
+   cover in this document and effort.  This is
    o  A look across the three new encapsulations, while taking lots of
       previous work into account
-   o  Its focus started with the class of encaps that would run over IP/
-      UDP.  That was done to avoid being distracted by the data-plane
-      and control-plane interaction diving into control-plane
-      interaction which is more significant for protocols that are
-      designed to run over MPLS and other protocols that maintain
-      session or path state.
+   o  Focus on the class of encapsulations that would run over IP/UDP.
+      That was done to avoid being distracted by the data-plane and
+      control-plane interaction, which is more significant for protocols
+      that are designed to run over "transports" that maintain session
+      or path state.
    o  We later expanded the scope somewhat to consider how the
       encapsulations would play with MPLS "transport", which is
       important because SFC and BIER seem to target being independent of
@@ -238,10 +237,11 @@
       congestion control in the endpoints might be even less trusted
       than on the Internet at large.
 
-   The goal to implement these encapsulations in hardware and software
-   hence we can't assume that the needs of either implementation
-   approach can trump the needs of the other.  In particular, around
-   extensibility the needs and constraints might be quite different.
+   The goal is to implement these encapsulations in hardware and
+   software hence we can't assume that the needs of either
+   implementation approach can trump the needs of the other.  In
+   particular, around extensibility the needs and constraints might be
+   quite different.

 6.  Terminology
@@ -284,7 +284,7 @@
    information in addition the entropy field and outer IP header, they
    can not use arbitrary parts of the encapsulation header since that
    might result in OAM frames taking a different path.  Likewise if
-   routers look past the encapsulation header the need to be aware of
+   routers look past the encapsulation header they need to be aware of
    the extensibility mechanism(s) in the encapsulation format to be able
    to find the inner headers in the presence of extensions; OAM frames
    might use some extensions e.g. for timestamps.
@@ -326,16 +326,17 @@
 
    The transport delivery mechanism for the encapsulations we discuss in
    this document need some way to indicate which encapsulation header
-   will follow.  Some encapsulations might be identified by a UDP port;
-   others might be identified by an Ethernet type or IP protocol number.
-   Which approach is used is a function of the preceding header the same
-   was as IPv4 being identified by both an Ethernet type and an IP
-   protocol number (for IP-in-IP).  In some cases the header type is
-   implicit in some session (L2TP) or path (MPLS) setup.  But this is
-   largely beyond the control of the encapsulation protocol.  For
-   instance, if there is a requirement to carry the encapsulation after
-   an Ethernet header, then an Ethernet type is needed.  If required to
-   be carried after an IP/UDP header, then a UDP port number is needed.
+   (or other payload) comes next in the packet.  Some encapsulations
+   might be identified by a UDP port; others might be identified by an
+   Ethernet type or IP protocol number.  Which approach is used is a
+   function of the preceding header the same was as IPv4 being
+   identified by both an Ethernet type and an IP protocol number (for
+   IP-in-IP).  In some cases the header type is implicit in some session
+   (L2TP) or path (MPLS) setup.  But this is largely beyond the control
+   of the encapsulation protocol.  For instance, if there is a
+   requirement to carry the encapsulation after an Ethernet header, then
+   an Ethernet type is needed.  If required to be carried after an IP/
+   UDP header, then a UDP port number is needed.
 
    The encapsulation needs to indicate the type of its payload, which is
    in scope for the design of the encapsulation.  We have existing
@@ -344,8 +345,8 @@
    o  Reuse Ethernet types - makes it easy to carry existing L2 and L3
       protocols
    o  Reuse IP protocol numbers - makes it easy to carry e.g., ESP but
-      brings in protocol numbers which might not make sense as an encaps
-      payload.
+      brings in all existing protocol numbers many of which would never
+      be used directly on top of the encapsulation protocol.
    o  Define their own next-protocol number space, which can use fewer
       bits than an Ethernet type and give more flexibility, but at the
       cost of administering that numbering space.
@@ -442,9 +443,10 @@
    This assumes that the bit or next protocol, respectively, would not
    affect entropy/ECMP in the underlay.
 
-   A single marker bit in the encaps header would be useful in order to
-   delineate measurement epochs on the encapsulator and decapsulator and
-   use that to compare counters to determine packet loss.
+   There has been suggestions that one (or more) marker bits in the
+   encaps header would be useful in order to delineate measurement
+   epochs on the encapsulator and decapsulator and use that to compare
+   counters to determine packet loss.
 
    A result of the above is that OAM is likely to evolve and needs some
    degree of extensibility from the encapsulation format; a bit or two
@@ -466,21 +468,89 @@
 
 11.  Security Considerations
 
-   TBD Spell out threat models - might be different for data centers
-   using NVO3 and SFC than a service provider using BIER for multicast
-   packet delivery.
+   Different encapsulation use cases will have different requirements
+   around security.  For instance, when encapsulation is used to build
+   overlay networks for network virtualization, isolation between
+   virtual networks may be paramount.  BIER support of multicast may
+   entail different security requirements than encapsulation for
+   unicast.
+
+   In real deployment, the security of the underlying network may be
+   considered for determining the level of security needed in the
+   encapsulation layer.  However for the purposes of this discussion, we
+   assume that network security is out of scope and that the underlying
+   network does not itself provide adequate or as least uniform security
+   mechanisms for encapsulation.
 
    There are at least three considerations for security:
    o  Anti-spoofing/virtual network isolation
    o  Interaction with packet level security such as IPsec
    o  Privacy (e.g., VNI ID confidentially for NVO3)
 
-   This description assumes the the network does not itself provide
-   adequate security so mechanisms must be implemented in encapsulation
-   layer, and the security mechanisms are end to end between
-   encapsulator and decapsulator (middle boxes do not have keys)
+   This section uses a VNI ID in NVO3 as an example.  A SFC or BIER
+   encapsulation is likely to have fields with similar security and
+   privacy requirements.
+
+11.1.  Encapsulation-specific considerations
+
+   Some of these considerations appear for a new encapsulation, and
+   others are more specific to network virtualization in datacenters.
+   o  New attack vectors:
+      *  DDOS on specific queued/paths by attempting to reproduce the
+         5-tuple hash for targeted connections.
+      *  Entropy in outer 5-tuple may be too little or predictable.
+      *  Leakage of identifying information in the encapsulation header
+         for an encrypted payload.
+      *  Vulnerabilities of using global values in fields like VNI ID.
+   o  Trusted versus untrusted tenants in network virtualization:
+      *  The criticality of virtual network isolation depends on whether
+         tenants are trusted or untrusted.  In the most extreme cases,
+         tenants might not only be untrusted but may be considered
+         hostile.
+      *  For a trusted set of users (e.g. a private cloud) it may be
+         sufficient to have just a virtual network identifier to provide
+         isolation.  Packets inadvertently crossing virtual networks
+         should be dropped similar to a TCP packet with a corrupted port
+         being received on the wrong connection.
+      *  In the presence of untrusted users (e.g. a public cloud) the
+         virtual network identifier must be adequately protected against
+         corruption and verified for integrity.  This case may warrant
+         keyed integrity.
+   o  Different forms of isolation:
+      *  Isolation could be blocking all traffic between tenants (or
+         except as allowed by some firewall)
+      *  Could also be about performance isolation i.e. one tenant can
+         overload the network in a way that affects other tenants
+      *  Physical isolation of traffic for different tenants in network
+         may be required, as well as required restrictions that tenants
+         may have on where their packets may be routed.
+   o  New attack vectors from untrusted tenants:
+      *  Third party VMs with untrusted tenants allows internally borne
+         attacks within data centers
+      *  Hostile VMs inside the system may exist (e.g. public cloud)
+      *  Internally launched DDOS
+      *  Passive snooping for mis-delivered packets
+      *  Mitigate damage and detection in event that a VM is able to
+         circumvent isolation mechanisms
+   o  Tenant-provider relationship:
+      *  Tenant might not trust provider, hypervisors, network
+      *  Provider likely will need to provide SLA or a least a statement
+         on security
+      *  Tenant may implement their own additional layers of security
+      *  Regulation and certification consuderations
+   o  Trend towards tighter security:
+      *  Tenants' data in network increases in volume and value, attacks
+         become more sophisticated
+      *  Large DCs already encrypt everything on disk
+      *  DCs likely to encrypt inter-DC traffic at this point, use TLS
+         to Internet.
+      *  Encryption within DC is becoming more commonplace, becomes
+         ubiquitous when cost is low enough.
+      *  Cost/performance considerations.  Cost of support for strong
+         security has made strong network security in DCs prohibitive.
+      *  Are there lessons from MacSec?
 
-11.1.  Anti-spoofing/virtual network isolation
+11.2.  Anti-spoofing/virtual network isolation
 
    The first requirement is anti-spoofing.  An attacker should not be
    able to illegitimately inject packets into a virtual network.  This
@@ -536,7 +606,7 @@
    standard UDP destination port number for each encapsulation protocol
    would facilitate such blocking.
 
-11.2.  Packet level security
+11.3.  Packet level security
 
    An encapsulated packet may itself be encapsulated in IPsec (e.g.
    ESP).  This should be straightforward and in fact is what would
@@ -567,105 +637,6 @@
    this should be okay as long as semantics are clear and processing is
    deterministic.
 
-11.3.  Raw Notes
-
-   TBD: Can we remove this material?  Need to verify that it is covered
-   in the section above.  Should at least cover something more about
-   threat models and different requirements or assumptions above.
-
-   Tenants assume isolation with different strength:
-   o  For a trusted set of users (e.g. a private cloud) it may be
-      sufficient to have just a virtual network identifier.  Packets
-      inadvertently crossing virtual networks should be dropped, similar
-      to a TCP packet with a corrupted port being received on the wrong
-      connection.
-   o  In the presence of untrusted users (e.g. a public cloud) the
-      virtual network identifier must be adequately protected against
-      corruption and verified for integrity.  These case may warrant
-      keyed integrity.  This might be analogous to L3VPN using VRF
-      support in routers vs. IPsec VPNs.
-
-   Different forms of isolation:
-   o  Isolation could be blocking all traffic between tenants (or except
-      as allowed by some firewall)
-   o  Could also be about performance isolation i.e. one tenant can't
-      overload the network in a way that affects other tenants
-   o  Physical isolation of traffic for different tenants in network may
-      be required, as well as required restrictions that tenants may
-      have on where their packets may be routed.
-
-   Hostile VMs inside the system:
-
-   o  Internally launched DDOS
-   o  Passive snooping for mis-delivered packets
-   o  Mitigate damage and detection in event that a VM is able to
-      circumvent isolation mechanisms
-
-   Tenant might not trust the hypervisors:
-   o  Provider likely will need to provide SLA or a least a statement on
-      security
-   o  Tenant may implement their own additional layers of security
-
-   Other motivating factors:
-   o  Regulation and certification
-
-   Demotivating factors:
-   o  Additional cost to encrypt everything?
-      *  Trend is to encrypt inter-DC traffic at this point, use TLS to
-         Internet.
-      *  Encryption within DC is becoming more commonplace, becomes
-         ubiquitous when cost is low enough.
-   o  Poor ease of use - managing keys, certs, etc
-
-   Encaps-specific considerations:
-   o  Does it create any new attack vectors?
-      *  DDOS on specific queued/paths by attempting to reproduce the
-         5-tuple hash for targeted connections
-      *  Leakage of identifying information in the encapsulation header
-         for an encrypted payload
-      *  Vulnerability of using global values for tenants in things like
-         VNI ID which may be exploited.
-   o  Does it create a larger attack surface e.g.  NVO3 vs. VLANs due to
-      deployment across a larger topology?
-
-   Providing security on behalf of a third party:
-   o  Multi-tenant isolation
-   o  Security services for tenants?  Security SLAs?  E.g. are disks
-      encrypted?  Data in flight?  Perception?
-      *  Large DC operators probably encrypt all disk
-      *  Assume trend will be towards tighter security as tenants' data
-         in network increases in volume and value, attacks become more
-         sophisticated
-
-   Key management - role of orchestration?
-
-   MacSec?  Working on multi-hop.
-   o  MacSec has different deployment considerations than IPsec
-   o  Key management might be different
-
-
-   o  Expectations is that MacSec is included in the box?
-   Question: MacSec on high speed interfaces?  I heard there are still
-   challenges in 100G for example
-
-   Layered security:
-   o  Could have NVE-NVE encryption
-      *  In DC possibly implemented as host-host encryption as an
-         independent layer of NVEs
-      *  Encrypt per policy, tenant contract
-   o  In addition VM to VM if tenant wants it
-
-   Encaps specific question - visibility of tunnels
-   o  Can tenant see the tunnels it is using?
-   o  Can tenantA see the tunnels used by tenantB?  Or service like IDS
-      needing to look at payload for tenantB
-   o  Can the underlay control/subvert a tunnel?
-
-   Cost of encryption
-   o  Encryption might not be more expensive (in hardware) than only
-      doing integrity protection.  Integrity-only protection might be
-      useful in above case of IDS.
-
 
 12.  QoS
 
@@ -777,8 +748,7 @@
    Circuit breakers are not sufficient to make a network with different
    congestion control when the goal is to provide a predictable service
    to different tenants.  The fallback would be to rate limit different
-   traffic.  TBD: Any (more) pointers to rate limiting vs/and circuit
-   breakers to add?
+   traffic.

 14.  Header Protection
@@ -826,7 +796,7 @@
    RSS, and middlebox operations.  As UDP-Lite uses a different protocol
    number than UDP and it is not widely implemented in middleboxes, this
    value is lost.  A possible solution is to incorporate the same
-   partial-checksum concept from UDP-Lite or other header checksum
+   partial-checksum concept as UDP-Lite or other header checksum
    protection into the encapsulation header and continue using UDP as
    the outer protocol.  One potential challenge with this approach is
    the use of NAT or other form of translation on the outer header will
@@ -1161,6 +1131,168 @@
       hardware can explore different combinations of headers in
       parallel, however there is some added cost for this.
 
+18.1.  Considerations for NIC offload
+
+   This section provides guidelines to provide support of common
+   offloads for encapsulation in Network Interface Cards (NICs).
+   Offload mechanisms are techniques that are implemented separately
+   from the normal protocol implementation of a host networking stack
+   and are intended to optimize or speed up protocol processing.
+   Hardware offload is performed within a NIC device on behalf of a
+   host.
+
+   There are three basic offload techniques of interest:
+   o  Receive multi queue
+   o  Checksum offload
+   o  Segmentation offload
+
+18.1.1.  Receive multi-queue
+
+   Contemporary NICs support multiple receive descriptor queues (multi-
+   queue).  Multi-queue enables load balancing of network processing for
+   a NIC across multiple CPUs.  On packet reception, a NIC must select
+   the appropriate queue for host processing.  Receive Side Scaling
+   (RSS) is a common method which uses the flow hash for a packet to
+   index an indirection table where each entry stores a queue number.
+
+   UDP encapsulation, where the source port is used for entropy, should
+   be compatible with multi-queue NICs that support five-tuple hash
+   calculation for UDP/IP packets as input to RSS.  The source port
+   ensures classification of the encapsulated flow even in the case that
+   the outer source and destination addresses are the same for all flows
+   (e.g. all flows are going over a single tunnel).
+
+18.1.2.  Checksum offload
+
+   Many NICs provide capabilities to calculate standard ones complement
+   payload checksum for packets in transmit or receive.  When using
+   encapsulation over UDP there are at least two checksums that may be
+   of interest: the encapsulated packet's transport checksum, and the
+   UDP checksum in the outer header.
+
+18.1.2.1.  Transmit checksum offload
+
+   NICs may provide a protocol agnostic method to offload transmit
+   checksum (NETIF_F_HW_CSUM in Linux parlance) that can be used with
+   UDP encapsulation.  In this method the host provides checksum related
+   parameters in a transmit descriptor for a packet.  These parameters
+   include the starting offset of data to checksum, the length of data
+   to checksum, and the offset in the packet where the computed checksum
+   is to be written.  The host initializes the checksum field to pseudo
+   header checksum.  In the case of encapsulated packet, the checksum
+   for an encapsulated transport layer packet, a TCP packet for
+   instance, can be offloaded by setting the appropriate checksum
+   parameters.
+
+   NICs typically can offload only one transmit checksum per packet, so
+   simultaneously offloading both an inner transport packet's checksum
+   and the outer UDP checksum is likely not possible.  In this case
+   setting UDP checksum to zero (per above discussion) and offloading
+   the inner transport packet checksum might be acceptable.
+
+   There is a proposal in [I-D.herbert-remotecsumoffload] to leverage
+   NIC checksum offload when an encapsulator is co-resident with a host.
+
+18.1.2.2.  Receive checksum offload
+
+   Protocol encapsulation is compatible with NICs that perform a
+   protocol agnostic receive checksum (CHECKSUM_COMPLETE in Linux
+   parlance).  In this technique, a NIC computes a ones complement
+   checksum over all (or some predefined portion) of a packet.  The
+   computed value is provided to the host stack in the packet's receive
+   descriptor.  The host driver can use this checksum to "patch up" and
+   validate any inner packet transport checksum, as well as the outer
+   UDP checksum if it is non-zero.
+
+   Many legacy NICs don't provide checksum-complete but instead provide
+   an indication that a checksum has been verified (CHECKSUM_UNNECESSARY
+   in Linux).  Usually, such validation is only done for simple TCP/IP
+   or UDP/IP packets.  If a NIC indicates that a UDP checksum is valid,
+   the checksum-complete value for the UDP packet is the "not" of the
+   pseudo header checksum.  In this way, checksum-unnecessary can be
+   converted to checksum-complete.  So if the NIC provides checksum-
+   unnecessary for the outer UDP header in an encapsulation, checksum
+   conversion can be done so that the checksum-complete value is derived
+   and can be used by the stack to validate an checksums in the
+   encapsulated packet.
+
+
+
+18.1.3.  Segmentation offload
+
+   Segmentation offload refers to techniques that attempt to reduce CPU
+   utilization on hosts by having the transport layers of the stack
+   operate on large packets.  In transmit segmentation offload, a
+   transport layer creates large packets greater than MTU size (Maximum
+   Transmission Unit).  It is only at much lower point in the stack, or
+   possibly the NIC, that these large packets are broken up into MTU
+   sized packet for transmission on the wire.  Similarly, in receive
+   segmentation offload, small packets are coalesced into large, greater
+   than MTU size packets at a point low in the stack receive path or
+   possibly in a device.  The effect of segmentation offload is that the
+   number of packets that need to be processed in various layers of the
+   stack is reduced, and hence CPU utilization is reduced.
+
+18.1.3.1.  Transmit Segmentation Offload
+
+   Transmit Segmentation Offload (TSO) is a NIC feature where a host
+   provides a large (larger than MTU size) TCP packet to the NIC, which
+   in turn splits the packet into separate segments and transmits each
+   one.  This is useful to reduce CPU load on the host.
+
+   The process of TSO can be generalized as:
+   o  Split the TCP payload into segments which allow packets with size
+      less than or equal to MTU.
+   o  For each created segment:
+      1.  Replicate the TCP header and all preceding headers of the
+          original packet.
+      2.  Set payload length fields in any headers to reflect the length
+          of the segment.
+      3.  Set TCP sequence number to correctly reflect the offset of the
+          TCP data in the stream.
+      4.  Recompute and set any checksums that either cover the payload
+          of the packet or cover header which was changed by setting a
+          payload length.
+
+   Following this general process, TSO can be extended to support TCP
+   encapsulation UDP.  For each segment the Ethernet, outer IP, UDP
+   header, encapsulation header, inner IP header if tunneling, and TCP
+   headers are replicated.  Any packet length header fields need to be
+   set properly (including the length in the outer UDP header), and
+   checksums need to be set correctly (including the outer UDP checksum
+   if being used).
+
+   To facilitate TSO with encapsulation it is recommended that optional
+   fields should not contain values that must be updated on a per
+   segment basis-- for example an encapsulation header should not
+   include checksums, lengths, or sequence numbers that refer to the
+   payload.  If the encapsulation header does not contain such fields
+   then the TSO engine only needs to copy the bits in the encapsulation
+   header when creating each segment and does not need to parse the
+   encapsulation header.
+
+18.1.3.2.  Large Receive Offload
+
+   Large Receive Offload (LRO) is a NIC feature where packets of a TCP
+   connection are reassembled, or coalesced, in the NIC and delivered to
+   the host as one large packet.  This feature can reduce CPU
+   utilization in the host.
+
+   LRO requires significant protocol awareness to be implemented
+   correctly and is difficult to generalize.  Packets in the same flow
+   need to be unambiguously identified.  In the presence of tunnels or
+   network virtualization, this may require more than a five-tuple match
+   (for instance packets for flows in two different virtual networks may
+   have identical five-tuples).  Additionally, a NIC needs to perform
+   validation over packets that are being coalesced, and needs to
+   fabricate a single meaningful header from all the coalesced packets.
+
+   The conservative approach to supporting LRO for encapsulation would
+   be to assign packets to the same flow only if they have identical
+   five-tuple and were encapsulated the same way.  That is the outer IP
+   addresses, the outer UDP ports, encapsulated protocol, encapsulation
+   headers, and inner five tuple are all identical.
+
 
 19.  Middlebox Considerations
 
@@ -1380,6 +1512,11 @@
               Encapsulation", draft-herbert-gue-03 (work in progress),
               March 2015.
 
+   [I-D.herbert-remotecsumoffload]
+              Herbert, T., "Remote checksum offload for encapsulation",
+              draft-herbert-remotecsumoffload-01 (work in progress),
+              November 2014.
+
    [I-D.ietf-mpls-in-udp]
               Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
               "Encapsulating MPLS in UDP", draft-ietf-mpls-in-udp-11