Subject: Re: [nvo3] Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 21 Feb 2020 20:00:57 -0800

Hi Ilango,

Sorry for the delayed response.

On Thu, Dec 12, 2019 at 08:00:04PM +0000, Ganga, Ilango S wrote:
> Hello Benjamin,
> 
> Thanks for your review and comments. Please see below for our responses to some of your comments, enclosed in <Response> </Response>. Let us know if you are satisfied with this resolution.
> We will send responses to your other comments separately. 
> 
> Thanks,
> Ilango Ganga
> Geneve Editor
> 
> From: Benjamin Kaduk via Datatracker <[email-masked]> 
> Sent: Wednesday, December 4, 2019 4:37 PM
> To: The IESG <[email-masked]>
> Cc: [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
> Subject: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)
> 
> Benjamin Kaduk has entered the following ballot position for
> draft-ietf-nvo3-geneve-14: Discuss
> 
> 1. Security and transit devices comments:
> 
> Benjamin> “to what extent the Geneve architecture includes support for middleboxes that inspect (but do not modify!) the Geneve header and inner payload”
> 
> IG> <Response> 1.   Looks like you are interpreting middlebox as a transit device, but middle box has a broader meaning including service functions (appliances).  So we clarify that the transit devices are forwarding elements like switches and routers (see section 1.2 for definition) . We reiterate that only NVEs can generate and terminate Geneve headers and transit devices can only interpret Geneve headers and that this is optional. A transit device not interpreting or not able to interpret options can forward the packets like any other UDP packet. So it does not affect the forwarding of Geneve packets end to end.  An example use case of a transit device (like a switch/router) is it may interpret information in Geneve headers to perform ECMP forwarding. If a transit device is not able interpret it will still forward based on outer UDP header information.
> </Response>

I'm not sure whether I was consciously interpreting "middlebox" as a
transit device specifically, but I'm happy to hear that we're envisioning a
strict separation between "transit device" that functions analogously to a
switch or a router and "middlebox" as service functions or appliances that
are principal entities and participants in the protocol.  The discussion
here (and some clarifications by Martin on the telechat) do a lot to help
reassure me that we are not setting ourselves up for failure.

I will try to make some textual suggestions that would help other readers
avoid the confusion/concerns that I had.

> Benjamin> “to what extent the Geneve architecture is intended to be applicable to scenarios where (end-to-end per-tunnel) underlay confidentiality protection is necessary”
> 
> IG> <Response> 2.   NVE to NVE end to end encryption (underlay confidentiality) is prevalent in multi-tenant data centers use cases (see section 6.1). In these cases a transit device will still forward Geneve like any other UDP/IP packets. Hence does not affect the forwarding operation of a transit device. 
> </Response>

Ok.

> Benjamin> "Interposing advanced middleboxes" and "service interposition" are conceived as possible uses for Geneve metadata in Sections 1 and 2.2 as a consideration for why structured tagging is needed on the data plane and not just the control plane, which to me suggests that such usage is considered a first-class use case for Geneve.
> 
> IG> <Response> 3.   In the “service interposition” use case noted above the reference middle box is a network service function, this is not a transit device. The service would implement a NVE function to generate and terminate Geneve headers. Hence this is consistent with Geneve specifications. If data confidentiality is required by the data center operator this is NVE to NVE encryption as expected by the Geneve/NVO3 architecture. 
> </Response>

I think we should clarify in the text that these NVE functions are
terminating and generating Geneve headers, akin to a "back to back user
agent" in the SIP world (e.g., RFC 7092); even just "interposing advanced
middleboxes that terminate and reencapsulate Geneve traffic" would go a
long way.  In Section 2.2 we could use "terminating/reencapsulating service
interposition".  I thought there was another place in the text that
discussed this general sort of thing but couldn't find it by searching;
I'll try to read through the -15 when it appears to see if I notice
anything then.

> Benjamin> “Section 6.1.1 discusses encryption for traffic traversing untrusted links between geographically separated data centers (though perhaps in this case an encrypted tunnel would be used just for that untrusted transit and leaving the in-datacenter traffic visible to middleboxes), but Section 6.1 discusses cases where the tenant may expect the service provider to provide confidentiality as part of the service.  Would this be above or below the Geneve encapsulation?
> ”
> IG> <Response> 4.   In the scenario described in 6.1.1 (geographically separated data centers) the Geneve packets will be transported over an encrypted service (e.g. a VPN service). 
> <Response 5>.  In the scenario described in 6.1 (multi-tenancy) encryption may be used either at the Geneve payload level or at the IP/UDP level in which case the Geneve header will not be visible to transit devices. 
> </Response>

Okay.  (In either case, if an interposing service is
terminating/reencapsulating Geneve, then the encryption will be
appropriately targetted anyway.)

> Benjamin> "The consideration from Section 6.1 that the provider of the underlay and the provider of the overlay may not be the same could be taken to imply that the overlay provider itself wants (cryptographic) protection from the underlay provider.  I don't have a clear picture of how these considerations interact.  (I also note that, since DTLS is mentioned, DTLS 1.3 is going the way of TLS 1.3 and not defining any authentication-only ciphersuites, so if authentication-only service is desired, DTLS may not be the way of the future, leaving IPsec AH as the leading candidate.)"
> 
> IG> <Response> 6.   We expect that the overlay network provider will provide the cryptographic protection with the entire payload being encrypted. 

Okay.  I don't know whether it makes sense to try to convey this
expectation in the document as well, or leave it unspecified.

> </Response>
> 
> 2. UDP Checksum with IPv6 comment:
> 
> Benjamin> Section 4.3.1
>    2.  If Geneve is used with zero UDP checksum over IPv6 then such
>        tunnel endpoint implementation MUST meet all the requirements
>        specified in section 4 of [RFC6936] and requirements 1 as
>        specified in section 5 of [RFC6936].
> 
> "This seems to implicitly be saying that the other numbered requirements in Section 5 of RFC 6936 can be ignored, which is updating the behavior of a standards-track document.  We need to either be explicit about the update or justify why (the rest of) that applicability statement is not applicable here.  If, as the paragraph following the enumerated list says, the requirements specified in RFC 6936 continue to apply in full, why do we need to call out a MUST-level requirement here?"
> 
> IG> <Response> 7.  Requirement 1 is a consideration for Geneve implementers, which is why it is highlighted. The others are design considerations for the protocol itself, which have already been incorporated or are not applicable to Geneve. It is not implying that the other requirements should be ignored. We have tried to indicate that with the text stating that all requirements continue to apply. This text is a resolution to TSVART early review (David Black). The text is very similar to RFC 8086 (and RFC 7510).  
> </Response>

Thanks for helping me understand the history and precedent here (Martin
also helped with this).  I'd suggest rewording slightly to clarify that
requirement 1 of Section 5 of RFC 6936 is mentioned due to its particular
relevance (as opposed to its exclusive relevance), but I do not consider
this point to be at the DISCUSS-level anymore.

Thanks,

Ben

> </End of Responses>
> 
> -----Original Message-----
> From: Benjamin Kaduk via Datatracker <[email-masked]> 
> Sent: Wednesday, December 4, 2019 4:37 PM
> To: The IESG <[email-masked]>
> Cc: [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
> Subject: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)
> 
> Benjamin Kaduk has entered the following ballot position for
> draft-ietf-nvo3-geneve-14: Discuss
> 
> When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> This first point is a "discuss discuss" for which I'd like to get a sense of what the rest of the IESG feels.  I've read the discussion at https://mailarchive.ietf.org/arch/msg/last-call/ywRKREnxWAlunHR7MSaTM4ScsDs
> but I'm left with a similar sense of uncertainty that Daniel has as to whether the question is fully resolved.  Specifically, "the question"
> that I have in mind is to what extent the Geneve architecture includes support for middleboxes that inspect (but do not modify!) the Geneve header and inner payload, to what extent the Geneve architecture is intended to be applicable to scenarios where (end-to-end per-tunnel) underlay confidentiality protection is necessary, and whether those requirements are both strong enough to be deemed an internal inconsistency of requirements/applicability.  "Interposing advanced middleboxes" and "service interposition" are conceived as possible uses for Geneve metadata in Sections 1 and 2.2 as a consideration for why structured tagging is needed on the data plane and not just the control plane, which to me suggests that such usage is considered a first-class use case for Geneve.  Section 6.1.1 discusses encryption for traffic traversing untrusted links between geographically separated data centers (though perhaps in this case an encrypted tunnel would be used just for that untrusted transit and leaving the in-datacenter traffic visible to middleboxes), but Section 6.1 discusses cases where the tenant may expect the service provider to provide confidentiality as part of the service.  Would this be above or below the Geneve encapsulation?
> Might some customers insist on one or the other?  The consideration from Section 6.1 that the provider of the underlay and the provider of the overlay may not be the same could be taken to imply that the overlay provider itself wants (cryptographic) protection from the underlay provider.  I don't have a clear picture of how these considerations interact.  (I also note that, since DTLS is mentioned, DTLS 1.3 is going the way of TLS 1.3 and not defining any authentication-only ciphersuites, so if authentication-only service is desired, DTLS may not be the way of the future, leaving IPsec AH as the leading candidate.)
> 
> Some other section-by-section discuss-level points follow, mostly self-contained/localized issues.
> 
> Section 3.5.1
> 
>    o  Some options may be defined in such a way that the position in the
>       option list is significant.  Options MUST NOT be changed by
>       transit devices.
> 
>    o  An option SHOULD NOT be dependent upon any other option in the
>       packet, i.e., options can be processed independently of one
>       another.  [...]
> 
> As was already noted, I don't see how these two requirements are self-consistent.
> 
>    size.  A particular option is specified to have either a fixed
>    length, which is constant, or a variable length, which may change
>    over time or for different use cases.  This property is part of the
>    definition of the option and conveyed by the 'Type'.  For fixed
> 
> This text is written as if this specification is going to specify further substructure for the "Type", with respect to certain types that have fixed length and others that may vary.  Otherwise the property would be attached to the option value and not the type value, in my understanding.  With the current way the registry is laid out it seems like we need to explicitly say that the entity allocating the option class value needs to specify the interpretation of the 'type' field when used with that option class.
> 
> Section 4.3.1
> 
>    2.  If Geneve is used with zero UDP checksum over IPv6 then such
>        tunnel endpoint implementation MUST meet all the requirements
>        specified in section 4 of [RFC6936] and requirements 1 as
>        specified in section 5 of [RFC6936].
> 
> This seems to implicitly be saying that the other numbered requirements in Section 5 of RFC 6936 can be ignored, which is updating the behavior of a standards-track document.  We need to either be explicit about the update or justify why (the rest of) that applicability statement is not applicable here.  If, as the paragraph following the enumerated list says, the requirements specified in RFC 6936 continue to apply in full, why do we need to call out a MUST-level requirement here?
> 
>    4.  The Geneve tunnel endpoint that encapsulates the tunnel MAY use
>        different IPv6 source addresses for each Geneve tunnel that uses
>        Zero UDP checksum mode in order to strengthen the decapsulator's
>        check of the IPv6 source address (i.e the same IPv6 source
>        address is not to be used with more than one IPv6 destination
>        address, irrespective of whether that destination address is a
>        unicast or multicast address).  When this is not possible, it is
>        RECOMMENDED to use each source address for as few Geneve tunnels
>        that use zero UDP checksum as is feasible.
> 
> This functionality is not usable without some mechanism to signal from encapsulator to decapsulator that it is in use.
> 
>    The requirement to check the source IPv6 address in addition to the
>    destination IPv6 address, [...]
> 
> I do not see this specified as a requirement, only a MAY-level suggestion.
> 
> Section 4.6
> 
>    o  When performing LSO, a NIC MUST replicate the entire Geneve header
>       and all options, including those unknown to the device, onto each
>       resulting segment.  However, a given option definition may
>       override this rule and specify different behavior in supporting
>       devices.  [...]
> 
> This second sentence makes the MUST in the first no longer a MUST.
> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> Section 2.2.1
> 
>    recipient.  As new functionality becomes sufficiently well defined to
>    add to tunnel endpoints, supporting options can be designed using
>    ordering restrictions and other techniques to ease parsing.
> 
> I'm having trouble parsing the second half of this sentence -- what does "supporting options" mean as a noun?
> 
>    Further, either tunnel endpoints or transit devices MAY use offload
>    capabilities of NICs such as checksum offload to improve the
>    performance of Geneve packet processing.  The presence of a Geneve
>    variable length header SHOULD NOT prevent the tunnel endpoints and
>    transit devices from using such offload capabilities.
> 
> I agree with the directorate reviewer that this implementation guidance is unenforcable as normative keywords.
> 
> Section 3.1, 3.2
> 
> If we're going to give concrete values for the IPv4 protocol/IPv6 NextHeader (17) and destination port (6081), shouldn't we also use the concreve value for Geneve protocol type (0x6558) that corresponds to the inner ethernet frame?
> 
> I'd also suggest some visual distinction that the "Variable Length Options" do in fact have variable length, perhaps using the '~'
> character in vertical lines.
> Similarly, the original ethernet payload need not be 4-byte-aligned and the figure could make that more prominent.
> 
> It's a little awkward to expand FCS on second usage, not first usage.
> 
> Section 3.4
> 
>       The critical bit allows hardware implementations the flexibility
>       to handle options processing in the hardware fastpath or in the
>       exception (slow) path without the need to process all the options.
>       For example, a critical option such as secure hash to provide
>       Geneve header integrity check must be processed by tunnel
>       endpoints and typically processed in the hardware fastpath.
> 
> I think I'm failing to make a connection between some of these steps.
> How does having a critical bit let a header integrity check happen in the hardware fastpath while deferring other option processing to software?
> 
>    Transit devices MUST maintain consistent forwarding behavior
>    irrespective of the value of 'Opt Len', including ECMP link
>    selection.  These devices SHOULD be able to forward packets
>    containing options without resorting to a slow path.
> 
> There seem to be two broad aspects in play here.  First, requiring insensitivity to "Opt Len" might be because the value would change as a packet traverses the network.  I think this is forbidden by virtue of transit devices not being allowed to add/delete options, but please confirm.  Second, this affects the ability of transit devices to look past the geneve header to the inner ethernet header and payload.  Given the substantial discussion we've had in the broader IETF about IPv6 extension headers and the inability of hardware to examine such variable-length chains to get to the actual upper layer protocol (with the result that extension headers are largely unusuable on substantial portions of the internet), it seems like we might conclude from this statement that either we expect transit devices to not inspect the upper-layer content or there's a significant chance that this requirement will be ignored (possibly just by capping the 'Opt Len'
> value that is supported), or both.  What makes this setup different from
> IPv6 EH such that we expect hardware compliance and a usable deployment?
> This is particularly poigniant given that we claim this to be a requirement on transit devices but allow (in Section 4.5) for endpoints to use profiles that have a restricted maximum length for the options.
> If such profiles are common, the incentive for transit devices to slip and use the lower maximum length increases.
> 
> Section 3.5
> 
>       The high order bit of the option type indicates that this is a
>       critical option.  If the receiving tunnel endpoint does not
>       recognize this option and this bit is set then the packet MUST be
>       dropped.  If the 'C' bit (critical bit) is set in any option then
>       the 'C' bit in the Geneve base header MUST also be set.  Transit
>       devices MUST NOT drop packets on the basis of this bit.  The
> 
> nit: since we mention the Geneve header, one might claim that "this bit"
> in "MUST NOT drop packets on the basis of this bit" is ambiguous (but since we said this before for the Geneve header one, I assume we're talking about the one in the Type field now).
> 
> Section 4.4.1
> 
>    It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191],
>    [RFC8201]) be used by setting the DF bit in the IP header when Geneve
>    packets are transmitted over IPv4 (this is the default with IPv6).
> 
> Is it the default or the only specified behavior for IPv6?
> 
> Section 4.4.3
> 
>    outside of the scope of this document.  When physical multicast is in
>    use, the 'C' bit in the Geneve header may be used with groups of
>    devices with heterogeneous capabilities as each device can interpret
>    only the options that are significant to it if they are not critical.
> 
> Please double-check this sentence, particularly the "may be used".  If the intent is, as written, to note that the packets with the 'C' bit set might take paths with heterogenous paths, I suggest being more explicit about the consequences that the traffic might only be delivered to some but not all endpoints.
> 
> Section 6
> 
>    untrusted boundaries.  In addition, tunnel endpoints should only be
>    operated in environments controlled by the service provider, such as
>    the hypervisor itself rather than within a customer VM.
> 
> Can you say a bit more about how this "should only be operated in environments controlled by the service provider" meshes with the note in Section 4.1 that "[i]t is intended for use in public or private data center environments" (specifically the "public data center" portion) and the note in Section 6.1 that the provider of the overlay may not be the same as the provider of the underlay?
> 
> Section 6.1.1
> 
>    traversing public networks.  Any Geneve overlay data leaving the data
>    center network beyond the operator's security domain SHOULD be
>    secured by encryption mechanisms such as IPsec or other VPN
>    mechanisms to protect the communications between the NVEs when they
>    are geographically separated over untrusted network links.
> 
> Since we use "mechanisms" in both the IPsec clause and the "other VPN"
> clause, the "encryption" does not automatically bind to both clauses from a grammatical perspective.  Given that "VPN" is currently in use for both encrypted and non-encrypted schemes (much to my chagrin), please clarify that the other VPN mechanisms also need to provide cryptographic confidentiality protection.  (Replacing "VPN mechanisms"
> with "VPN technologies" would probably suffice.)
> 
> Section 6.2
> 
>    network.  To prevent such attacks, an NVE MUST NOT propagate Geneve
>    packets beyond the NVE to tenant systems and SHOULD employ packet
> 
> We also care about not propagating Geneve packets from the tenant systems past the NVE, right?
> 
>    filtering mechanisms so as not to forward unauthorized traffic
>    between TSs in different tenant networks.
> 
> What does "TS" stand for, here?
> 
> Section 10.2
> 
> RFCs 1191, 2460 (er, 8200), 6040, and 8201 should be listed as normative references.
> 
>    [ETYPES]   The IEEE Registration Authority, "IEEE 802 Numbers", 2013,
>               <http://www.iana.org/assignments/ieee-802-numbers/ieee-
>               802-numbers.xml>.
> 
> Hmm, firefox claims the content of this resource is invalid XML, sigh.
> 
>

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 21 Feb 2020 21:05:38 -0800

On Wed, Jan 15, 2020 at 04:23:14AM +0000, Ganga, Ilango S wrote:
> Hello Benjamin,
> 
> 
> 
> Thanks for your detailed review. Here are the responses to rest of your comments, enclosed in <Response> </Response> inline below. Please let us know if you are satisfied with this resolution.

Lack of response indicates satisfactory resolution; comments inline.

> 
> 
> Regards,
> 
> Ilango Ganga
> 
> Geneve Editor
> 
> 
> 
> For reference our previous responses to your comments are posted at:  https://mailarchive.ietf.org/arch/msg/nvo3/G37hH5brjYzYPQLHAfUr54_-Fwg
> 
> Rest of our responses are noted inline below enclosed within <Response> </Response>
> 
> 
> 
> Some other section-by-section discuss-level points follow, mostly
> 
> self-contained/localized issues.
> 
> 
> 
> Section 3.5.1
> 
> 
> 
> IG>> <Response>
> These are different requirements. An option position being significant does not mean the options are dependent on one another. For example, in some use cases, certain option may be placed at the beginning for the ease of parsing in hardware.

Perhaps I am assigning some subtlety of meaning to "significant" vs.
"deliberate" or "convenient" -- to me, if the position of an option is
"significant" that means there are protocol-level semantics associated to
that positioning, so that a receiver should alter their protocol operation
if option A is listed before vs. after option B.  A case where options are
placed earlier in the list for ease of parsing in hardware is a matter of
convenience and may well be worth documenting in the RFC, but that's
operational guidance as opposed to an indication that ordering is
significant at the protocol level.

> IG>> <Response> This document provides the generic rules for defining the options. The option type should be defined by the entity that owns the option class.  We will add a sentence at the end of this paragraph as follows to provide clarity:
> 
> 
> 
> “Actual definition of the option type is beyond the scope of this document.  The option type and its interpretation should be defined by the entity that owns the option class.”

Thanks.

> IG>> <Response>
> We already responded to this comment (see link below)
> 
> https://mailarchive.ietf.org/arch/msg/nvo3/G37hH5brjYzYPQLHAfUr54_-Fwg
> 
> </Response>
> 
> 
> 
> 
> 
> IG>> <Response> Enablement of this functionality between the tunnel end points can be signaled by the NVE control plane. The definition of control plane is beyond the scope of this document. This text is based on resolution to TSVART early review (David Black). Similar text is in RFC 8086.

I think we need to explicitly state that the recipient can only apply these
more-stringent checks if it has out-of-band knowledge that the sender is
applying the indicated behavior.

> IG>> <Response> We will change the sentence as follows:
> 
> “The use of source IPv6 address in addition to the destination IPv6 address, plus …”
> 
> </Response>
> 
> 
> 
> 
> 
> Section 4.6
> 
> 
> 
> IG>> <Response> This is not a conflicting requirement. The NIC must follow the rule unless an option allows an exception. This what is captured in this paragraph. Maybe we could rephrase this paragraph as below to provide better clarity:
> 
> 
> 
> “When performing LSO, a NIC MUST replicate the entire Geneve header
> 
>       resulting segment unless an option allows an exception.”

That would help, thanks.

> IG>> <Response> We will remove “supporting” from this sentence as follows:
> 
> 
> 
> “As new functionality becomes sufficiently well-defined to
> 
>    add to tunnel endpoints, options can be designed using
> 
>    ordering restrictions and other techniques to ease parsing.”

I can parse that now, which is good.
It does seem to lead us further down the path of "option ordering has
significance at a protocol-semantics level", per the discussion above,
though.

> IG>> <Response> Even though it is an implementation consideration, it provides guidance to implementors to ensure the correct use of Geneve variable length options.
> 
> We will lower the case to “should not” in this sentence as follows:
> 
> 
> 
>    “Further, either tunnel endpoints or transit devices MAY use offload
> 
>    variable length header should not prevent the tunnel endpoints and
> 
>    transit devices from using such offload capabilities.”
> 
> </Response>
> 
> 
> 
> Section 3.1, 3.2
> 
> 
> 
> IG>> <Response> The Geneve Protocol type follows the ETYPES convention (See section 3.4). So it is appropriate to use “Protocol Type” in this illustration and not use a specific value. Ethernet is just shown as an example payload; other protocols can be carried over Geneve.
> 
> 
> 
> We will show visual distinction to illustrate “Variable length options”
> 
> We will add a note to say that Ethernet payload need not be 4-byte aligned.
> 
> </Response>
> 
> 
> 
> 
> 
> IG>> <Response> We will expand FCS on its first usage.
> 
> </Response>
> 
> 
> 
> 
> 
> Section 3.4
> 
> 
> 
> IG>> <Response>
> The Critical bit applies to the entire tunnel end point irrespective of a component (HW or SW) of implementation. So we will remove this paragraph from section 3.4.
> 
> </Response>
> 
> 
> 
> 
> 
> There seem to be two broad aspects in play here.  First, requiring insensitivity to "Opt Len" might be because the value would change as a packet traverses the network.  I think this is forbidden by virtue of transit devices not being allowed to add/delete options, but please confirm.  Second, this affects the ability of transit devices to look past the geneve header to the inner ethernet header and payload.  Given the substantial discussion we've had in the broader IETF about IPv6 extension headers and the inability of hardware to examine such variable-length chains to get to the actual upper layer protocol (with the result that extension headers are largely unusuable on substantial portions of the internet), it seems like we might conclude from this statement that either we expect transit devices to not inspect the upper-layer content or there's a significant chance that this requirement will be ignored (possibly just by capping the 'Opt Len' value that is supported), or both.  What makes this setup different from IPv6 EH such that we expect hardware compliance and a usable deployment? This is particularly poigniant given that we claim this to be a requirement on transit devices but allow (in Section 4.5) for endpoints to use profiles that have a restricted maximum length for the options. If such profiles are common, the incentive for transit devices to slip and use the lower maximum length increases.
> 
> 
> 
> IG>> <Response> The key requirement is that the transit devices must maintain consistent forwarding behavior irrespective of the value of the option length. The value of option length is set by the end point and it does not change while traversing the network. Transit devices can use the “Opt Length” to simply skip over the options (without traversing individual options) to reach the inner payload if such devices choose to do so. For example maintaining consistent forwarding behavior for ECMP is possible by using the source UDP port number for entropy”

Okay.  (I think I realized after I sent this that part of the problem with
IPv6 EH is that there is in theory a chain of them that must be processed
in sequence, and hardware design constraints effectively mean that you can
design to fast-path a fixed number of such options but no more; the Opt Len
field here is a single length of *all* the options, so would require a
fixed amount of hardware in order to do the processing that skips options.)

> nit: since we mention the Geneve header, one might claim that "this bit" in "MUST NOT drop packets on the basis of this bit" is ambiguous (but since we said this before for the Geneve header one, I assume we're talking about the one in the Type field now).
> 
> 
> 
> IG>> <Response> In order to remove the ambiguity, we will rephrase the third sentence as follows:
> 
> 
> 
> “The high order bit of the option type indicates that this is a critical option.  If the receiving tunnel endpoint does not recognize this option and this bit is set then the packet MUST be dropped. If this bit is set in any option then the 'C' bit in the Geneve base header MUST also be set. Transit devices MUST NOT drop packets on the basis of this bit.”
> 
> </Response>
> 
> 
> 
> 
> 
> Section 4.4.1
> 
> 
> 
> IG>> <Response> We will rephrase the sentence as follows:
> 
> “It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191],
> 
>    [RFC8201]) be used to prevent or minimize fragmentation.” The use of path MTU Discovery….
> 
> </Response>
> 
> 
> 
> 
> 
> Section 4.4.3
> 
> 
> 
> IG>> <Response> We will update the sentence as follows to provide better clarity:
> 
> “When physical multicast is in use, devices with heterogeneous capabilities in the same group may use the ‘C’ bit to ignore options not supported by that device provided they are not critical. Requirements outlined in Section 3.4 apply for critical options.”

That helps, though I would counter-propose:

When physical multicast is in use, devices with heterogeneous capabilities
may be present in the same group.  Some options may only be interpretable
by a subset of the devices in the group, and other devices can safely
ignore such options unless the 'C' bit is set to mark the unknown option as
critical.  Requirements outlined in Section 3.4 apply for critical options.

> IG>> <Response> We will remove the following sentence from this paragraph:
> 
> “In addition, tunnel endpoints should only be operated in environments controlled by the service provider, such as the hypervisor itself rather than within a customer VM”.
> 
> </Response>
> 
> 
> 
> 
> 
> Section 6.1.1
> 
> 
> 
> Since we use "mechanisms" in both the IPsec clause and the "other VPN" clause, the "encryption" does not automatically bind to both clauses from a grammatical perspective. Given that "VPN" is currently in use for both encrypted and non-encrypted schemes (much to my chagrin), please clarify that the other VPN mechanisms also need to provide cryptographic confidentiality protection.  (Replacing "VPN mechanisms" with "VPN technologies" would probably suffice.)
> 
> 
> 
> IG>> <Response> Agreed. We will replace “VPN mechanisms” to “VPN technologies”
> 
> </Response>
> 
> 
> 
> 
> 
> Section 6.2
> 
> 
> 
> IG>> <Response> We will add the following sentence at the end of the paragraph for symmetry.
> 
> “An NVE MUST NOT interpret Geneve packets from tenant systems other than as frames to be encapsulated”.
> 
> </Response>
> 
> 
> 
> 
> 
> IG>>  <Response> We will expand TSs to “tenant systems”.
> 
> </Response>
> 
> 
> 
> 
> 
> Section 10.2
> 
> 
> 
> IG>> <Response> Agree. We will change these references to normative.
> 
> </Response>
> 
> 
> 
> 
> 
>               <http://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xml>.
> 
> 
> 
> IG>> <Response> We checked. This URL works fine.

It works for me now, too (though on a different computer and using a
different browser).  Maybe IANA was having trouble the last time I looked.

Thanks for all the updates; I'll watch for the -15.

-Ben

> <End of Responses>
> From: Ganga, Ilango S <[email-masked]>
> Sent: Thursday, December 12, 2019 12:00 PM
> To: Benjamin Kaduk <[email-masked]>; The IESG <[email-masked]>
> Cc: [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]; [email-masked]
> Subject: RE: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)
> 
> 
> 
> Hello Benjamin,
> 
> 
> 
> From: Benjamin Kaduk via Datatracker <[email-masked]<mailto:[email-masked]>>
> 
> To: The IESG <[email-masked]<mailto:[email-masked]>>
> 
> Cc: [email-masked]<mailto:[email-masked]>; Matthew Bocci <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
> 
> From: Benjamin Kaduk via Datatracker <[email-masked]<mailto:[email-masked]>>
> 
> To: The IESG <[email-masked]<mailto:[email-masked]>>
> 
> Cc: [email-masked]<mailto:[email-masked]>; Matthew Bocci <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
>

---

From: "Ganga, Ilango S" <[email-masked]>
Date: Tue, 25 Feb 2020 23:54:21 +0000

Hi Ben,

Thanks for your responses and suggestions.  Please see below for our responses and proposed text based on your suggestions.

Regards,

Ilango

-----Original Message-----
From: Benjamin Kaduk <[email-masked]>
Sent: Friday, February 21, 2020 8:01 PM
To: Ganga, Ilango S <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)

Hi Ilango,

> Cc: [email-masked]<mailto:[email-masked]>; Matthew Bocci

> <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>;

> [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

> Subject: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with

> DISCUSS and COMMENT)

> </Response>

I'm not sure whether I was consciously interpreting "middlebox" as a transit device specifically, but I'm happy to hear that we're envisioning a strict separation between "transit device" that functions analogously to a switch or a router and "middlebox" as service functions or appliances that are principal entities and participants in the protocol.  The discussion here (and some clarifications by Martin on the telechat) do a lot to help reassure me that we are not setting ourselves up for failure.

I will try to make some textual suggestions that would help other readers avoid the confusion/concerns that I had.

IG>> Ok, we will incorporate your suggestions below in -15.

Ok.

> </Response>

I think we should clarify in the text that these NVE functions are terminating and generating Geneve headers, akin to a "back to back user agent" in the SIP world (e.g., RFC 7092); even just "interposing advanced middleboxes that terminate and reencapsulate Geneve traffic" would go a long way.  In Section 2.2 we could use "terminating/reencapsulating service interposition".  I thought there was another place in the text that discussed this general sort of thing but couldn't find it by searching; I'll try to read through the -15 when it appears to see if I notice anything then.

IG>> <Response> Agreed. We will update the text in Section 1 and Section 2.2 as suggested:

Revised text in Section 1:

“There are nearly endless uses for this metadata,
   ranging from storing input ports for simple security policies to
   service based context for advanced middlebox applications that terminate and re-encapsulate Geneve traffic.”

Revised text in Section 2:

“It should be noted that while options can be used to support non-
   wirespeed control packets, they are equally important on data packets
   as well to segregate and direct forwarding (for instance, the
   examples given before of input port based security policies and
   terminating/re-encapsulating service interposition both require tags to be placed on data packets).”

Section 1.2: To provide better clarity, we will add definition for middlebox in the context of Geneve to section 1.2 Terminology.

“Middlebox. The term middlebox in the context of this document refers to network
service functions or appliances for service interposition that would typically
implement NVE function, which terminate or re-encapsulate Geneve traffic.”
</Response>

> </Response>

Okay.  (In either case, if an interposing service is terminating/reencapsulating Geneve, then the encryption will be appropriately targetted anyway.)

IG>> Yes, that is correct.

> IG> <Response> 6.   We expect that the overlay network provider will provide the cryptographic protection with the entire payload being encrypted.

Okay.  I don't know whether it makes sense to try to convey this expectation in the document as well, or leave it unspecified.

IG>> <Response>We can add the following sentence (in quotes) at the end of the first paragraph in Section 6.1 to convey this expectation.

   A tenant may expect the
   overlay service provider to provide data confidentiality as part of
   the service or a tenant may bring its own data confidentiality
   mechanisms like IPsec or TLS to protect the data end to end between
   its tenant systems. “The overlay provider is expected to provide
   cryptographic protection in cases where the underlay provider is not the
   same as the overlay provider to ensure the payload is not exposed to the underlay.”
</Response>

> </Response>

Thanks for helping me understand the history and precedent here (Martin also helped with this).  I'd suggest rewording slightly to clarify that requirement 1 of Section 5 of RFC 6936 is mentioned due to its particular relevance (as opposed to its exclusive relevance), but I do not consider this point to be at the DISCUSS-level anymore.

IG> <Response> We will revise the text for requirement 1 as suggested for better clarity.

“If Geneve is used with zero UDP checksum over IPv6 then such

       specified in section 4 of [RFC6936] and requirement 1

       specified in section 5 of [RFC6936] that is relevant to Geneve”.

</Response>

Thanks,

Ben

> Cc: [email-masked]<mailto:[email-masked]>; Matthew Bocci

> <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>;

> [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

> Subject: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with

> DISCUSS and COMMENT)

> When responding, please keep the subject line intact and reply to all

> email addresses included in the To and CC lines. (Feel free to cut

> this introductory paragraph, however.)

> Please refer to

> https://www.ietf.org/iesg/statement/discuss-criteria.html

> This first point is a "discuss discuss" for which I'd like to get a

> sense of what the rest of the IESG feels.  I've read the discussion at

> https://mailarchive.ietf.org/arch/msg/last-call/ywRKREnxWAlunHR7MSaTM4

> ScsDs but I'm left with a similar sense of uncertainty that Daniel has

> as to whether the question is fully resolved.  Specifically, "the question"

> Might some customers insist on one or the other?  The consideration

> from Section 6.1 that the provider of the underlay and the provider of

> the overlay may not be the same could be taken to imply that the

> overlay provider itself wants (cryptographic) protection from the

> underlay provider.  I don't have a clear picture of how these

> considerations interact.  (I also note that, since DTLS is mentioned,

> DTLS 1.3 is going the way of TLS 1.3 and not defining any

> authentication-only ciphersuites, so if authentication-only service is

> desired, DTLS may not be the way of the future, leaving IPsec AH as

> the leading candidate.)

> value that is supported), or both.  What makes this setup different

> from

---

From: "Ganga, Ilango S" <[email-masked]>
Date: Wed, 26 Feb 2020 07:25:29 +0000

Hi Ben,

Thanks for your review and suggestions. Please see our responses inline and proposed text based on your suggestions. We will be incorporating the revised text in -15.

Regards,

Ilango

-----Original Message-----
From: Benjamin Kaduk <[email-masked]>
Sent: Friday, February 21, 2020 9:06 PM
To: Ganga, Ilango S <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]; [email-masked]; Martin Vigoureux <[email-masked]>
Subject: Re: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14: (with DISCUSS and COMMENT)

> For reference our previous responses to your comments are posted at:

> Rest of our responses are noted inline below enclosed within

> <Response> </Response>

>    o  Some options may be defined in such a way that the position in

> the

"deliberate" or "convenient" -- to me, if the position of an option is "significant" that means there are protocol-level semantics associated to that positioning, so that a receiver should alter their protocol operation if option A is listed before vs. after option B.  A case where options are placed earlier in the list for ease of parsing in hardware is a matter of convenience and may well be worth documenting in the RFC, but that's operational guidance as opposed to an indication that ordering is significant at the protocol level.

IG>> <Response> There is a more general principle that transit devices should not modify options, so we will simplify this requirement  to remove any ambiguity. You also had a similar comment on 2.2.1 below on options ordering.  There is related text in 4.5.1 that describes the constraints on options ordering.  So we will revise the text in sections 3.5.1, 2.2.1 and provide reference to 4.5.1 to make it consistent. See below:

2.2.1 OLD:
   Tunnel endpoints must be able to parse the variable header, including
   any options, and take action.  Since these devices are actively
   participating in the protocol, they are the most affected by Geneve.

   However, as tunnel endpoints are the ultimate consumers of the data,
   transmitters can tailor their output to the capabilities of the
   recipient.  As new functionality becomes sufficiently well defined to
   add to tunnel endpoints, supporting options can be designed using
   ordering restrictions and other techniques to ease parsing.

2.2.1 NEW (revised text, after removing last sentence)

   Tunnel endpoints must be able to parse the variable header, including

   any options, and take action.  Since these devices are actively

   participating in the protocol, they are the most affected by Geneve.

   However, as tunnel endpoints are the ultimate consumers of the data,

   transmitters can tailor their output to the capabilities of the

   recipient.

3.5.1 OLD

In tunnel endpoints, the generation and interpretation of options is

   determined by the control plane, which is out of the scope of this

   document.  However, to ensure interoperability between heterogeneous

   devices some requirements are imposed on options and the devices that

   process them:

   o  Receiving tunnel endpoints MUST drop packets containing unknown

      options with the 'C' bit set in the option type.  Conversely,

      transit devices MUST NOT drop packets as a result of encountering

      unknown options, including those with the 'C' bit set.

      transit devices.

      another.  Architecturally, options are intended to be self-

      descriptive and independent.  This enables parallelism in option

      processing and reduces implementation complexity.

   When designing a Geneve option, it is important to consider how the

   option will evolve in the future.  Once an option is defined it is

   reasonable to expect that implementations may come to depend on a

   specific behavior.  As a result, the scope of any future changes must

   be carefully described upfront.

3.5.1 NEW (revised text in quotes)

In tunnel endpoints, the generation and interpretation of options is

   determined by the control plane, which is out of the scope of this

   document.  However, to ensure interoperability between heterogeneous

   devices some requirements are imposed on options and the devices that

   process them:

   o  Receiving tunnel endpoints MUST drop packets containing unknown

      options with the 'C' bit set in the option type.  Conversely,

      transit devices MUST NOT drop packets as a result of encountering

      unknown options, including those with the 'C' bit set.

   o  “The contents of the options and their ordering MUST NOT be modified by

      transit devices”.

   When designing a Geneve option, it is important to consider how the

   option will evolve in the future.  Once an option is defined it is

   reasonable to expect that implementations may come to depend on a

   specific behavior.  As a result, the scope of any future changes must

   be carefully described upfront.

   “Architecturally, options are intended to be self-descriptive and independent.

   This enables parallelism in option processing and reduces implementation complexity.

   However, the control plane may impose certain ordering restrictions as described in Section 4.5.1”

4.5.1 OLD

While Geneve options are more flexible, a control plane may restrict

   the number of option TLVs as well as the order and size of the TLVs,

   between tunnel endpoints, to make it simpler for a data plane

   implementation in software or hardware to handle

   [I-D.ietf-nvo3-encap]. For example, there may be some critical

   information such as a secure hash that must be processed in a certain

   order to provide lowest latency.

4.5.1 NEW (revised text in last sentence)

While Geneve options are more flexible, a control plane may restrict

   the number of option TLVs as well as the order and size of the TLVs,

   between tunnel endpoints, to make it simpler for a data plane

   implementation in software or hardware to handle

   [I-D.ietf-nvo3-encap]. For example, there may be some critical

   information such as a secure hash that must be processed in a certain

   order to provide lowest latency or there may be other scenarios where

   the options must be processed in a certain order due to protocol semantics.

</Response>

Thanks.

>        different IPv6 source addresses for each Geneve tunnel that

> uses

>        Zero UDP checksum mode in order to strengthen the

> decapsulator's

>        unicast or multicast address).  When this is not possible, it

> is

>        RECOMMENDED to use each source address for as few Geneve

> tunnels

> IG>> <Response> Enablement of this functionality between the tunnel end points can be signaled by the NVE control plane. The definition of control plane is beyond the scope of this document. This t…

I think we need to explicitly state that the recipient can only apply these more-stringent checks if it has out-of-band knowledge that the sender is applying the indicated behavior.

IG> <Response> We will add a note to the end of the paragraph of this requirement 4:

“Note that (for requirements 3 and 4) the receiving tunnel endpoint can apply these checks only if it has out-of-band knowledge that the encapsulating tunnel endpoint is applying the indicated behavior. One possibility to obtain this out-of-band knowledge is through signaling by the NVE control plane. The definition of NVE control plane is beyond the scope of this document.”
</Response>

>    o  When performing LSO, a NIC MUST replicate the entire Geneve

> header

>       and all options, including those unknown to the device, onto

> each

>       and all options, including those unknown to the device, onto

> each

That would help, thanks.

>    recipient.  As new functionality becomes sufficiently well defined

> to

It does seem to lead us further down the path of "option ordering has significance at a protocol-semantics level", per the discussion above, though.

IG>> Please see resolution to comment on section 3.5.1 above.

>    “Further, either tunnel endpoints or transit devices MAY use

> offload

IPv6 EH is that there is in theory a chain of them that must be processed in sequence, and hardware design constraints effectively mean that you can design to fast-path a fixed number of such options but no more; the Opt Len field here is a single length of *all* the options, so would require a fixed amount of hardware in order to do the processing that skips options.)

>       recognize this option and this bit is set then the packet MUST

> be

>       dropped.  If the 'C' bit (critical bit) is set in any option

> then

>    [RFC8201]) be used by setting the DF bit in the IP header when

> Geneve

When physical multicast is in use, devices with heterogeneous capabilities may be present in the same group.  Some options may only be interpretable by a subset of the devices in the group, and other devices can safely ignore such options unless the 'C' bit is set to mark the unknown option as critical.  Requirements outlined in Section 3.4 apply for critical options.

IG>> <Response> Yes, this looks good, we accept your suggested text and will update Section 4.4.3 accordingly. </Response>

>    operated in environments controlled by the service provider, such

> as

>    traversing public networks.  Any Geneve overlay data leaving the

> data

> clause, the "encryption" does not automatically bind to both clauses

> from a grammatical perspective. Given that "VPN" is currently in use

> for both encrypted and non-encrypted schemes (much to my chagrin),

> please clarify that the other VPN mechanisms also need to provide

> cryptographic confidentiality protection.  (Replacing "VPN mechanisms"

> IG>> <Response> We checked. This URL works fine.

It works for me now, too (though on a different computer and using a different browser).  Maybe IANA was having trouble the last time I looked.

-Ben

> From: Ganga, Ilango S <[email-masked]<mailto:[email-masked]>>

> To: Benjamin Kaduk <[email-masked]<mailto:[email-masked]>>; The IESG <[email-masked]<mailto:[email-masked]>>

> <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

> Subject: RE: Benjamin Kaduk's Discuss on draft-ietf-nvo3-geneve-14:

> (with DISCUSS and COMMENT)

> From: Benjamin Kaduk via Datatracker

> <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>

> To: The IESG <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>

> Cc:

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>

> >; Matthew Bocci

> <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>

> From: Benjamin Kaduk via Datatracker

> <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>

> To: The IESG <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>

> Cc:

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>

> >; Matthew Bocci

> <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>;

> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>

> ScsDs

>    o  Some options may be defined in such a way that the position in

> the

>        different IPv6 source addresses for each Geneve tunnel that

> uses

>        Zero UDP checksum mode in order to strengthen the

> decapsulator's

>        unicast or multicast address).  When this is not possible, it

> is

>        RECOMMENDED to use each source address for as few Geneve

> tunnels

>    o  When performing LSO, a NIC MUST replicate the entire Geneve

> header

>       and all options, including those unknown to the device, onto

> each

>    recipient.  As new functionality becomes sufficiently well defined

> to

>       recognize this option and this bit is set then the packet MUST

> be

>       dropped.  If the 'C' bit (critical bit) is set in any option

> then

>    [RFC8201]) be used by setting the DF bit in the IP header when

> Geneve

>    outside of the scope of this document.  When physical multicast is

> in

>    devices with heterogeneous capabilities as each device can

> interpret

>    operated in environments controlled by the service provider, such

> as

>    traversing public networks.  Any Geneve overlay data leaving the

> data