Subject: [nvo3] GUE extensibility (was Re: Consensus call on encap proposals)

From: Tom Herbert <[email-masked]>
Date: Mon, 25 Jul 2016 07:49:15 -0700

> I object to GUE due to its inability to have a significant number of extensions in a regular and interoperable way. The base flags structure is limited (note 7 of 16 flags have already been used before the document has been published) and requires specific knowledge in the dataplane of all prior defined extensions to even parse a given field. The draft does have private data but since this is unstructured and untyped, it will be very difficult to make it interoperable between different users. In the past, ways around this have been proposed such as a flag indicating that an extended set of flags are present or overlaying the private data with some structure (such as TLVs). However, having multiple extension mechanisms in a single protocol seems overly complex and likely to result in incomplete implementations.
>
Jesse,

As described in the GUE draft an optional flag extension field can be
added to provide another 32 bits of flags. That process could then be
repeated in the flag extension header field to give another 32 bits of
flags an so on. We have been running GUE for over three years now and
beyond the basics extensions (security, checksum, RCO, fragmentation),
I don't have reason believe we'll see new extensions in great
numbers-- at most one or two added a year.

It is true that GUE does not have an open ended extensibility format
like Geneve. As you point out, it could be added, but I don't
currently see the need for that. Can you maybe shed some light on the
rationale and real use of TLVs in Geneve? How many TLVs have been
defined? What kinds of TLVs are being defined? How many TLVs are
present in individual packets? Are individual vendors creating their
own TLVs, and if so how are these interoperable?

Thanks,
Tom

---

From: Jesse Gross <[email-masked]>
Date: Fri, 29 Jul 2016 20:52:29 +0000

> On Jul 25, 2016, at 7:49 AM, Tom Herbert <[email-masked]> wrote:
> 

In terms of Geneve TLVs, there are a number that I know of that are being used or have been proposed:
 * The INT telemetry framework that I mentioned in Berlin to collect data from the underlay (http://p4.org/wp-content/uploads/fixed/INT/INT-current-spec.pdf)
 * OVN, open source network virtualization software uses Geneve to carry logical port information (https://github.com/openvswitch/ovs/blob/v2.5.0/ovn/ovn-architecture.7.xml#L960)
 * Cilium can use Geneve to carry policy labels between containers (https://docs.google.com/presentation/d/16s41VrUuHbaHUwWCkeSBa7DvY2O0PwVcOe-sO8TQ3o0/edit?usp=sharing)
 * In the remote checksum offload draft, you mentioned that the checksum offload field could be carried in a Geneve TLV (https://tools.ietf.org/html/draft-herbert-remotecsumoffload-00)
 * I guess it’s probably not much of a secret that VMware is planning on using Geneve as part of NSX (http://download3.vmware.com/vmworld/2014/downloads/session-pdfs/NET1674_Final_US.pdf)

These have all more or less been defined by different organizations and different use cases. By having an open ended format, it becomes possible for new types to be defined very quickly and easily by anyone. I remember that a while back you were looking at defining remote checksum offload for VXLAN and weren’t really able to have a standard bit to use for this even though there was physically space in the header. I think if GUE were to be widely adopted and deployed you might actually run into the same problem - it would not be easy to quickly agree on an allocation of a bit. Since all implementations going forward would need to be updated to know at least the size, I suspect that it would not be an easy process.

Private data might achieve the same purpose in some cases but there is probably little chance of it ever becoming interoperable with other implemetations. With Geneve TLVs, it is possible to define a new option with the intention of it becoming a standard or just as an experiment or private usage. In any case it is easy to allocate a type without having an impact on others’ implementation and yet is typed so that it can be picked up by others and interoperate if it proves to be useful.

In terms of number of options, I would generally expect to see just a couple options on each packet. Typically if there is more than one this would be from different components that are layered together (i.e. network virtualization metadata could be one layer and telemetry could be another layer). These layers are often defined and work independently but within a layer interoperability is through any of the usual ways - public standards, open source, vendors working together, or private data center operators choosing to do their own thing. In practice all of those will happen and I don’t think that it’s necessary to try to force everyone through a single chokepoint.

As a hypothetical question, how would you handle a situation where the security token you have defined for GUE is shown to be broken and needs to be replaced with a new option? I’m sure that in that case, you would feel the need to react immediately. It seems like the two choices would be to start using one of the unallocated flag fields and standardize and migrate before there is a collision with another option or resort to the private data field. Both seem to have some downsides.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 14:59:25 -0700

> As a hypothetical question, how would you handle a situation where the security token you have defined for GUE is shown to be broken and needs to be replaced with a new option? I’m sure that in tha…

The security option is defined with length(s) but no semantics. The
particular security algorithm used between two hosts is done out of
band agreement (key exchange is needed anyway). The only case where we
need to reconsider security this is when we need a larger token, in
that case we can define an extended security field option (but we
already have a 32 bit option, more than that and we're starting to
majorly impact MTU)..

The transform option contains an 8 bit transform type. Right now the
value for DTLS is defined, more transform types can be defined as
needed (we allow for private transform types also).

Tom

---

From: Jesse Gross <[email-masked]>
Date: Fri, 29 Jul 2016 22:13:22 +0000

> On Jul 29, 2016, at 2:59 PM, Tom Herbert <[email-masked]> wrote:
> 

Sorry, I should have been clearer - security was mostly just framing for the question on extensibility. The general point that I was trying to make was that there are a variety of cases where having open ended extensibility is useful. At least in some situations it’s probably important to be able to react to and deploy new options very quickly.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 15:31:39 -0700

> Sorry, I should have been clearer - security was mostly just framing for the question on extensibility. The general point that I was trying to make was that there are a variety of cases where havin…

True, but if I'm using a new extension that means I need end points to
understand and use these extensions. The fact the option is in a
flag-field shouldn't have any bearing on speed of deployment, in any
case some method of extending the protocol is done that presumably
changes the bits on the wire.

I don't deny that open ended extensibility can be useful in some
situations, but I do think you need to also acknowledge that this
flexibility is not without cost. Items like security and RCO checksum
offload might be sent in every packet within the data center. An
additional four bytes overhead, the type lookup, and validation of the
TLVs is overkill in that context. Extensibility is a requirement, but
so is an efficient and low overhead encapsulation.

Tom

---

From: Jesse Gross <[email-masked]>
Date: Sat, 30 Jul 2016 00:03:26 +0000

> On Jul 29, 2016, at 3:31 PM, Tom Herbert <[email-masked]> wrote:
> 

In this case, I’m mostly talking about “speed of consensus” vs. “speed of implementation”. I agree that flag fields shouldn't affect how long it takes to implement but it does make a difference in whether it is necessary to have central coordination for the extension space. Geneve effectively has a much sparser address space for its extensions compared to GUE, which means that it’s possible to define new options independently.

As a side note, the Geneve implementation in Open vSwitch actually does allow new TLVs to be defined in some cases without changing any code (specifically those that can be handled on a flow basis, which is OVS’s basic unit of work). This is a nice property of having a regular and self-describing format for parsing.

> so is an efficient and low overhead encapsulation.

Sure, there are certainly tradeoffs to be made in this area. I think that extensibility (in general) is probably more difficult to implement compared to a fixed version of the same thing but this is often compensated by the fact that only a single implementation of it is necessary. Similarly, while a more open ended format has some costs, it also has the benefit of a consistent datapath that only needs to know how to parse the TLVs that it cares about and not all options that have been defined. However, I absolutely agree that there are pros and cons to each approach and it depends on the use case.

The only thing that I can say is that over the past several years since the protocol was defined our experience with this tradeoff has been pretty good. Both the number of uses of Geneve and implementations have increased and as time has gone on, the uses have take more advantage of the flexibility and have not run into any significant implementation issues (either software or hardware).

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 17:16:31 -0700

Jesse,

In the list of TLV definitions you sent out I could only discern two
actual TLVs that have been defined. Cilium and VmWare stuff are
marketing slides, and an offhand reference to Geneve TLV in RCO draft
is by no means a specification. For comparison GUE has six defined
formally in I-Ds. So unless you're holding out a whole bunch of
development, I'm not seeing that the "inability to have a significant
number of extensions" is a material issue with GUE.

Tom

---

From: Jesse Gross <[email-masked]>
Date: Sat, 30 Jul 2016 00:49:27 +0000

> On Jul 29, 2016, at 5:16 PM, Tom Herbert <[email-masked]> wrote:
> 

Tom, I think that we’re going in circles here. All of the references I gave with the exception of remote checksum have implementations behind them. Not all of them are public. You can do as you wish with that information and the other examples I gave.

I think one practical difference between Geneve and GUE is that Geneve has implementations and option definitions by a number of different parties. To the best of my knowledge, all of the work on GUE has been done with your involvement which makes it much easier to manage for your specific use cases. I suspect that if GUE were to become a standard, you would find your flexibility in this regard would be significantly reduced and might run into some of the same issues as I’ve been describing. However, only time will tell.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 18:02:43 -0700

> I think one practical difference between Geneve and GUE is that Geneve has implementations and option definitions by a number of different parties. To the best of my knowledge, all of the work on G…

I think it's equally possible that if Geneve becomes a standard you
will wind up with a mess of vendor specific options that overlap in
functionality but don't have any interoperability except between each
vendors devices-- great for vendors, miserable for users. You might
end up wishing that extensibility in Geneve had been limited in the
first place. Only time will tell...

Anyway, I suppose we can agree that extensibility is a strong
requirement, but we'll have to agree to disagree on what form
extensibility should take and how much an encapsulation protocol
should allow!

Tom

---

From: Jesse Gross <[email-masked]>
Date: Sat, 30 Jul 2016 01:11:35 +0000

> On Jul 29, 2016, at 6:02 PM, Tom Herbert <[email-masked]> wrote:

Yes, I think we have consensus on that :)