Subject: [nvo3-dt-encap] Encap draft published by design team

From: Sam Aldrin <[email-masked]>
Date: Thu, 9 Feb 2017 09:03:45 -0800

Hello NVo3 WG,

NVo3 Design Team for encap has put in quite a bit of effort to meet,
discuss and hashout various requirements and issues and coming up with a
draft on proposed encap. Thanks to all who have participated and made it
possible.

This document could be found at
URL:            https://www.ietf.org/internet-drafts/draft-dt-nvo3-encap-00.
txt
Status:         https://datatracker.ietf.org/doc/draft-dt-nvo3-encap/
Htmlized:       https://tools.ietf.org/html/draft-dt-nvo3-encap-00

Kindly go through the document and review thoroughly and provide your
comments.
This will enable DT to close any issues or pending gaps.

cheers
Sam & Matthew

---

From: Tom Herbert <[email-masked]>
Date: Thu, 9 Feb 2017 11:33:20 -0800

On Thu, Feb 9, 2017 at 10:46 AM, Joe Touch <[email-masked]> wrote:
> Hi, all,
>
> I found this document to have useful context for discussion for the most
> part.
>
> Some notes below:
>
> - it would be useful to provide a summary of the unique requirements of NVO3
> that necessitate a new encapsulation protocol
>
> - there are many encapsulation issues not addressed, e.g., MTUs,
> frag/reassembly and fragment ID size, etc.
>
> -  the discussion of TLV vs. bitfields needs a bit more expansion. E.g., TLV
> necessarily requires serial processing, whereas bitfields can be processed
> in parallel.
>
It looks like the intent is to have a control plane that defines the
TLVs that will be used, a required ordering, and the assumption that
all TLVs are all the same size. If that is true then parsing TLVs can
be turned into a parallel problem, e.g. that can be accomplished with
2^N entries in a TCAM the same as bit fields. But, now implies that
explicit tunnels need to be established by the control plane before
the data plane can be used, and to be practical probably means that a
node will expect to receive pretty much the same set of TLVs with the
same order from all communicating devices (or maybe the intent is to
have a universal ordering of TLVs?).

It would be good to have a reference to other low layer protocols that
include TLVs or the like, especially IPv4 and IPv6, and an explanation
as to how TLVs in nvo3 will address all the issues of those protocols
that have deterred deployment of TLVs.

Tom

> - A "jump over" option is trivially possible in any TLV system, but begs the
> question of why the internal packet  flow information needs to be examined
> (vs using the flow info in the outermost header). Why is this an NVO3
> requirement?
>
> - TLV == bitfield for the first TLV field; i.e., TLV is a trivial way to
> extend bitfield to multiple bitfields if needed. That might be worth
> considering.
>
> - It might be useful to note that bitfields are much more difficult to
> handle optionally (e.g., it would require multiple bits, one for the value
> and several to tag the capability as "drop if not supported", "ignore if not
> supported", etc.).
>
> - I disagree with MUST support a minimum of 64 bytes of options; making that
> requirement effectively limits everyone to using only 64. IMO, that number
> is both too small and sets the protocol up to have "dead wood" (i.e., larger
> option space is specified but can't be relied upon so might never gain
> traction).
>
> Joe
>
>
> On 2/9/2017 9:03 AM, Sam Aldrin wrote:
>
> Hello NVo3 WG,
>
> NVo3 Design Team for encap has put in quite a bit of effort to meet, discuss
> and hashout various requirements and issues and coming up with a draft on
> proposed encap. Thanks to all who have participated and made it possible.
>
> URL:
> https://www.ietf.org/internet-drafts/draft-dt-nvo3-encap-00.txt
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 9 Feb 2017 12:05:41 -0800

On Thu, Feb 9, 2017 at 11:42 AM, Joe Touch <[email-masked]> wrote:
> Hi, Tom,
>
>
> On 2/9/2017 11:33 AM, Tom Herbert wrote:
>>> ...
>> 2^N entries in a TCAM the same as bit fields.
>
> That's true only if the TLVs in use are known in advance - if that's
> true, then the set of TLVs effectively becomes a large, static bitfield
> in the context of that tunnel.
>
Yep, in that case even the meanings of the bit fields would haven't to
fixed the whole data packet format could just be negotiated by the
control plane. The only issue with that is if intermediate boxes are
expected to parse the optional data then they need to be privy to what
was negotiated, but that would also be true if we expected middleboxes
to be able to efficiently parse TLVs and they need to see the control
plane negotiation to set up the processing path.

> If that's true, then easy hardware processing is possible only in the
> context of each individual tunnel, which requires a reconfigurable
> hardware parser for each tunnel.
>
> Otherwise, the only way to know where the Nth TLV field starts is to
> parse through the N-1th, which - recursively - reduces to requiring
> iterative processing.
>
Yes, and iterative processing of an open ended list of TLVs is still
hard to do in hardware _and_ software and in fact is an obvious DOS
vector. Last I checked OpenFlow, P4, and eBPF don't fully implement
loops that are needed to process an arbitrary list of TLVs (they
resolved this through some hackery and artificial constraints). The
unlimited flexibility of TLVs does not come without a cost...

Tom

>> But, now implies that
> (agreed - the "known in advance case, as above)
>
> Agreed, especially if those are the "inner" headers that might need to
> be parsed anyway.
>
> Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 9 Feb 2017 12:35:00 -0800

> This will enable DT to close any issues or pending gaps.
The Security Considerations section needs content. First and foremost,
in a multi-tenant data center ensuring strict isolation between
different tenants traffic seems fundamental and the mechanisms for
doing that should be explicit in the description of an encapsulation.
Bear in mind that when we use UDP for encapsulation there is typically
nothing in a host to prevent an unprivileged application from spoofing
well formed nvo3 packets and sending them to arbitrary destinations
(this is harder to do with other protocols such as TCP or GRE). A
24-bit VNI is not sufficient to provide any guarantee of virtual
network isolation.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Feb 2017 13:23:24 -0800

> option space is specified but can't be relied upon so might never gain
Having both a minimum and a different maximum value is superfluous.
The minimum value is the maximum value in practice. Consider that we
deploy a solution using various vendors each of which support a
different number of bytes in options. As long as we configure
something less than the minimum number of option bytes then everyone
is happy. The second we try to use more than the minimum number things
start to break. Even though the vendors can claim compliance with the
protocol we have lost interoperability.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Feb 2017 17:09:19 -0800

Hi Sam and DT,

Thanks for the work.

A few comments:

>From section 6.6:

"...if options are added over time and different subsets of options
are likely to be implemented in different pieces of hardware, then it
would be hard for the IETF to specify which options should get the
early bit fields."

- Many IETF protocols have flag fields. Please quantify why this is a
particular issue for nvo3, or why getting a bit an flags field is
particularly harder than getting an option number in something like IP
or TCP.

"But transit devices would have issues with future GUE bits being
defined for future options as well."

- I don't believe this is true (albeit "issues" is rather ambiguous).
When a new option is added in GUE their flags can only be added at the
tail of the flag bits. This will in no way impact the ability of
transit devices to process the options it already knows about.

"A benefit of TLVs from a HW perspective is that they are self
describing i.e., all the information is in the TLV."

- There seems to be a running theme in this draft about optimizing for
HW implementation. The nvo3 protocol is not a hardware protocol, the
requirement is that it is hardware friendly. Even if I were to believe
that implementing TLVs in HW was easier than bit fields, which I
don't, TLVs are still much harder and less efficient in when
implemented in software. Please try to strike some balance between the
descriptions of both HW and SW implementation.

-  IMO, this draft is unnecessarily speculative and for forward
looking. There is already a lot of practical experience deploying
protocols with things like TLVs and bit fields which is relevant since
most this draft's content is not specific to network virtualization.
I've already mentioned that the feasibility for using TLVs needs to be
reconciled with the experiences for IP options and IPv6 EH, and
mentioned previously on the list that GRE already provides a example
of a deployed encapsulation protocol that supports bit fields which
are commonly supported in HW for some time.

It would be nice to have some reference draft-ietf-rtgwg-dt-encap-02.
There is a lot of overlap between these drafts although the latter
goes in to significantly more depth for some topics.

Thanks,
Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 9 Feb 2017 10:46:47 -0800

Hi, all,

Some notes below:

- it would be useful to provide a summary of the unique requirements of
NVO3 that necessitate a new encapsulation protocol

-  the discussion of TLV vs. bitfields needs a bit more expansion. E.g.,
TLV necessarily requires serial processing, whereas bitfields can be
processed in parallel.

- A "jump over" option is trivially possible in any TLV system, but begs
the question of why the internal packet  flow information needs to be
examined (vs using the flow info in the outermost header). Why is this
an NVO3 requirement?

- It might be useful to note that bitfields are much more difficult to
handle optionally (e.g., it would require multiple bits, one for the
value and several to tag the capability as "drop if not supported",
"ignore if not supported", etc.).

- I disagree with MUST support a minimum of 64 bytes of options; making
that requirement effectively limits everyone to using only 64. IMO, that
number is both too small and sets the protocol up to have "dead wood"
(i.e., larger option space is specified but can't be relied upon so
might never gain traction).

Joe

On 2/9/2017 9:03 AM, Sam Aldrin wrote:
> discuss and hashout various requirements and issues and coming up with
> a draft on proposed encap. Thanks to all who have participated and
> made it possible.
>
> <https://www.ietf.org/internet-drafts/draft-dt-nvo3-encap-00.txt>
> <https://datatracker.ietf.org/doc/draft-dt-nvo3-encap/>
> <https://tools.ietf.org/html/draft-dt-nvo3-encap-00>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 9 Feb 2017 11:42:34 -0800

Hi, Tom,

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 9 Feb 2017 13:10:33 -0800

> unlimited flexibility of TLVs does not come without a cost...
Wouldn't the obvious solution be to let the entire encapsulation
protocol be negotiated by the endpoints anyway?

I.e., take the specification of that protocol out of purview of NVO3?
The only thing NVO3 needs to do is:

    - develop a negotiation protocol
    - indicate how to use some subset of encapsulations

If the key issue here is the debate between bitfield and TLV, that
should be taken off the table as an issue.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 Feb 2017 13:40:19 -0800

On 2/10/2017 1:23 PM, Tom Herbert wrote:
> The minimum value is the maximum value in practice.
Agreed - that was basically my point (though you put it more succinctly).

Joe

---

From: Sami Boutros <[email-masked]>
Date: Wed, 15 Feb 2017 17:36:18 +0000

Hi Tom,

>24-bit VNI is not sufficient to provide any guarantee of virtual

Can you please elaborate more on why 24- bit is not sufficient to provide network isolation?
We have the section 6.2.2 on security and integrity that we borrowed the text you supplied for it’s content.
We can refer in the security considerations to the 6.2.2 section? Is this what you are looking for?

Thanks,

Sami

---

From: Tom Herbert <[email-masked]>
Date: Wed, 15 Feb 2017 14:36:23 -0800

> Can you please elaborate more on why 24- bit is not sufficient to provide network isolation?

1) From a security standpoint small identifiers are easily guessable
by an attacker and does not allow much entropy, a single bit flip in
the VNI could result in mis-delivery of a packet to the wrong tenant.
Mis-delivery due to corruption is why the UDP checksum is important to
enable, but even that is too weak to guarantee isolation-- this one
reason why we defined header security in GUE.
2) I don't believe that 24-bit identifiers scale to large deployments.
It is quite possible that a large provider may have upwards of 1M
tenants and sub-tenants that need identifiers. If we take into account
that these may have hierarchical allocation, flag bits (e.g. trusted
vs. not trusted tenant), and a strong desire to avoid ever having to
renumber networks-- 24 bits starts to look small and even 32 bits
might not be enough. I really wish the design time provided more of an
analysis on the scaling properties of 24 bit VNI instead of just
saying VXLAN and Geneve already have that size so it must be okay.

> We can refer in the security considerations to the 6.2.2 section? Is this what you are looking for?

Section 6.2.2 only describes a problem from a rather high level about
how it _might_ be solved in Geneve not how it is _actually_ solved.
This is indicative of the most of the draft I think. There are a lot
of "cans" and "possibilities" in the draft but very few concrete
statements on what the protocol does already and how it performs in
the real world. For instance, in the absence of any actual TLVs being
defined and implemented how can we _really_ know what the operational
characteristics are? How can we know how these are implemented in HW
or even if it is feasible,  how the C-bit might help, rather error
handling and corner cases are actually covered, how is this going to
withstand DOS attack, or even if the primary technical concern with
Geneve can be addressed? The current answers to these questions seem
to be based only on speculation which doesn't inspire confidence in
those answers for me.

If the decision is to pursue Geneve for standardization, I really hope
that the chairs and ADs quickly prioritize defining and implementing
some critical TLVs. If this is deferred any longer I think these will
die on the vine in the same way that IP options became implicitly
deprecated even before they were even used.

Tom

---

From: Sami Boutros <[email-masked]>
Date: Thu, 16 Feb 2017 18:12:13 +0000

>saying VXLAN and Geneve already have that size so it must be okay.

We discussed mandating UDP checksum and having Security extensions to secure 
both the tunnel header and payload for security and integrity. 
We can add the above text in the security considerations.

We saw value in keeping the VNI as part of the tunnel header given 
the existing implementations being able to identify the service quickly.

As well, we discussed having more than 24 bit VNI via using extensions.

>to be based only on speculation which doesn't inspire confidence in

In the security section you provided text for, we talked about the 
possibility of authenticating the tunnel header and payload via extensions 
to address concern of spoofing VNI and payload security.
If you think we need to beef the text more to discuss how C bit could help?
Or how can we withstand DOS attack? Please suggest some text?

However, wouldn’t discussing a solution such as (1) defining TLV(s) and (2) 
Exact operational characteristics be out of scope of this draft?

>deprecated even before they were even used.

Agreed,

Sami

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 12:27:56 -0800

> to address concern of spoofing VNI and payload security.

Please look at draft-herbert-gue-extensions, that draft realizes the
"possibility of authenticating the tunnel header and payload". These
extensions could, along with the others in that draft, be considered a
good set of fundamental extensions. They could be implemented as TLVs
in Geneve. One caveat though-- processing order of extensions is
important. For instance, it makes sense to authenticate the header
before decrypting the payload. Processing order in GUE is not an issue
since bit-fields allow random access at O(1). This is harder with
TLVs. Searching for one particular TLV is an O(N) operation for number
of TLVs in packet, searching for M of them becomes O(M * N). There are
known techniques to get the search back to O(N), but they tend to be
somewhat complex. One common method is to build a scoreboard in one
walk over the TLVs and then processing them in order from the
scoreboard-- this is how we implement options processing in TCP.

> Or how can we withstand DOS attack? Please suggest some text?
The problems of TLVs, particularly that they are unordered, require
iterative processing, and are susceptible to DOS are well known. I and
others have raised these issues as serious technical concerns of
Geneve several times for quite some time now (please look through all
the discussion in nvo3 list). Not only that, there are high profile
low level dataplane protocols that have included TLVs, notably IP
options and IPv6 EHs, for which TLVs have never achieved widespread
use. It's not like we don't want to use these, in fact just last week
we proposed some good use cases for HBH options in Facebook network,
but we also seem to get tripped up by some deployed devices that don't
handle them. If you look at the discussion on v6ops you'll see a
similar story, a lot of service providers disallow IPv6 EH because
they are a known DOS vector.

Answers to how to these problems have yet to be forthcoming. The
proposed answers usually seem to involve imposing ordering
requirements on TLVs, as the draft alludes to in the discussion of
using a control plane to enforce order. But that inevitably leads to
more complexity, more error handling, and in the case of this draft
creates dependencies of the dataplane on a control plane that has yet
to be defined.

> Exact operational characteristics be out of scope of this draft?
This not about knowing exact operational characteristics for the
protocol, the problem is without any implementation at all we don't
know anything about how these protocol mechanisms will function or if
they are even feasible (except that we could try to extrapolate from
experiences with similar deployed protocols as I mentioned, but then
that doesn't seem to bode well for Geneve).

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 13:11:33 -0800

On 2/16/2017 12:27 PM, Tom Herbert wrote:
> iterative processing,
That's trivially avoided by forcing the order.

As I noted before, all that is required for equivalently easy processing
is that both TLVs and bitfields use only known variants in only known
orders.

Which can be negotiated, if you're looking for flexibility.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 13:14:07 -0800

> is that both TLVs and bitfields use only known variants in only known
Joe, do you know of any protocols that enforce such an ordering?

Thanks,
Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 13:21:02 -0800

> Joe, do you know of any protocols that enforce such an ordering?

No, because in most cases the "T" is intended to allow arbitrary reordering.

My point is just that it isn't TLV itself that affects hardware and
parallelization; it's the potential for variation.

The same variation and need for serial processing could be true for new
definitions for previously undefined bitfields values. E.g., consider
that the first few bits of an IP packet determine whether the addresses
are 32 bits or 128 bits, etc.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 15:26:12 -0800

> No, because in most cases the "T" is intended to allow arbitrary reordering.
I'd feel more comfortable with calling this a "trivial" solution for
TLVs if there was a working example of protocol that has implemented
it and had been successfully deployed.

> are 32 bits or 128 bits, etc.
Not in the same way. At question here is how the packet is parsed to
identify the protocol fields, not how individual fields are processed.
Since TLVs are unordered that means the number of discrete packet
header formats including extensions is combinatoric. Specifically, the
number of possible combinations is

SUM(K! * (N choose K)) == SUM(N! / (N-K)!) == N! * SUM(1 / K!), for K
= 0..N which approximates to e * N!.

For bit-fields the number of possible combinations is always 2^N (or
2^(N+1) to allow both plain IPv4 and IPv6)

The former function grows at a much steeper rate. For instance, with
just 8 extensions there are 109601 possible combinations of TLVs in a
packet. For bit-fields there are just 256. So without some constraints
applied to ordering, TLVs are not amenable to use with HW
parallelization techniques such as TCAM. If the problem is
unconstrained, HW processing degenerates to be iterative over the
packet (alignment and making TLVs same size don't help here). The
situation in SW is not particularly better, bit-fields processing can
be loop free, whereas TLVs will require a loop.

Susceptibility of the TLV protocol mechanism to DOS is demonstrated
when an attacker spoofs a packet containing the maximum number of
minimum sized TLVs. An attackers job is made easier if the protocol
allows TLVs that can be ignored. For instance, to attack Geneve they
could just create a list of random tiny "ignore" TLVs and set the
C-bit to ensure the receiver processes the TLVs. An obvious mitigation
to this might be to require mandatory options to appear before
ignorable options (that is introduce a weak ordering requirement).
With that the C-bit could be completely eliminated since it's just as
easy to check the type of the first option to see if mandatory options
are present. Also, if the receiver only processes mandatory options
then they know to stop processing when they hit the first
non-mandatory option. This might be sufficient defense against the DOS
attack I described, but then of course if everyone were to ignore
non-mandatory TLVs in this manner then we'd have to ask what the point
is of even including them in the protocol definition.

Admittedly, without any actual TLVs defined in Geneve all of this is
all just speculation on my part!

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 15:30:09 -0800

> all just speculation on my part!
Agreed, and more specifically, regardless of the flexibility of TLVs in
general, if the negotiation protocol specifies a fixed set of them, each
with fixed, known length, then even though the TLV allows flexibility in
what COULD appear, a given pair of endpoints can rely on a fixed set
that is easy to parse in parallel.

That's why TLVs are a superset of bitfields; a bitfield is just a single
TLV.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 15:45:25 -0800

> that is easy to parse in parallel.
Sure, if you require protocol negotiation to precede use of the
dataplane then not only can we define the required order of TLVs, but
we can also define the allowable set of TLVs that each side can send.
The concept of having ignorable TLVs could just go away (that is a
good thing IMO). Option negotiation is probably one of things that
mades TCP options deployable and avoids the concept of ignoring
options after negotiation.

But, as I said this idea creates a new dependency on a control plane
which is TBD. I'm afraid this could be a opening a Pandora's box of
new complexity that the group didn't bargain for...

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 15:52:23 -0800

> new complexity that the group didn't bargain for...
You need a control plane to setup the endpoints of a tunnel anyway.
Indicating a fixed set of features for that tunnel is as easy as "use
Bob", where "Bob" is defined elsewhere.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 16:10:10 -0800

> Bob", where "Bob" is defined elsewhere.
The interaction between the control plane and dataplane will need to
be explicit in the definition of the protocol as it is in TCP. And
this method creates new conditions that need to be handled. For
instance, if Bob says that he'll send option A and then option B, but
we get a packet from him with option B before option A then what does
that mean? Is this an error? What if Bob wants to send options A,B,C
in that order, but Sally wants to only receive them in order C,B,A?
Whose ordering requirements take precedence? What about middleboxes
that need to parse TLVs, would they have a say in this negotiation?
What about options in a multicast packet, what ordering of TLVs would
be used for those? And so on...

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 16:20:47 -0800

Hi Tom,

On 2/16/2017 4:10 PM, Tom Herbert wrote:
> that mean? Is this an error? 
It depends on what we define, but IMO that's a silent drop or - at best
- a rate-limited warning in the log file.

> What if Bob wants to send options A,B,C
I was using "Bob" to refer to the protocol configuration, the way that
we use strings to refer to such parameterization of more flexible
systems for security algorithms. You're using Bob to refer to an
endpoint, so let's call the protocol configuration we want to use "kiwi".

E.g., see the list of encryption transforms for IKE.

In this case, we could define one required protocol that can easily be
"X with TLVs A, B, C in that order only".

My point is that the protocol used need not be so ossified in its
specification, e.g., the encapsulation protocol could specify TLVs
A,B,C,D,E, and F, and allow any order in general. However, the
complexity of dealing with all possible combinations and orders need not
impact NVO3.

> Whose ordering requirements take precedence?
The same as any negotiation protocol - there's typically an offer and a
counter based on a subset. You say you want "kiwk, orange, or grape" (in
order of preference) and the receiver either picks ONE or refuses.
That's going to have to happen anyway.

> What about middleboxes
Middleboxes don't play by any known rules. They'd have just as much
trouble with TLV ordering and selection as they would with changes from
"don't care" to "care" of bits in bitfields.

> be used for those? And so on...

Same issues apply in all cases. TLVs are no different from bitfields.
You need to negotiate what they mean in both cases for all uses.

I'm not arguing for TLVs, just pointing out that the claimed reasons for
picking bitfields over TLVs is nonsense.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 16:39:40 -0800

On Thu, Feb 16, 2017 at 4:20 PM, Joe Touch <[email-masked]> wrote:
> It depends on what we define, but IMO that's a silent drop or - at best - a
> rate-limited warning in the log file.
>
> I was using "Bob" to refer to the protocol configuration, the way that we
> use strings to refer to such parameterization of more flexible systems for
> security algorithms. You're using Bob to refer to an endpoint, so let's call
> the protocol configuration we want to use "kiwi".
>
> In this case, we could define one required protocol that can easily be "X
> with TLVs A, B, C in that order only".
>
> A,B,C,D,E, and F, and allow any order in general. However, the complexity of
> dealing with all possible combinations and orders need not impact NVO3.
>
> order of preference) and the receiver either picks ONE or refuses. That's
> going to have to happen anyway.
>
> What about middleboxes
> Middleboxes don't play by any known rules. They'd have just as much trouble
> with TLV ordering and selection as they would with changes from "don't care"
> to "care" of bits in bitfields.
>
> Same issues apply in all cases. TLVs are no different from bitfields. You
> need to negotiate what they mean in both cases for all uses.
>

The operational issues we see with TLVs in terms of performance and
DDOS are not aberrations, they are fundamental issues we face in
deployment. Maybe with enough work and new implementation these issues
can be addressed in Geneve, but honestly given the history of similar
protocols in IETF and that of nvo3 I have doubts. It would be great if
I'm proven wrong in this :-).

Thanks for the interesting discussion...

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 16:48:50 -0800

On 2/16/2017 4:39 PM, Tom Herbert wrote:
> deployment.
Agreed, in the case where TLV sets are not fixed for a given path. The
same is also true for bitfields: Ethernet uses a different Ethertypes
for IPv4 and IPv6, even though they're intended to be treated as a
single protocol class with internal versioning indicated by bitfields.

Unknowns are the cause of the problem - in either case.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Feb 2017 18:17:56 -0800

> Unknowns are the cause of the problem - in either case.
Joe,

I agree with that, however there are fewer unknowns to deal with when
using bit-fields as opposed to TLVs. Once the sender and receiver
agree on options to be used, with bit-fields the order and length are
fixed. With TLVs these are variables that need to be considered with
each packet. This is why bit-fields naturally yield the simpler and
more feasible implementation. The trade-off for this simplicity is
loss of flexibility as pointed out by the draft. Defining 100s of
bit-fields in GUE probably wouldn't work. My response to that argument
is to ask why would we ever want to define 100s of extensions in a
protocol? Again, looking at other protocols that have extensibility we
see relatively few extensions being defined. IP has around 20 options,
TCP around 50. Grant it some of these options allow different lengths,
but for the most part defining new extensions seems to be a rare
occurrence. For GUE we estimated that after an initial set of
extensions have been defined (about ten), we'll maybe add one per
year. Extrapolating from this, we believe the protocol can accommodate
all the necessary extensibility for at least a thirty years.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2017 22:57:31 -0800

> On Feb 16, 2017, at 6:17 PM, Tom Herbert <[email-masked]> wrote:
> 
> fixed.

I gave an example above where that's not the case. A value in a single bit field can redefine other fields - and that can't be avoided as a possibility unless all values of all fields are known in advance and don't do so.

> With TLVs these are variables that need to be considered with
> each packet.

Incorrect - the same is true for TLV. If you specify the order, then they're ordered. If you specify which are required, then they're required. They're just different size bit fields.

> This is why bit-fields naturally yield the simpler and
> more feasible implementation.

This is the conclusion that cannot be reached. If you want to pick bitfields arbitrarily, fine. If you have another reason, present it. But the same rules apply to bitfields and TLVs - known patterns are equally easy, and unknowns are equally hard.

TLVs make the difficulties easier to see, but not more difficult to avoid.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 17 Feb 2017 09:04:55 -0800

> I gave an example above where that's not the case. A value in a single bit field can redefine other fields - and that can't be avoided as a possibility unless all values of all fields are known in…
We expressly disallow that in GUE. New extensions are always additive
information. From section 3.3.1:

"Flags (or paired flags) are idempotent such that new flags must not
cause reinterpretation of old flags. Also, new flags should not alter
interpretation of other elements in the GUE header nor how the message
is parsed (for instance, in a data message the proto/ctype field
always holds an IP protocol number as an invariant)."

> Incorrect - the same is true for TLV. If you specify the order, then they're ordered. If you specify which are required, then they're required. They're just different size bit fields.

But again, we don't have any examples of a protocol with ordered TLVs
that does this and there is no concrete proposal for doing this in
Geneve so this idea is just speculation. There are already
well-deployed protocols with bit-fields such that options order is
invariant (e.g. GRE) and that is characteristic is used to implement
efficient parsing.

> This is the conclusion that cannot be reached. If you want to pick bitfields arbitrarily, fine. If you have another reason, present it. But the same rules apply to bitfields and TLVs - known patter…
I've made pretty much presented all the rationale. Processing a list
of TLVs is demonstrably less efficient than an equivalent set of
bit-fields. In HW we can use TCAM to parse bit fields, in software we
can do it without loops. I've also given the argument for random
access of extensions when processing order is relevant. You can
readily see these characteristics in open source SW implementations.
Last time I checked with HW experts about this they claimed TLVs were
problematic, that was a couple of years ago I suppose someone might
have solved the combinatorics of parsing since then but haven't heard
that from anyone yet.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 17 Feb 2017 09:55:47 -0800

On 2/17/2017 9:04 AM, Tom Herbert wrote:
> Geneve so this idea is just speculation.
Ordered TLVs are the same thing as bitfields in known orders. The only
difference is whether the protocol allows the flexibility, in alternate
configurations, of supporting alternate fields in alternate orders.

However, the larger issue here is that the doc assumes a requirement for
efficient support in current hardware.

Joe

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 13 Feb 2017 18:39:59 -0800

Dear Authors, WG Chairs, et. al,
firstly, many thanks to the Design Team for very good start. Please find my
notes, comments to the document below. Greatly appreciate your kind
consideration.

   - Section 2 states "The chosen design should also operate well with ICMP
   and in ECMP environments." While I understand ECMP environment and possible
   ways to generate entropy value for path selection, I'm bit puzzled by role
   of ICMP can play. Is it to suggest use of particular OAM mechanism?
   - Section on Telemetry, in my opinion, overlooks value of mechanism that
   generates entropy value at the edge instead of generating it at transit
   nodes. This mechanism is known in IP/MPLS as Entropy label and being
   proposed in BIER encapsulation header as Entropy field. I think that NVO3
   header can benefit from allocating the Entropy field. Then, ensuring that
   active OAM messages share the same path as data packets would be as simple
   as using the same Entropy value. More so, by applying different values
   active OAM be able to monitor all available multipaths even if there's no
   data traffic.
   - I believe that "in band parameters" is misleading and confusing
   terminology. Active OAM can be and, in my opinion, must be in-band, i.e.
   receive the same forwarding and scheduling treatment as data flow being
   monitored.

   - Now I'm making assumption that the draft-ietf-nvo3-geneve should be
   used to discuss format of NVO3 header proposed by the design team.
      - If that is correct, I'm interested to discuss benefit of using O
      bit to specify that OAM message follows the NVO3 header vs. define OAM as
      one of values of Protocol Type field.
      - If the OAM message identified by value of Protocol Type field, then
      the O bit can be reused for passive performance measurement
(allocating two
      bit-long field would be ideal).

Regards,
Greg

On Thu, Feb 9, 2017 at 9:03 AM, Sam Aldrin <[email-masked]> wrote:

> URL:            https://www.ietf.org/internet-drafts/draft-dt-nvo3-
> encap-00.txt