Subject: [nvo3] Security in VXLAN-GPE (was Re: Consensus call on encap proposals)

From: Tom Herbert <[email-masked]>
Date: Wed, 27 Jul 2016 13:43:31 -0700

On Wed, Jul 27, 2016 at 1:15 PM, Fabio Maino <[email-masked]> wrote:
> On 7/27/16 12:27 PM, Tom Herbert wrote:
>>
>> On Wed, Jul 27, 2016 at 11:00 AM, Fabio Maino <[email-masked]> wrote:
>>>
>>> On 7/27/16 10:53 AM, Tom Herbert wrote:
>>>>
>>>> On Wed, Jul 27, 2016 at 10:44 AM, Dino Farinacci <[email-masked]>
>>>> wrote:
>>>>>>
>>>>>> On Jul 26, 2016, at 3:11 PM, Fabio Maino (fmaino) <[email-masked]>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 7/26/16 12:08 PM, Dino Farinacci wrote:
>>>>>>>>>
>>>>>>>>> Having VXLAN as an Independent Stream RFC gives a document to be
>>>>>>>>> used
>>>>>>>>> to
>>>>>>>>> innovate from.   I believe that was the intent of VXLAN-GPE - to
>>>>>>>>> provide the ability
>>>>>>>>> for needed extensions.
>>>>>>>>
>>>>>>>> The only new feature the VXLAN-GPE brings to the table is a way to
>>>>>>>> identify that NSH is being encapsulated.
>>>>>>>
>>>>>>> or any other shim header, NSH just happens to be one.
>>>>>>
>>>>>> Precisely.  GPE addresses the limitation associated with VXLAN, namely
>>>>>> the lack of multi-protocol support.  As we've seen on the list, other
>>>>>> protocols have come up: MPLS for example.
>>>>>
>>>>> This is technically not true. As long as you are willing to spend 14
>>>>> bytes on a MAC header, then anything with an ethertype can be
>>>>> encapsulated
>>>>> with VXLAN.
>>>>>
>>>>> And I have working code that encapsulates both IPv4 and IPv6 in VXLAN
>>>>> with ethertypes 0x0800 and 0x86dd, respectively.
>>>>>
>>>>>> There are some other technical benefits as well: version and OAM.
>>>>
>>>> As do Geneve and GUE. But this thread is not about touting the
>>>> features of these protocols, it is about the technical objections to
>>>> them. My major objections (from the list I posted) to VXLAN-GPE is
>>>> that it has no extensibility and offers no security of the VNI. These
>>>> are showstoppers in my deployment.
>>>
>>>
>>>
>>> Tom,
>>>
>>> extensibility is achieved via shim header:
>>>
>>> - if you want to carry end to end metadata you can define the appropriate
>>> shim header
>>>
>>> - If you want to protect the integrity of the VNI, you can include an
>>> HMAC
>>> of the VNI in the shim header
>>>
>> Please point me to the specification where this shim header with the
>> HMAC is described and add a reference to it in the security
>> considerations section of the VXLAN-GPE draft.
>
>
> Hi Tom,
> there are a lot of use cases that can be addressed with shim headers, and
> each solution may look fairly different depending on the requirements.
>

The security option including the specific header format is described
in draft-herbert-gue-extensions. You can use the same data format.

> Can you point me to an nvo3 specification of what are the requirements that
> need to be addressed to provide VNI integrity protection? If the
> requirements are the same you used for GUE, the shim header could use an
> encoding similar to the GUE security option.
>

Section 6.2 of draft-ietf-nvo3-security-requirements describes the
dataplane requirements. The most important requirement of nvo3 is to
maintain strict isolation between tenant virtual networks. The problem
is exacerbated in UDP encapsulation since any non privileged
application can send a UDP packet to any port thereby making it easier
to inject packets into a virtual network than in a non-UDP based encap
protocol like nvgre. In our large scale network we need strong
mechanisms to guarantee this isolation; Ethernet CRC and network
mechanisms like firewalls or address spoofing are not sufficient for
multi-tenant nv security.

> Lacking the requirement document, I'm sure we can add a paragraph in the
> security section that describes how it can be done.
>
Please be specific in that. If you just say "use NSH" that is not
helpful at all to your potential implementors or users. If we were to
implement security we need to know the exact bits that are set in the
packet.

Tom

---

From: Fabio Maino <[email-masked]>
Date: Fri, 29 Jul 2016 11:11:45 -0700

> multi-tenant nv security.

Tom,
I went back and re-read the draft. Requirement 10, that is the one that 
seems relevant here, is not a MUST. From that standpoint VXLAN-GPE does 
comply with that draft.

I don't want to argue that there are use cases where authenticaton of 
the VNI might be useful, but certainly there are others where it isn't. 
Otherwise the WG would have settled for a MUST there.

In the design of VXLAN-GPE we have choosen to not burden the cost of ALL 
the implementations with a mandatory authentication field. However, 
thanks to VXLAN-GPE flexibility and extensibility, this function can be 
added via shim header.

> implement security we need to know the exact bits that are set in the

That should not go in the VXLAN-GPE draft. All that draft needs to 
specify is that there might be a following shim header. The VXLAN-GPE 
draft does that today.

if I read your comment right, you seem to agree that it can be done. Is 
this your MAJOR objection to VXLAN-GPE? If we were to specify such a 
draft, would that change your opinion in supporting VXLAN-GPE for WG 
adoption?

Thanks,

Fabio

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 11:45:09 -0700

On Jul 29, 2016 11:12 AM, "Fabio Maino" <[email-masked]> wrote:
>>>>>> On Wed, Jul 27, 2016 at 10:44 AM, Dino Farinacci <[email-masked]
>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Jul 26, 2016, at 3:11 PM, Fabio Maino (fmaino) <[email-masked]
>
>>>>>>>> wrote:
>>>>>>>>>
>
> I don't want to argue that there are use cases where authenticaton of the
VNI might be useful, but certainly there are others where it isn't.
Otherwise the WG would have settled for a MUST there.
> In the design of VXLAN-GPE we have choosen to not burden the cost of ALL
the implementations with a mandatory authentication field. However, thanks
to VXLAN-GPE flexibility and extensibility, this function can be added via
shim header.
> That should not go in the VXLAN-GPE draft. All that draft needs to
specify is that there might be a following shim header. The VXLAN-GPE draft
does that today.
> if I read your comment right, you seem to agree that it can be done. Is
this your MAJOR objection to VXLAN-GPE? If we were to specify such a draft,
would that change your opinion in supporting VXLAN-GPE for WG adoption?
My major objections are lack of security and extensibility. The counter
argument seems to be use a shim header for security or extensions, but
there has never be any specifics offered (I have asked several times now
how to implement a 64 bit security cookie in VXLAN-GPE). Also the security
considerations section in the draft provides no useful guidance.

In my data center if we are going to commit to using a protocol for the
long term (like if we buy into HW support), I need to believe it is
adaptable to face changing security risks and new technologies. I know how
to extend GUE and in fact we have useful extensions defined. I believe I
would know how to extend Geneve, although I still wish there were some
standard TLVs spec'ed. However, I don't know how extend VXLAN-GPE. I can't
afford to find out two years from now that I need switch protocols because
we can't adapt the running protocol. This is particularly true in data
center security, the demands for strong security mechanisms will only be
increasing-- I need a protocol that can deal with that...

This is all just my opinion of course :-)

Tom

>
> Thanks,
>
> Fabio
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 29 Jul 2016 12:16:32 -0700

> My major objections are lack of security and extensibility.

I don't know if this helps your security concern but the LISP WG has draft-ietf-lisp-crypto that is supported by the LISP encapsulating header. 

VXLAN-GPE also had a goal to make VXLAN and LISP converge on a similar header at the same time as having more demux options. This is why, at the same time VXLAN-GPE was created, the LISP-GPE header was also  created. 

So if the LISP control-plane is used for these data-planes, then key exchange, encryption, and authentication can be supported for all the header formats above. 

All that needs to be spec'ed is to allocate 2 bits from the *-GPE headers for a key-id field. 

Dino

---

From: Fabio Maino <[email-masked]>
Date: Fri, 29 Jul 2016 12:17:15 -0700

On 7/29/16 11:45 AM, Tom Herbert wrote:
>
> On Jul 29, 2016 11:12 AM, "Fabio Maino" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> >> On Wed, Jul 27, 2016 at 1:15 PM, Fabio Maino <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >>>
> >>>> On Wed, Jul 27, 2016 at 11:00 AM, Fabio Maino <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >>>>>
> >>>>>> On Wed, Jul 27, 2016 at 10:44 AM, Dino Farinacci 
> <[email-masked] <mailto:[email-masked]>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> On Jul 26, 2016, at 3:11 PM, Fabio Maino (fmaino) 
> <[email-masked] <mailto:[email-masked]>>
> >>>>>>>> wrote:
> >>>>>>>>>
> > I went back and re-read the draft. Requirement 10, that is the one 
> that seems relevant here, is not a MUST. From that standpoint 
> VXLAN-GPE does comply with that draft.
> >
> > I don't want to argue that there are use cases where authenticaton 
> of the VNI might be useful, but certainly there are others where it 
> isn't. Otherwise the WG would have settled for a MUST there.
> >
> > In the design of VXLAN-GPE we have choosen to not burden the cost of 
> ALL the implementations with a mandatory authentication field. 
> However, thanks to VXLAN-GPE flexibility and extensibility, this 
> function can be added via shim header.
> >
> >
> >>
> > if I read your comment right, you seem to agree that it can be done. 
> Is this your MAJOR objection to VXLAN-GPE? If we were to specify such 
> a draft, would that change your opinion in supporting VXLAN-GPE for WG 
> adoption?
> >
> My major objections are lack of security and extensibility. The 
> counter argument seems to be use a shim header for security or 
> extensions, but there has never be any specifics offered (I have asked 
> several times now how to implement a 64 bit security cookie in 
> VXLAN-GPE). Also the security considerations section in the draft 
> provides no useful guidance.
>
> In my data center if we are going to commit to using a protocol for 
> the long term (like if we buy into HW support), I need to believe it 
> is adaptable to face changing security risks and new technologies. I 
> know how to extend GUE and in fact we have useful extensions defined. 
> I believe I would know how to extend Geneve, although I still wish 
> there were some standard TLVs spec'ed. However, I don't know how 
> extend VXLAN-GPE. I can't afford to find out two years from now that I 
> need switch protocols because we can't adapt the running protocol. 
> This is particularly true in data center security, the demands for 
> strong security mechanisms will only be increasing-- I need a protocol 
> that can deal with that...
>

I think that the point at hand here is that the AD is pushing the WG to 
have a discussion on which encapsulations should be selected for 
standard track. What should guide that decision are the requirements 
emerging from the set of documents that are adopted by the WG.

As I said in the previous message, IMO VXLAN-GPE does address the 
security requirements as expressed in 
draft-ietf-nvo3-security-requirements. I note that you have not objected 
to that point.

Also, I pointed to NSH as ONE example of how VXLAN-GPE can be extended 
using a shim header. NSH implements SFC and metadata transport. Looking 
at what NSH + VXLAN-GPE achieves, it makes fairly hard to maintain that 
VXLAN-GPE is not extensible. Extensibility is achieved via layering. It 
might not be the same way extensibility is achieved by the other 
contenders, but it's a fairly established way to design protocols.

I think this should answer the objections to security and extensibility 
for the purpose of the selection of an nvo3 encapsulation.

We can certainly have side conversations on how to address particular 
functions of particular use cases using VXLAN-GPE, but IMO it won't  add 
much to the selection process that the AD is pushing for.

Fabio

> > [email-masked] <mailto:[email-masked]>

---

From: Fabio Maino <[email-masked]>
Date: Fri, 29 Jul 2016 12:19:12 -0700

On 7/29/16 12:16 PM, Dino Farinacci wrote:
> I don't know if this helps your security concern but the LISP WG has 
> draft-ietf-lisp-crypto that is supported by the LISP encapsulating 
> header.
>
> VXLAN-GPE also had a goal to make VXLAN and LISP converge on a similar 
> header at the same time as having more demux options. This is why, at 
> the same time VXLAN-GPE was created, the LISP-GPE header was also 
>  created.
>
> So if the LISP control-plane is used for these data-planes, then key 
> exchange, encryption, and authentication can be supported for all the 
> header formats above.
>

or use the appropriate shim header in combination with VXLAN-GPE as 
defined today.

Fabio

---

From: Jesse Gross <[email-masked]>
Date: Fri, 29 Jul 2016 19:44:01 +0000

> On Jul 29, 2016, at 12:17 PM, Fabio Maino <[email-masked]> wrote:
> 
>> > I went back and re-read the draft. Requirement 10, that is the one that seems relevant here, is not a MUST. From that standpoint VXLAN-GPE does comply with that draft.
>> >
>> > I don't want to argue that there are use cases where authenticaton of the VNI might be useful, but certainly there are others where it isn't. Otherwise the WG would have settled for a MUST there.
>> >
>> > In the design of VXLAN-GPE we have choosen to not burden the cost of ALL the implementations with a mandatory authentication field. However, thanks to VXLAN-GPE flexibility and extensibility, this function can be added via shim header.
>> >
>> >
>> >>
>> > That should not go in the VXLAN-GPE draft. All that draft needs to specify is that there might be a following shim header. The VXLAN-GPE draft does that today.
>> >
>> > if I read your comment right, you seem to agree that it can be done. Is this your MAJOR objection to VXLAN-GPE? If we were to specify such a draft, would that change your opinion in supporting VXLAN-GPE for WG adoption?
>> >
>> My major objections are lack of security and extensibility. The counter argument seems to be use a shim header for security or extensions, but there has never be any specifics offered (I have asked several times now how to implement a 64 bit security cookie in VXLAN-GPE). Also the security considerations section in the draft provides no useful guidance.
>> 
>> In my data center if we are going to commit to using a protocol for the long term (like if we buy into HW support), I need to believe it is adaptable to face changing security risks and new technologies. I know how to extend GUE and in fact we have useful extensions defined. I believe I would know how to extend Geneve, although I still wish there were some standard TLVs spec'ed. However, I don't know how extend VXLAN-GPE. I can't afford to find out two years from now that I need switch protocols because we can't adapt the running protocol. This is particularly true in data center security, the demands for strong security mechanisms will only be increasing-- I need a protocol that can deal with that...
>> 
> I think that the point at hand here is that the AD is pushing the WG to have a discussion on which encapsulations should be selected for standard track. What should guide that decision are the requirements emerging from the set of documents that are adopted by the WG. 
> 
> As I said in the previous message, IMO VXLAN-GPE does address the security requirements as expressed in draft-ietf-nvo3-security-requirements. I note that you have not objected to that point. 
> 
> Also, I pointed to NSH as ONE example of how VXLAN-GPE can be extended using a shim header. NSH implements SFC and metadata transport. Looking at what NSH + VXLAN-GPE achieves, it makes fairly hard to maintain that VXLAN-GPE is not extensible. Extensibility is achieved via layering. It might not be the same way extensibility is achieved by the other contenders, but it's a fairly established way to design protocols.
> 
> I think this should answer the objections to security and extensibility for the purpose of the selection of an nvo3 encapsulation.

The issue with shim headers is that introducing a new one breaks the ability of existing implementations to parse the packet. There are a lot of use cases where just being able to get to the packet data is hugely useful - a NIC steering incoming frames to a VM shouldn’t need to be modified for every use case. Even simple examples like tcpdump/Wireshark benefit from this - it’s really useful to be able to generally get a look at a packet even if it doesn’t understand everything. With both Geneve and GUE this is possible but not with the design that you are describing with VXLAN-GPE.

What you are describing is essentially the same as IPv6 extension headers vs. IPv4 options. Anecdotally, I’ve seen a lot more bugs in implementations of IPv6 extension headers even in very common cases. The reason is that with IPv6, it’s necessary to traverse different shim headers, which themselves have different locations of next protocol field is, etc. At least with IPv4, getting to the data is very simple and consistent.

To make this very concrete: I know that there a number of NIC vendors that have implemented support for VXLAN-GPE offloading already. If a shim header was introduced to add VNI security as you are describing, these offloads would suddenly become useless. That does not seem like a very good definition of extensibility to me.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 14:35:17 -0700

> To make this very concrete: I know that there a number of NIC vendors that have implemented support for VXLAN-GPE offloading already. If a shim header was introduced to add VNI security as you are…

+1. I'd also point out that header chains most likely are harder to
parser than TLVs (as in the IPv6 EH vs. IPv4 options), so this a
counter argument to the objection that Geneve with TLVs would have
more processing complexity than VXLAN-GPE+NSH.

Tom

---

From: Fabio Maino <[email-masked]>
Date: Fri, 29 Jul 2016 16:39:44 -0700

> The issue with shim headers is that introducing a new one breaks the ability of existing implementations to parse the packet. There are a lot of use cases where just being able to get to the packet…

That approach forces your customers to pay upfront for the cost of 
buffering metadata that their devices don't know how to handle, and/or 
they may never use. To implement a new feature, you'll need buffers AND 
logic.

Even if you allocate buffer in advance (before the feature is 
specified), you'll have to spin a new version of the HW to add the logic 
that addresses that new feature.

In this sense VXLAN-GPE allows an implementor to be sensible in 
allocating resources, by using only the buffer space that is actually 
needed by that specific implementation.

I suspect that the result of the pressure of cost on implementations 
will be such that, some HW implementations of Geneve, may implement only 
the basic functions,  restricting to very few, if any, support for 
variable length options. Packets with unknown options will be eventually 
punted to the slow path, and vendors will claim support for the protocol.

This is not very different from how an unknown shim header will be 
handled. Flexibility and HW implementations might be a bit of a unicorn...

> What you are describing is essentially the same as IPv6 extension headers vs. IPv4 options. Anecdotally, I’ve seen a lot more bugs in implementations of IPv6 extension headers even in very common c…

This is actually a good example for the counter argument: many vendors 
couldn't justify the cost of supporting IPv4 option in the fast path, 
and packet with IP options, in many implementations, were punted to the 
slow path. This quickly drove developers to abandon use of IP options in 
their applications, as it was almost a guarantee to send their flows in 
the slow path. Some time too much flexibility is bad (especially when 
cost is an issue).

For IPv6 extension headers, use cases will determine (are determining) 
which ones are worth the extra cost in the fast path and vendors will 
act accordingly. Fittest options will survive...
> To make this very concrete: I know that there a number of NIC vendors that have implemented support for VXLAN-GPE offloading already. If a shim header was introduced to add VNI security as you are…

Similarly to those vendors that might have implemented Geneve in the 
fast path, and want to add support for a new optional extension. With 
the difference that their customers will have paid already for the cost 
of extra buffering even in the first release that had no support for 
that new option.

Fabio

---

From: Jesse Gross <[email-masked]>
Date: Sat, 30 Jul 2016 01:38:17 +0000

> On Jul 29, 2016, at 4:39 PM, Fabio Maino <[email-masked]> wrote:
> 
> That approach forces your customers to pay upfront for the cost of buffering metadata that their devices don't know how to handle, and/or they may never use. To implement a new feature, you'll need buffers AND logic.
> 
> Even if you allocate buffer in advance (before the feature is specified), you'll have to spin a new version of the HW to add the logic that addresses that new feature.
> 
> In this sense VXLAN-GPE allows an implementor to be sensible in allocating resources, by using only the buffer space that is actually needed by that specific implementation.
> 
> I suspect that the result of the pressure of cost on implementations will be such that, some HW implementations of Geneve, may implement only the basic functions,  restricting to very few, if any, support for variable length options. Packets with unknown options will be eventually punted to the slow path, and vendors will claim support for the protocol.
> 
> This is not very different from how an unknown shim header will be handled. Flexibility and HW implementations might be a bit of a unicorn…

I think that you are making some specific assumptions about the types of implementations here. In particular, you are assuming that the only device class is a fixed-function switching ASIC. The comments don’t really apply to software, NICs, or even the more programmable switching ASICs that are starting to come out.

In that specific situation, I agree that it doesn’t make sense to include elements of an implementation that can’t be usefully deployed. However, in cases where the device would need to be respun to include new logic, I doubt that anyone would prematurely include vestigal pieces that would drive up the cost. They would simply implement the support as necessary and the cost would reflect what the device is capable of doing. This is similar to VXLAN-GPE in that regard.

However, I don’t believe that networks consist exclusively of fixed switching ASICs. As I mentioned before, there are many other components that can take advantage of extensibility. I don’t see that it is necessary to deny them those benefits even if some devices may need to be incrementally respun.

> This is actually a good example for the counter argument: many vendors couldn't justify the cost of supporting IPv4 option in the fast path, and packet with IP options, in many implementations, were punted to the slow path. This quickly drove developers to abandon use of IP options in their applications, as it was almost a guarantee to send their flows in the slow path. Some time too much flexibility is bad (especially when cost is an issue).
> 
> For IPv6 extension headers, use cases will determine (are determining) which ones are worth the extra cost in the fast path and vendors will act accordingly. Fittest options will survive…

Yes, I certainly agree that vendors will act according to the needs that they see in the market. I think it’s been said a number of times in the working group that extensibility is a “use it or lose it” type of thing. Since IP options didn’t have immediate important uses when it was first being implemented, there was little sense in handling them. In contrast, TCP options are alive and well these days because there are good uses for them. I know that TCP is generally implemented in software instead of hardware but I can assure you the desire to take shortcuts is plenty strong in the software industry.

In the specific case of Geneve, I gave some information about some of the implementations I’ve personally seen and their capabilities a while back (https://mailarchive.ietf.org/arch/msg/nvo3/HTSXROpKwman1Bd4fAlmmfZF_P8). Since options are the core reason for Geneve existing (it’s the main thing that separates it from VXLAN), there’s been much more awareness that it is important and useful to implement them.

> Similarly to those vendors that might have implemented Geneve in the fast path, and want to add support for a new optional extension. With the difference that their customers will have paid already for the cost of extra buffering even in the first release that had no support for that new option.

In the specific example that I gave (NICs), that is not really true. As new Geneve options are introduced the existing cards will continue to work (GUE also has this property). This is because these offloads are primarily operating on the payload while the option logic is handled in the host CPU. For server-based tunnel termination this ability to continue to offload as the protocol evolves is a significant benefit.