Subject: [nvo3-dt-encap] NVO3 Design Team - 1/3/2017 Discussion Summary

From: Pankaj Garg <[email-masked]>
Date: Tue, 3 Jan 2017 10:53:27 -0800

Hi All,

Thank you for another productive discussion today. I am summarizing the
discussion. Please feel free to add if I missed anything.

Participants: 
- Erik Nordmark (Arista) 
- Michael Smith (Cisco) 
- Uri Elzur (Intel) 
- Sami Boutros (VMware) 
- Tal Mizrahi (Marvell) 
- David Mozes (Mellanox)
- Pankaj Garg (Microsoft)

1. We first went through Tal's slide deck on comparison of current
encapsulation proposals and discussed various pros and cons. In this
discussion, we focused on various important aspects of encapsulation protocol
as described in our summary on 12/12.

- Based on the discussion, we felt that Geneve is most suitable as a starting
  point for proposed standard for network virtualization, as described below.

- We discussed whether VNI should be in base header or in extensions and
  whether it should be 24-bit or 32-bit. Based upon the discussion, design
  team feels that VNI is critical information for network virtualization and
  must be present in all packets. Design team also feels that 24-bit VNI
  matches the existing widely used encapsulation format i.e. VxLAN and NVGE
  and hence more suitable to use going forward.

- We discussed using NSH with VxLAN-GPE but given that NSH is targeted at
  service chaining and contains service chaining information, it is less
  suitable for network virtualization. The other downside for VxLAN-GPE was
  lack of header length in VxLAN-GPE and hence makes skipping over the headers
  to process inner payload more difficult.

- We discussed TLV vs Bit-fields style extension and it was deemed that
  parsing both TLV and bit-fields is expensive and while bit-fields may be
  slightly better, it is also restrictive.

- We briefly discussed multi-vendor NVE case and need to allow vendors to put
  their own extensions in the NV header. This is possible with TLVs.

- We also discussed presence of C bit in Geneve and how it is helpful to allow
  receiver NVE to easily decide whether to process options or not. We took
  example of a UUID based packet trace and how an optional extension such as
  that can be ignored by receiver NVE and thus make it easy for NVE to skip
  over the options.

- Based on this discussion, it was concluded that Geneve seems to be a good
  starting point.

2. We next focused our discussion on Geneve and how we can address
shortcomings to make it more suitable to hardware and yet provide the
flexibility for software.

- 260 byte vs shorter header: Geneve allows up to 260 bytes in options length,
  This might be too much for hardware middleboxes that simply wants to skip
  over and access inner packets. We discussed restricting the header but
  agreed that we should not limit future potential of the protocol. We
  discussed how control plane can limit option size in a deployment based on
  hardware capability. We also agreed that we should add some text in the
  document to indicate what is minimum a compliant implementaiton MUST
  support. For example, we would propose text such as, an implementation of
  Geneve MUST support at least 64-bytes in options length.

- We next discussed TLV processing overhead and how it may not be easy to be
  processed in TCAM. There may be some critical information such as secure
  hash that must get processed at lowest latency. We discussed how we can
  require TLV 0 to be the first TLV in the options. We can use a bit in Geneve
  header to indicate presence of this TLV and define format of this TLV that
  is hardware friendly. David, Tal and Michael are going to think about if
  this would make hardware processing easier and right format for such a TLV.

- We understand that such a fixed TLV will not meet everyone's requirements,
  however it would provide necessary room to allow hardware friendly
  extensions without limiting flexibility which is required for many other
  scenarios.

- We discussed split-NVE case and how to support carrying extensions over
  other protocols such as 802.1q. There are two options on the table, we can
  either allocate a ETYPE for Geneve and allow carrying of it in 802.1q or
  define a shim header.

This is around when we concluded the discussion. Please let me know if I
missed something.

Thanks
Pankaj (on behalf of NVO3 Encap Design Team)

---

From: Rajeev Manur <[email-masked]>
Date: Tue, 3 Jan 2017 11:40:31 -0800

Happy 2017!

Sorry I missed today's meeting. Thanks for the detailed notes. Please see
my comments inline...

>   and hence more suitable to use going forward.
[Rajeev] Agree with you guys that keeping the std VNID field to 24-bits
seems reasonable. It would be good to allow extension of VNID using some
TLV/optional hdr.

>   to process inner payload more difficult.
[Rajeev] Did we discuss possibility of NSH with GENEVE? Wondering if it is
possible to normalize all metadata encap under one std format.

>   slightly better, it is also restrictive.
[Rajeev] Did we consider leaving the semantics of bit-fields to be defined
by the control plane, so that every deployment gets to define it to suit
their requirement, would that eliminate the "restrictive"ness mentioned
above. Also, the last bit in the bit field can indicate if there are TLVs
pressent from that point on in the packet. Would a mixed approach like
above be better?

>   Geneve MUST support at least 64-bytes in options length.
[Rajeev] Agreed.

> [Rajeev] Could you please help me understand what we are trying to solve
here so that I can also think about the HW friendly formats.

> _______________________________________________
> nvo3-dt-encap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3-dt-encap
>

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 3 Jan 2017 19:13:42 -0800

Inline.
> seems reasonable. It would be good to allow extension of VNID using some
PG: Yes, TLV can be used to have a larger VNI.
> possible to normalize all metadata encap under one std format.
PG: We did discuss it but the fact that NSH has extra header and
contains service chaining bits makes it not as suitable for network
virtualization.
> pressent from that point on in the packet. Would a mixed approach like
PG: This would make it very hard to deploy and ensure everything works
correctly. At least looking at packet on the wire, one must be able to
decode what the packet contains. Also to support vendor specific
extensions, TLV deemed more appropriate.
> here so that I can also think about the HW friendly formats.
PG: Let us consider critical metadata such as secure hash, that must be
validated for each packet.  Or take an example of a packet tracing
feature that uses a single bit and if it is set, packet should be traced
by each device in the path. To ensure each device doesn't have to do TLV
processing, we can put such critical information in the TLV0 which is
first in the list and indicate presence of TLV0 in main Geneve header.
This would allow quick lookup of the packet. What we discussed was, if
we propose TLV0 to be first TLV, then what would be ideal format for
TLV0 such that it can be processed in TCAM and yet has some flexibility.
I think Tal and David being hardware guys can elaborate this further.

---

From: Rajeev Manur <[email-masked]>
Date: Tue, 3 Jan 2017 22:12:34 -0800

Pl see inline...

On Tue, Jan 3, 2017 at 7:13 PM, Pankaj Garg <[email-masked]> wrote:

> > >   team feels that VNI is critical information for network
> virtualization
> > > and
[Rajeev] Assuming you mean path hdr is extra and not required for
virtualization usecases. We could look into the possibility of making path
hdr optional with a bit in base-hdr indicating that? We could potentially
use a new MD_TYPE value to indicate the lack of path-hdr. I am not sure
which service chaining bit are you refering to.

> extensions, TLV deemed more appropriate.
[Rajeev] Not sure I understand. I am just talking about the possibility of
hybrid approach where we have a bit field with the last bit in the bit
field can represent a variable length TLV. This way we may be able to get
best of both, bit-field and TLVs.

> >
> > >
> > > - We briefly discussed multi-vendor NVE case and need to allow vendors
> to
> > > put
> > > - We also discussed presence of C bit in Geneve and how it is helpful
> to
> > > allow
> > >   example of a UUID based packet trace and how an optional extension
> such
> > > as
> > >   discussed how control plane can limit option size in a deployment
> based
> > > on
> > > - We next discussed TLV processing overhead and how it may not be easy
> to
> > > be
> > >   this would make hardware processing easier and right format for such
> a
> > > TLV.
> > >
> > >   other protocols such as 802.1q. There are two options on the table,
> we
> > > can

---

From: Pankaj Garg <[email-masked]>
Date: Wed, 4 Jan 2017 07:58:06 -0800

On Tue, Jan 03, 2017 at 10:12:34PM -0800, Rajeev Manur wrote:
> > > >   suitable for network virtualization. The other downside for
> > VxLAN-GPE was > > > >   lack of header length in VxLAN-GPE and hence makes skipping over the
> > > > headers
PG: I mean both headers, the NSH header and path header are extra. It also
complicates other processing, e.g. if we use NSH, NextProtocol in Geneve would
need to be set to NSH. This would break NIC offload support and require extra
lookup to get to actual inner header. We can't use options length in Geneve to
encompass NSH header either. At that point, I am not sure why we should use
Geneve. We should just use VXLAN-GPE+NSH and as discussed already, it has
limitations.
> best of both, bit-field and TLVs.
PG: Thanks Rajeev, now I understand.  We did discuss this in our meeting. A
concern raised on this in the meeting was creation of two extension mechanism
i.e. a bit-field based and a TLV based. Everyone on the call felt that we
should only only have one extension mechanism i.e. TLV based. We can discuss
it again next week, but I feel a TLV0 based extension header that is TCAM
processing friendly, followed by regular TLVs is more suitable than adding
both bit-fields based and TLV based extensions.

---

From: David Mozes <[email-masked]>
Date: Tue, 10 Jan 2017 15:29:43 +0000

Hi * ,
TLV 0 :  It will help to the HW implementation  if we have a well know TLV that is the first one, qualify  by Geneva reserve bit .
To be more specific , it will help on these conditions: 
1) The HW need to  parse  just TLV 0 and don't need to search on other TLV's on the packet.
2) If this TLV is always exist with a fix size on case we have  Geneva options on the packet.
 With these assumptions the definition of this TLV 0 is flexible.
  
If this is not the case and the HW need to go and search for other Geneve  TLV's on the packet TLV 0 has to have the same format of the other  Geneve TLV.
If   will help to the HW on case some HW  need just to parse the TLV-0,althow  there are other TLV on the packet.

NSH over Geneve :  Geneve has ethertype option 

-----Original Message-----
From: nvo3-dt-encap [mailto:[email-masked]] On Behalf Of Pankaj Garg
Sent: Wednesday, January 04, 2017 5:58 PM
To: Rajeev Manur <[email-masked]>
Cc: [email-masked]
Subject: Re: [nvo3-dt-encap] NVO3 Design Team - 1/3/2017 Discussion Summary

On Tue, Jan 03, 2017 at 10:12:34PM -0800, Rajeev Manur wrote:
> > > Sorry I missed today's meeting. Thanks for the detailed notes. 
> > > Please see my comments inline...
> > >
> > > > Thank you for another productive discussion today. I am 
> > > > summarizing the discussion. Please feel free to add if I missed anything.
> > > >
> > > > Participants:
> > > > - Erik Nordmark (Arista)
> > > > - Michael Smith (Cisco)
> > > > - Uri Elzur (Intel)
> > > > - Sami Boutros (VMware)
> > > > - Tal Mizrahi (Marvell)
> > > > - David Mozes (Mellanox)
> > > > 1. We first went through Tal's slide deck on comparison of 
> > > > current encapsulation proposals and discussed various pros and 
> > > > cons. In this discussion, we focused on various important 
> > > > aspects of encapsulation protocol as described in our summary on 
> > > > 12/12.
> > > >
> > > > - We discussed whether VNI should be in base header or in 
> > > > extensions
> > and
> > > >   whether it should be 24-bit or 32-bit. Based upon the 
> > > > discussion,
> > design
> > > >   must be present in all packets. Design team also feels that 
> > > > 24-bit
> > VNI
> > > >   matches the existing widely used encapsulation format i.e. 
> > > > VxLAN and
> > NVGE
> > > [Rajeev] Agree with you guys that keeping the std VNID field to 
> > > 24-bits seems reasonable. It would be good to allow extension of 
> > > VNID using some TLV/optional hdr.
> > > > - We discussed using NSH with VxLAN-GPE but given that NSH is 
> > > > targeted
> > at
> > > >   service chaining and contains service chaining information, it 
> > > > is
> > less
> > > [Rajeev] Did we discuss possibility of NSH with GENEVE? Wondering 
> > > if it
> > is
> virtualization usecases. We could look into the possibility of making 
> path hdr optional with a bit in base-hdr indicating that? We could 
> potentially use a new MD_TYPE value to indicate the lack of path-hdr. 
> I am not sure which service chaining bit are you refering to.
PG: I mean both headers, the NSH header and path header are extra. It also complicates other processing, e.g. if we use NSH, NextProtocol in Geneve would need to be set to NSH. This would break NIC offload support and require extra lookup to get to actual inner header. We can't use options length in Geneve to encompass NSH header either. At that point, I am not sure why we should use Geneve. We should just use VXLAN-GPE+NSH and as discussed already, it has limitations.
> > > >   parsing both TLV and bit-fields is expensive and while 
> > > > bit-fields
> > may be
> > > by the control plane, so that every deployment gets to define it 
> > > to suit their requirement, would that eliminate the 
> > > "restrictive"ness mentioned above. Also, the last bit in the bit 
> > > field can indicate if there are TLVs pressent from that point on 
> > > in the packet. Would a mixed approach like above be better?
> > PG: This would make it very hard to deploy and ensure everything 
> > works correctly. At least looking at packet on the wire, one must be 
> > able to decode what the packet contains. Also to support vendor 
> > specific extensions, TLV deemed more appropriate.
> >
> [Rajeev] Not sure I understand. I am just talking about the 
> possibility of hybrid approach where we have a bit field with the last 
> bit in the bit field can represent a variable length TLV. This way we 
> may be able to get best of both, bit-field and TLVs.
PG: Thanks Rajeev, now I understand.  We did discuss this in our meeting. A concern raised on this in the meeting was creation of two extension mechanism i.e. a bit-field based and a TLV based. Everyone on the call felt that we should only only have one extension mechanism i.e. TLV based. We can discuss it again next week, but I feel a TLV0 based extension header that is TCAM processing friendly, followed by regular TLVs is more suitable than adding both bit-fields based and TLV based extensions.
> > > >   receiver NVE to easily decide whether to process options or 
> > > > not. We
> > took
> > > >   that can be ignored by receiver NVE and thus make it easy for 
> > > > NVE to
> > skip
> > > >   over the options.
> > > >
> > > > - Based on this discussion, it was concluded that Geneve seems 
> > > > to be a
> > good
> > > >   starting point.
> > > >
> > > > 2. We next focused our discussion on Geneve and how we can 
> > > > address shortcomings to make it more suitable to hardware and 
> > > > yet provide the flexibility for software.
> > > >
> > > >   This might be too much for hardware middleboxes that simply 
> > > > wants to
> > skip
> > > >   over and access inner packets. We discussed restricting the 
> > > > header
> > but
> > > >   hardware capability. We also agreed that we should add some 
> > > > text in
> > the
> > > >   processed in TCAM. There may be some critical information such 
> > > > as
> > secure
> > > >   hash that must get processed at lowest latency. We discussed 
> > > > how we
> > can
> > > >   is hardware friendly. David, Tal and Michael are going to 
> > > > think
> > about if
> > > > [Rajeev] Could you please help me understand what we are trying 
> > > > to
> > solve
> > PG: Let us consider critical metadata such as secure hash, that must 
> > be validated for each packet.  Or take an example of a packet 
> > tracing feature that uses a single bit and if it is set, packet 
> > should be traced by each device in the path. To ensure each device 
> > doesn't have to do TLV processing, we can put such critical 
> > information in the TLV0 which is first in the list and indicate presence of TLV0 in main Geneve header.
> > This would allow quick lookup of the packet. What we discussed was, 
> > if we propose TLV0 to be first TLV, then what would be ideal format 
> > for
> > > >   extensions without limiting flexibility which is required for 
> > > > many
> > other
> > > >   scenarios.
> > > >
> > > > - We discussed split-NVE case and how to support carrying 
> > > > extensions
> > over
> > > > This is around when we concluded the discussion. Please let me 
> > > > know if
> > I
> > > > missed something.
> > > >
> > > > Thanks