Subject: [tsvwg] UDP options and header-data split (zero copy)

From: Tom Herbert <[email-masked]>
Date: Thu, 15 Jul 2021 08:13:55 -0700

Hello,

Header-data split is a common technique in modern NICs where header
data and payload are DMA'ed into separate sets of receive buffers. The
header data goes to buffers that will be read and processed by CPU,
the data buffers are targeted to application processing (e.g. 4K
buffers for page cache, GPU memory). With sufficient capabilities,
NICs can split header-data at the ULP header to precisely provide the
application just its data. Header-data split is a foundation of zero
copy receive (https://legacy.netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf)

There is no such thing as header-data-trailer split. If a device
receives a packet with protocol trailers and is performing header-data
split, the trailers would be in the data buffers. The CPU may not be
able to read this data, e.g. the data was DMA into GPU memory which is
unreadable by the host CPU. So in the case of UDP options, this means
that when options are in trailers they might nor be visible to be
processed.

In the case of legacy mode in UDP options, protocol trailers are
required. If header-data split is being done then the host can safely
ignore the protocol trailers. In fragment mode,both headers and
trailers containing UDP options may be present in the same packet (the
latter are the options applied after reassembly). This is a problem
since the CPU may not be able to read and process the trailer options
(but will be able to process header options).

The fix for this is don't use protocol trailers in fragment mode. IPv6
fragmentation provides the clue how to do this, just place options
that apply to the reassemble packet after the fragment option in the
first packet of the fragment train. Processing is straightforward,
when a fragment is received, process the options up to the fragment
header. When a packet is reassembled, process the options after the
fragment header in the first packet.

For header-data split, NICs want to split the UDP options into the
header buffers and payload into data buffers. To do this, the device
needs to determine the length of the UDP options. This needs to be a
quick and easy process, for instance no device is going to parse a TLV
chain just to find EOL). This is most efficiently done by reading the
length of the options from a fixed field in the header (e.g. a header
lenght field like in
https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr-01).

Tom

---

From: Joseph Touch <[email-masked]>
Date: Thu, 15 Jul 2021 14:32:40 -0700

Hi, Tom,

> On Jul 15, 2021, at 8:13 AM, Tom Herbert <[email-masked]> wrote:
> 
> Hello,
> 
> There is no such thing as header-data-trailer split.

And there is no way to avoid trailers for non-fragmented packets.

We are designing options for UDP, not options only for fragmentation. 

> If a device

Then that endpoint cannot implement UDP options, because we are not *requiring* fragmentation just to use UDP options.

At that point, either we have systems that can handle both headers (the UDP header never goes away anyway) and trailers, or they don’t support UDP options.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 15 Jul 2021 14:50:43 -0700

> And there is no way to avoid trailers for non-fragmented packets.
Right trailers are only needed in that case. In the case of fragment
using both headers and trailers simultaneously is unnecessary
complexity that provides no benefit. If just headers are used then the
system doesn't need to support trailers for that case and header-data
split continues to work as expected.

Tom

---

From: Paul <[email-masked]>
Date: Fri, 16 Jul 2021 09:12:18 +0200

+1.

>  
> On Jul 15, 2021 at 23:51,  <Tom Herbert (mailto:[email-masked])>  wrote:
>  
>  
>  
>   >  And there is no way to avoid trailers for non-fragmented packets. Right trailers are only needed in that case. In the case of fragment using both headers and trailers simultaneously is unnecessary complexity that provides no benefit. If just headers are used then the system doesn't need to support trailers for that case and header-data split continues to work as expected. Tom 
>
>

---

From: Joseph Touch <[email-masked]>
Date: Fri, 16 Jul 2021 03:45:06 -0700

Hi, Tom,

> On Jul 15, 2021, at 2:50 PM, Tom Herbert <[email-masked]> wrote:
> 
>> ...
> complexity that provides no benefit.

If I put per-fragment as headers and post-fragment as trailers, I can do the following processing:

Outbound:
	A- add/process all (safe) per segment options
	B- if there are any unsafe options or fragmentation is needed for space:
		B.1- split as needed
		B.2- process per-fragment / unsafe options on the resulting fragments
	C- transmit

Inbound does the converse.

In both cases, I can simply skip step B if fragmentation isn’t used/needed.

If all the headers come first for fragments - even the per-segment ones, I either need two versions of step A (one for fragments, one for not) or I need to shuffle the trailer to the header between steps A and B.

—

Your claim that “trailers are ONLY needed…” (emphasis mine) does not consider that “trailer processing IS needed”.

And whenever there are trailers, there are both headers (the current UDP header) and trailers (UDP options), by necessity.

So having two versions of the option location complicates things in the middle of option processing.

The only objection you have raised is that this will hinder processing that’s already hindered. If there’s a slow path here, IMO it ought to be for fragments, not non-fragments.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 17 Jul 2021 14:57:19 -0700

> If all the headers come first for fragments - even the per-segment ones, I either need two versions of step A (one for fragments, one for not) or I need to shuffle the trailer to the header between…

If I'm reading the draft correctly, it is possible that UDP options
could appear both in the headers and trailers of a packet. If that is
correct, then that means an implementation needs to be able to handle
both headers and trailers within the *same* packet.

> The only objection you have raised is that this will hinder processing that’s already hindered. If there’s a slow path here, IMO it ought to be for fragments, not non-fragments.
Protocol trailers are the outlier here. There are existing
implementations and common implementation techniques that will not
handle protocol trailers gracefully (header-data split and RX zero
copy are one example). If protocol trailers are required in all cases
of UDP options, then it's likely that there will be a significant
portion of implementations that either won't support UDP options or
will summarily relegate them to a slow path-- the net effect is that
usability oI UDP options is limited and adoptions of UDP options is
hindered. Conversely, if the requirement is narrowed so that protocol
trailers are only required in legacy mode, then UDP options become
feasible at least in non-legacy mode for those implementations that
can't handle trailers and hence that facilitates wider adoption.

Tom

---

From: Joe Touch <[email-masked]>
Date: Sat, 17 Jul 2021 16:06:05 -0700

> On Jul 17, 2021, at 2:58 PM, Tom Herbert <[email-masked]> wrote:
> 

That is always the case for non-fragmented packets - the only mode that supports transparent use of options that don’t change the payload data with legacy receivers. 

Joe

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 18 Jul 2021 01:13:55 +0200

Without wanting to take a strong position here (one way or the other) due to this beeing far outside my area of expertise, I want to mention that ATM/AAL5 used trailers*, but each ATM cell carrying such a trailer used a bit in its header to signify that the trailer would reside in the last 8 octets of the 53 byte fixed ATM cell. No idea whether there ever was a "zero-copy" decapsulator in use. I naively assume that with its fixed offset of 41 octets into the ATM cell's payload and a simply way o discover this (via a fixed position bit in the ATM cell header) was considerably more regular and simply than what is discussed for UDP.

Regards
	Sebastian

*) Probably well known here, but I have not seen it come up in the discussion

> On Jul 18, 2021, at 01:06, Joe Touch <[email-masked]> wrote:
> 
> 
>

---

From: Tom Herbert <[email-masked]>
Date: Sat, 17 Jul 2021 18:05:54 -0700

> That is always the case for non-fragmented packets - the only mode that supports transparent use of options that don’t change the payload data with legacy receivers.

Perhaps I'm not reading it correctly, but it seems like in legacy mode
all options are in trailers in the surplus area. In the last fragment
case, the packet may look like
UDP-header/UDP-options-per-packet/Frag-data-UDP-options-for-reassembled-packets.
So in that case, there are UDP options in both headers and trailers in
the same packet. If that's true, then a device that can't handle
protocol headers can't handle the case where there are trailing
options in the last fragment. If those options were in the headers
instead, then it could handle that case. Or maybe the idea is that a
receiver can ignore options after the last fragment data since they
are in trailers like legacy mode?

Tom

---

From: Joseph Touch <[email-masked]>
Date: Sat, 17 Jul 2021 19:55:54 -0700

> On Jul 17, 2021, at 6:05 PM, Tom Herbert <[email-masked]> wrote:
> 
> the same packet.

Again, that happens for legacy packets too - the existing UDP header is in the front and options are in the trailer.

> If that's true, then a device that can't handle
> options in the last fragment.

I can’t parse this in a way that makes sense. A device that can’t handle protocol headers can’t do UDP’s existing header. A device that can’t ALSO handle trailers can’t process UDP options for non-fragment packets.

> If those options were in the headers

The idea is that, after reassembly, UDP processing *continues exactly as it would for a non-reassembled packet.

Those trailing options apply to the WHOLE reassembled packet, not just the final fragment.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 19 Jul 2021 07:37:04 -0700

On Sat, Jul 17, 2021 at 7:56 PM Joseph Touch <[email-masked]> wrote:
> I can’t parse this in a way that makes sense. A device that can’t handle protocol headers can’t do UDP’s existing header.

> A device that can’t ALSO handle trailers can’t process UDP options for non-fragment packets.

Joe,

To be specific, a device that can't handle protocol trailers would not
be able to process UDP options in the trailing surplus area.
Non-legacy mode, when UDP length is zero, can carry non-fragmented
packets. In non-legacy mode, such a device would be able to process
UDP options if they were contained in the headers of the packet.

Consider that there are whole classes of devices that terminate UDP
but will not be able to process protocol trailers. For instance many
routers that terminate VXLAN can only append headers for TX and can
only process headers in a limited parsing buffer on RX (these same
routers probably cannot compute the UDP checksum either which was the
motivation for RFC6935 and RFC6936). Can the benefits of UDP options
be enjoyed in an environment with such devices?

There will be deployments, say limited domains, that don't need or
want support for legacy mode and have devices that can't support
protocol trailers anyway. For example, consider a user who wants to
use UDP options but is only interested in fragmentation. Since the
fragment TLV lives in the headers this deployment scenario is
technically viable since protocol trailers are not involved. Is the
user allowed to use UDP options in this case?  If the answer is yes
then what is the correct behavior if someone misconfigures a node to
start sending options in protocol trailers? Does a receiver need to
drop the packet, ignore the trailer options, etc.?

Elaborating on this scenario, suppose a payload compression TLV is
developed. SImilar to fragmentation, this is a type of data transform
so the option must be processed and hence cannot be used in legacy
mode. Again, it seems like the user can safely configure this for
non-fragmented packets in non-legacy mode by doing per packet
compression where the compression TLV lives in the headers. But what
if the user wants to use fragmentation and compression at the same
time (i.e. because compression is more efficient over larger blocks of
data)? Per the current draft, I don't believe they would be able to do
this, since the compression TLV would need to be in protocol trailers
which their devices couldn't process. But if the compression TLV were
to appear in the headers then our user would be able to do this.

Eliminating the reliance on protocol trailers in non-legacy mode would
also reduce packet overhead. In the last fragment TLV, there are two
pointers: one to the start of the fragment data and one to the start
of the protocol trailer. If there are no protocol trailers then the
latter pointer is unneeded and can be removed saving two bytes and
making the format of the two fragment TLVs identical. Furthermore,
since all we need to know the length of the UDP options this could be
indicated by a single byte in a fixed header (perhaps with a
multiplier of four to allow up to 1020 bytes of options like in
draft-herbert-udp-space-hdr). This would reduce the size of the
fragment TLV by two bytes byte with a net savings of one byte, but
more significantly, when non-legacy mode is be used to carry a
non-fragmented packet, like in the compression use case described
above, a fragment TLV is not needed at all and so the packet overhead
is reduced from twelve bytes to one byte.

Furthermore, placing the options that apply to the reassembled packet
in the first fragment instead of the last allows some nice
implementation optimizations. For instance, with the aforementioned
compression TLV we could decompress packets as they are received
instead of waiting to get the final fragment before starting. Also,
there's the possibility that the final fragment might contain a
mandatory option that the receiver doesn't understand and so the
reassembled packet needs to be dropped; if the option is in the first
fragment instead of the last, the action to drop the packet can be
detected much earlier so fragment data received in subsequent packets
does not need to be stored which avoids unnecessarily tying up memory
resources in reassembly with data that is never used for anything.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Mon, 19 Jul 2021 14:32:38 -0700

Hi, Tom,

First, there’s a good reason to not assume we can put the fragment “whole packet” options in the first fragment: size. We currently have no limit on how much space options can use in general. Per-fragment options clearly need to fit inside the fragment with fragment data. But the options on the reassembled packet might be difficult to assume can be contained in the first fragment alone.

Further, you have proposed several scenarios that claim that users may either want or need to not support legacy mode UDP options, but do want to use fragment-based options. That’s not permitted in the current spec and never has been.

Mode is chosen by the transmitter, not the receiver. Support for legacy mode has always been required.

Further, putting a functionality limit (either in length or support for legacy mode) to enhance the performance of some implementations doesn’t seem prudent.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 19 Jul 2021 14:58:37 -0700

On Mon, Jul 19, 2021, 2:32 PM Joseph Touch <[email-masked]> wrote:

> Hi, Tom,
>
> First, there’s a good reason to not assume we can put the fragment “whole
> packet” options in the first fragment: size. We currently have no limit on
> how much space options can use in general. Per-fragment options clearly
> need to fit inside the fragment with fragment data. But the options on the
> reassembled packet might be difficult to assume can be contained in the
> first fragment alone.
>
> Further, you have proposed several scenarios that claim that users may
> either want or need to not support legacy mode UDP options, but do want to
> use fragment-based options. That’s not permitted in the current spec and
> never has been.
>

That limits the usability and hence adoption of the protocol.

> Further, putting a functionality limit (either in length or support for
> legacy mode) to enhance the performance of some implementations doesn’t
> seem prudent.

---

From: Joseph Touch <[email-masked]>
Date: Mon, 19 Jul 2021 15:02:42 -0700

> On Jul 19, 2021, at 2:58 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> On Mon, Jul 19, 2021, 2:32 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, Tom,
> 

That may be true, but it’s never been on the table.

The whole point is to support legacy mode. If we don’t, we could simply create a new protocol from scratch, which would limit the adoption of the protocol even further.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 19 Jul 2021 15:10:42 -0700

On Mon, Jul 19, 2021, 3:02 PM Joseph Touch <[email-masked]> wrote:

> The whole point is to support legacy mode. If we don’t, we could simply
> create a new protocol from scratch, which would limit the adoption of the
> protocol even further.
>

Please check the list archives, I have been pointing out the problems and
deployment impediments with protocol trailers and for a while. Neither,
AFAIK, has consensus been established on this. Allowing users to using UDP
options with requiring support for protocol headers seems entirely
reasonable to me.

---

From: Tom Herbert <[email-masked]>
Date: Mon, 19 Jul 2021 15:16:58 -0700

On Mon, Jul 19, 2021, 3:10 PM Tom Herbert <[email-masked]> wrote:

>>> First, there’s a good reason to not assume we can put the fragment
>>> “whole packet” options in the first fragment: size. We currently have no
>>> limit on how much space options can use in general. Per-fragment options
>>> clearly need to fit inside the fragment with fragment data. But the options
>>> on the reassembled packet might be difficult to assume can be contained in
>>> the first fragment alone.
>>>

Without requiring support that is.

---

From: Joe Touch <[email-masked]>
Date: Mon, 19 Jul 2021 17:58:24 -0700

> On Jul 19, 2021, at 3:11 PM, Tom Herbert <[email-masked]> wrote:
> 
> Please check the list archives, I have been pointing out the problems and deployment impediments with protocol trailers and for a while.

You have. I though you were limiting your comments to fragments at this time. 

> Neither, AFAIK, has consensus been established on this.

It has never been consensus to limit use to only fragment mode. In fact, that mode has been argued as optional most recently. 

> Allowing users to using UDP options with requiring support for protocol headers seems entirely reasonable to me.

It’s a non starter AFAICT. 

Others? 

Joey

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Jul 2021 07:49:06 -0700

> It has never been consensus to limit use to only fragment mode. In fact, that mode has been argued as optional most recently.
Limiting use to fragment mode is not what I am suggesting. There are
applications that need legacy mode and those that don't (DNS on the
Internet needs it, UDP tunnels in a datacenter don't). There are
implementations that support protocol trailers and those that don't
(like the routers I mentioned). Protocol trailers are technically
required in legacy mode, but there is no technical rationale why
non-legacy mode needs trailers. If trailers are required in all use
cases, then devices that don't support them can't use UDP options even
if they're only interested in non-legacy mode uses. For instance, UDP
options is a means to bring extensibility to otherwise non-extensible
UDP encapsulation protocols and augmenting VXLAN with a fragmentation
capability is compelling. But if protocol trailers are a forced
requirement, then I believe that would preclude us from ever using UDP
options with UDP encapsulation-- that would be unfortunate.

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Fri, 30 Jul 2021 20:11:08 -0700

On Thu, Jul 15, 2021 at 2:51 PM Tom Herbert wrote:
> On Thu, Jul 15, 2021 at 2:32 PM Joseph Touch wrote:

Let me add my voice to this. The specification of FRAG in -13 is
unnecessarily complex. We do not need both headers (for per-frag
options) and trailers (for per-datagram options). That format, as
Tom correctly notes, is an impediment to checksum offload for
encapsulated protocols. The generality of allowing any option to
be encoded as per-datagram or per-fragment is a level of flexibility
that is comparable to the flexibility offered by independently
steerable front wheels: it is neither necessary nor safe, as very
few options can serve in both capacities, and there are other ways to
deal with those.

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Fri, 30 Jul 2021 20:35:49 -0700

Hi, Tom,

> On Jul 20, 2021, at 7:49 AM, Tom Herbert <[email-masked]> wrote:
> 

I don’t follow this line of logic.

You claim that you’re not limiting use to frag-only, but then talk about “if trailers are required”.

There’s no IF. They are.

If an implementation can’t deal with trailers, it can’t support UDP options - the core set is “all or none”, not “pick what you want”.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Fri, 30 Jul 2021 20:40:08 -0700

Hi, Mike,

Trailers make a reassembled packet look *exactly* like one that isn’t reassembled. That has to be handled, regardless.

Having two *different* places for per-packet options, depending on whether they’re reassembled or not, would be the unnecessary complexity.

Besides, we’ve moved the FRAG option to the front of the list so hardware can find it without processing too much of a TLV chain. Your alternative, i.e., of using FRAG to split per-fragment vs post-fragment options, would mean that the FRAG option would *float* into the TLV chain further, complicating and slowing down reassembly.

I think we need to come to terms with the fact that UDP options means trailers. Once you accept that as a given, putting per-packet options at the end is the obvious place.

Joe

> On Jul 30, 2021, at 8:11 PM, C. M. Heard <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Sat, 31 Jul 2021 07:53:05 -0700

Joe,

That is an opinion, not an established fact. If you want to convince
us that trailers are a requirement in all use cases, specifically
non-legacy mode, then please provide at least one example of options
that could not be correctly conveyed in protocol headers. Given that
we have many examples of successful protocols that have protocol
options in headers, IP options, TCP options, and IPv6 which provides
the example of how per fragment options and options for the
reassembled packet, I am dubious the need for trailers can be proven.

> If an implementation can’t deal with trailers, it can’t support UDP options - the core set is “all or none”, not “pick what you want”.

If an implementation can't deal with trailers, then that means that
users of those implementations can't use UDP options. If users can use
the protocol we're defining then what is the point of bringing it to
IETF?

Tom

---

From: Joseph Touch <[email-masked]>
Date: Sat, 31 Jul 2021 09:18:11 -0700

Hi, Tom,

> On Jul 31, 2021, at 7:53 AM, Tom Herbert <[email-masked]> wrote:
> 
> On Fri, Jul 30, 2021 at 8:35 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi, Tom,
>> 
>>> Limiting use to fragment mode is not what I am suggesting.
>>>  ...If trailers are required in all use
>>> if they're only interested in non-legacy mode uses
>>> ….
>> 
> That is an opinion, not an established fact.

Trailers have been shown to be the only way to support legacy mode use of UDP options. Nobody else has ever suggested or accepted that legacy mode is an optional part of this spec.

> If you want to convince
> that could not be correctly conveyed in protocol headers.

Legacy mode.

> Given that

We’ve already done so for legacy mode.

If you’re looking for a successful version of trailers, perhaps Ethernet will convince you? How about HTTP since 1.1? HLDC and PPP?

> users of those implementations can't use UDP options.

Again, that IS the point.

> If users can[’t] use

(I’m presuming you meant “can’t” above)

We have two types of legacy users:
	- legacy UDP endpoints in general
	- updatable endpoints with legacy UDP acceleration hardware

Your claim is that we can’t support both groups, especially if they interact.Even if we were to accept your claim, we would always choose to support legacy endpoints in general. If we have to sacrifice, we should sacrifice performance to maintain interoperability.

That decision is not unique to UDP or UDP options; it is the approach used throughout the IETF.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 31 Jul 2021 10:17:00 -0700

> We’ve already done so for legacy mode.
But not for non-legacy mode.

> If you’re looking for a successful version of trailers, perhaps Ethernet will convince you? How about HTTP since 1.1? HLDC and PPP?
No, it's not convincing. HTTP is an application layer protocol.
Ethernet, HDLC, and PPP are layer 2 protocols that have headers and
use trailers for only FCS and CRC for the benefit of hardware
implementation. You could also add ESP to this list which again
primarily uses a header and puts authentication data. These are not
comparable to UDP options which is an open ended list of TLVs which is
more likely to be processed in software. For network layer and
transport layer protocols, protocol headers are the longstanding
convention. Implementations have assumed this convention, and
introducing a protocol that goes against this convention will have
adverse effects. The effects could be loss of performance or
functionality, or outright complete inability to support the protocol.

> Again, that IS the point.

I don't see how that isn't arbitrarily and knowingly eliminating the
number of users and use cases for the protocol.

> - updatable endpoints with legacy UDP acceleration hardware
What does "updatable" mean. If there is some piece of hardware that
doesn't support protocol trailers, it's not like we can just call up
the vendor and they'll fix it in a month. Fixing hardware takes years,
and that assumes the vendors see any business value to invest in the
fix which only exists if UDP options use becomes prevalent and support
is being demanded by customers.

> Your claim is that we can’t support both groups, especially if they interact.Even if we were to accept your claim, we would always choose to support legacy endpoints in general. If we have to sacri…
To the contrary, my claim is that we need to support as many users as
possible. Avoiding a requirement for protocol trailers aligns with
that goal, requiring protocol trailers opposes that goal and limits
the number of users that can use the protocol.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Sat, 31 Jul 2021 10:32:53 -0700

> On Jul 31, 2021, at 10:17 AM, Tom Herbert <[email-masked]> wrote:
> 

They’re not separable. If you can’t support non-legacy, you don’t support UDP options.

This has never been a “pick and choose” spec.

> implementation.

It doesn’t benefit or hinder hardware implementation. It affects queuing delay in either case, but that’s not an issue for an endpoint protocol like UDP.

> You could also add ESP to this list which again
> convention.

And if we were designing one from scratch, I would agree.

> Implementations have assumed this convention, and
> adverse effects.

We don’t have a choice for legacy mode.

> The effects could be loss of performance or

Performance, yes. But by your own admission, the TLV is likely to be processed in software (wherever that is) anyway. 

> number of users and use cases for the protocol.

The result of a “pick and choose” protocol is the same, if not worse.

Again, like most of the IETF, we’re focusing on ubiquity over performance.

> What does "updatable" mean.

I was giving your approach the benefit of the doubt that the software could be updated to support legacy acceleration hardware.

If that can’t happen, then UDP options can’t be supported because they aren’t right now.

> If there is some piece of hardware that
> the vendor and they'll fix it in a month.

Right. But you CAN just have it push the whole reassembled packet with trailers to the OS and let the OS software handle the rest - just like it would need to for legacy mode anyway.

> Fixing hardware takes years,

Agreed - which is even LESS reason to try to optimize for today’s hardware.

> the number of users that can use the protocol.

How does leaving legacy endpoints out “support as many users as possible” vs. merely using software to complement hardware acceleration, the latter which would support all users?

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 31 Jul 2021 11:27:31 -0700

Joe,

You seem to be missing my point. There are implementations that
terminate UDP and CANNOT process protocol trailers; this is not just
an inconvenient performance degradation. Yes, that means that they
can't use legacy mode, but I don't see the technical argument that UDP
couldn't be productively supported by these implementations if UDP
options are all contained in headers. I do not believe this is an
aberration, we know for instance there are HW routers terminating UDP
tunnels, and it would be a delusion to believe these implementations
could be fixed to support protocol trailers in a finite time. So if
protocol trailers are a forced requirement, even in non-legacy mode
which is what tunnels would use anyway since they're configured, it
seems like we're excluding use cases such as UDP tunnels where UDP
options might be compelling. Given this, what is your answer to
someone who wants to use UDP options with UDP tunnels?

Tom

---

From: Joseph Touch <[email-masked]>
Date: Sat, 31 Jul 2021 11:33:59 -0700

Hi, Tom,

> On Jul 31, 2021, at 11:27 AM, Tom Herbert <[email-masked]> wrote:
> 
> Joe,
> 
> an inconvenient performance degradation.

If that is the case, then they MUST NOT try to support UDP options. See below.

> Yes, that means that they
> options are all contained in headers.

Again, support for legacy receivers has always been a requirement of a solution.

> I do not believe this is an

There’s no way that endpoint can ensure it never sees non-fragment UDP options, so it cannot support UDP options.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 31 Jul 2021 11:57:35 -0700

> If that is the case, then they MUST NOT try to support UDP options. See below.
If UDP options cannot be used with UDP tunnel protocols, one of the
biggest use cases for UDP, then I believe the protocol is
fundamentally flawed. At the very least this needs to be documented in
the draft, and if UDP options are only meant to be used for certain
use cases that should be contained in an applicability statement in
the draft.

> There’s no way that endpoint can ensure it never sees non-fragment UDP options, so it cannot support UDP options.
For a tunnel, use of UDP options would just be another configuration
parameter at the end points; we don't need negotiation to send
non-legacy packets. If there's a mismatch in endpoint configuration
then the operator simply detects it and fixes it like any other tunnel
configuration problem.

---

From: Joe Touch <[email-masked]>
Date: Sat, 31 Jul 2021 12:42:13 -0700

> On Jul 31, 2021, at 11:58 AM, Tom Herbert <[email-masked]> wrote:
> 

Ok, the simply don’t set any per packet options and I’ll be fine.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 31 Jul 2021 13:42:45 -0700

> Ok, the simply don’t set any per packet options and I’ll be fine.
I think you might mean the other way around. Per packet options are
already defined in the draft to be in the headers, including the
fragment option, so there's no issue concerning protocol trailers.
There would only be a problem from options that apply to the
reassembled packet which would be in trailers of the last fragment.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Sat, 31 Jul 2021 14:08:13 -0700

Right - per fragment options would be OK, not per reassembled datagram.

Would that suffice?

Joe

> On Jul 31, 2021, at 1:42 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Joseph Touch <[email-masked]>
Date: Sat, 31 Jul 2021 14:17:49 -0700

FWIW, there’s another option here, which could be considered unnecessarily complex if there’s no need for per-reassembled datagram options for most UDP tunnels….

----

Right now, the terminal fragment has a header as follows:

                   +--------+--------+--------+--------+
                   | Kind=4 | Len=12 |   Frag. Start   |
                   +--------+--------+--------+--------+
                   |           Identification          |
                   +--------+--------+--------+--------+
                   |  Frag. Offset   |    Frag. End    |
                   +--------+--------+--------+--------+

                Figure 12   UDP terminal FRAG option format
As to whether to put the per-reassembled options before or after the terminal fragment, we could let the user decide (again, consistent with the idea of UDP), as follows.

The code cost is small; this way an implementation could decide when it can use a single DMA or no. The only hazard is that the transmitter decides - but for the UDP tunnel case, given both sides already need configuration, it should be simple enough to encourage transmitters to use the preferred case.

----

We could consider “Frag. End” to really just point to the start of the per-reassembled datagram options, so let’s instead call that field “Reassembled datagram option start”, or RDOS for short. Let’s assume there’s a “RDOE” (end pointer) as well, which we’ll compute.

If so, then we could allow those options to be either before the start of the fragment (where RDOS < Frag Start) or at the end of the packet (where RDOS > Frag Start).

If we allow that, then the code needs a condition to decide where the end of the packet is:

	if (RDOS < frag start) 
		then  // single DMA case
			frag end = IP packet end
			RDOE = frag start - 1
		else  // receive processing same as for non-terminals
			frag end = RDOS-1
			RDOE = IP packet end

At that point, the per-reassembled options are from RDOS to RDOE and the fragment is from frag start to frag end.

----

> On Jul 31, 2021, at 2:08 PM, Joseph Touch <[email-masked]> wrote:
>