Subject: Re: [Stackevo-discuss] [Stackevo] draft-byrne-opsec-udp-advisory

From: Brian Trammell <[email-masked]>
Date: Fri, 24 Jul 2015 15:00:40 +0200

[...and now copying the newly-created stackevo-discuss list, just created as a place for public discussion on IAB stack evolution program related topics...]

> On 24 Jul 2015, at 07:04, Ca By <[email-masked]> wrote:
> 
> On Thursday, July 23, 2015, Brian Trammell <[email-masked]> wrote:
> hi Cameron, all,

<snip>

>> Cool. So, if we assume that we can find a way to quickly and easily distinguish bad old UDP ("all" of which is reflected crap, for some high value of all) from UDP-as-layer-boundary-reinforcement -- a well-selected magic number being the most obvious way (to me) to do this statelessly.
>> 
> 
> I would simply call it udp2 with a new transport protocol number.

If we're going to do that we might as well fix other things with UDP. I see this as an excellent way forward for the second stage of this transition.

> But I imagine you want to preserve some way to blast through legacy nats, which is not a good strategic goal / constraint

It'd be nice, yes, but it's not the main concern.

On the NAT question, I'd have to see data to convince me that the proportion -- internet-wide -- of connections from end-user / that are NATted (in a way that a *new* UDP that has port numbers in the same place in the header won't make it through, which is I assume today all NAT devices). V6 takes a lot of address pressure away, true, and careful engineering of V6-enabled CPE can reduce NATting. But V6 only access connectivity to services that are V4 only itself requires NAT. So the slope points downward but I don't see NAT being an insignificant part of the deployed architecture for a long time. We want something that works tomorrow, not someday, with negligible risk to connectivity. Shiny new features that break the delivery of cat videos they promise to improve get switched off.

More important is that basically every programming environment on every internet connected platform can give applications access to UDP/17. It will take a long time for that to be the case with UDP2/144 (or whatever number this protocol would be assigned). I hope that the curve for getting kernel support for these things is faster than it was with e.g. SCTP: endpoint operating system maintenance has matured a lot in the past two decades. But it's still a long curve.

>> So, there are broadly two things I mean when I say UDP:
>> 
>> (1) Packets with a protocol identifier of 17 on them. This is what you want to filter on because most of them are crap.
>> 
>> 
> Yes
> 
>> (2) What happens when you open a SOCK_DGRAM socket. This service is basically "nonprivileged userspace raw sockets". When we're talking about making it possible to deploy new transport protocols, the "userspace" is IMO crucial here (and the "nonprivileged" equally so if you're talking about the new world where everything happens on devices where the user has no root).
>> 
>> 
> Udp2
> 
>> If we had a way to get this second thing without throwing a UDP header on it, that'd be great, but whatever it is you need port numbers on it, more because you can have multiple userspace processes on each eventual endpoint than you need to get around NAT.
> 
> 
> I am optimistic about IPv6 deployment and the removal of nats and middle boxes.

I'm far more optimistic about the former than the latter. Just because one enables the other doesn't mean that it will.

>  This is tied to LTE capacity and latency improvements outpacing middle box utility and capacity.
> I find it strange that the ietf is trying now to talk to middle boxes while they are waning. YMMV.

For my part, I see this effort as an attempt to make interactions with middleboxes *explicit* as opposed to implicit; this can happen even as more paths get more transparent, and the diagnostics enabled by this layer can accelerate this trend.

>>> > I can only think of 2 paths forward.  Remove the noise from UDP (aka, upgrade the internet to stop DRDoS [home routers with SSDP, open DNS and NTP, booter services...]) or stop assuming your UDP packets will be transported at scale (aka, the ietf needs to not load new things into UDP and accept the mess that grew out of UDP... learning from it in new transport protocols that are not transport protocol 17).
>>> 
>>> I see the use of UDP here as a transition technology: that is, something that will be deployed for the next twenty years or so. The eventual goal (at least mine) is to be able to do all of this over a new protocol. The stack looks something like this:
>> 
>>    |  transport services API (TAPS)  |
>>    +---------------------------------+
>>    | shiny new transport protocol(s) |  encrypted
>> ==========================================
>>    |   explicit cooperation layer    |  cleartext
>>    +-----------------+               |
>>    |       UDP       |               |
>>    +----------+------+---------------+
>>    |    IPv4  |           IPv6       |
>>    +----------+----------------------+
>> 
>>     ------------ time -------------->
>> 
>> > The former is impossible and outside the capabilities of anyone reading this email.
>> 
>> I'd submit that if a good-UDP / bad-UDP discriminator, as well as return routability proof and state exhaustion resistance, can be designed into future good-UDP protocols, that's a start.
> 
> But it does not solve change anything in the I-D at hand, as the I-D is tactical and operational in nature

It would, were it deployed: there would be an easy way to determine reflected crap from nearly-certainly-not-reflected not-crap. The I-D at hand is a good collection of observations about why UDP might not be appropriate here. It is not clear that generalizing these observations to the wider Internet is useful guidance in and of itself ("deprecate UDP"), though it can be input to efforts to improve the situation on UDP ("reduce reflected crap until you can deprecate UDP").

>> > The latter is inconvenient for some yet-to-be-standardized or broadly deployed technologies, and it is within the circle of influence here at the IETF to wind down UDP.
>> 
>> This is tantamount to saying the future of the Internet is over TCP, which is not a particularly useful future. The inability to get new transport protocols deployed isn't just an issue with NATs, it's an issue with the kernelspace/userspace boundary in the endpoints. This has more than NAT to do with why we failed to deploy SCTP.
>> 
> TCP is not a useful future?

TCP *alone* is not a useful future. A future where all traffic runs over TCP is a missed opportunity. Most network applications don't necessarily fit on a transport whose API was designed to make the network look like a tape drive, and wedging them into a stream API is inefficient and brittle. Loss-based TCP congestion control algorithms (basically all of the ones deployed for general use) don't play particularly nicely with link layers designed without a recognition that TCP *must* induce loss as a congestion control signal.

> From where I sit, udp is less than 5% of traffic at peak, except during attacks when it goes north of 50%.  This is why policing udp is low risk and high reward in real networks.
> 
> Put the good udp in IPSec. Better, put sctp on the wire or in IPSec. Xbox is doing this with IPv6
> 
> https://www.nanog.org/sites/default/files/wed.general.palmer.xbox_.47.pdf

Indeed, the diagram above looks a lot like a super-simplified version of the diagram in the IPsec RFC that says where you stick various IPv6 extension headers when using ESP, if you put the explicit cooperation stuff in hop by hop and destination options. But it's not clear to me how I would ship an app over SCTP over transport mode ESP when I'm not also the administrator *and* the vendor of the OS running on the endsystem, and it's not clear to me how I would handle keying for such a thing without a closed management framework.

So I still think we need something else here.

Cheers,

Brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 10:13:32 -0700

I'm left with the following questions:

- if you want a new UDP protocol number, why not just us 33 (DCCP)?

- if NATs would not be an issue to a new transport, why has it inhibited
all the recent attempts? (DCCP, SCTP)

Although I appreciate the implications on TAPS for "do what I mean"
negotiation of alternate transports, TAPS doesn't solve either of the
issues above.

AFAICT, they both highlight:

a) why this is hard

b) why this doesn't actually need a new IANA transport protocol codepoint

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 10:52:14 -0700

Hi, Dave,

Yes, I should have included UDPLite in the list below and it's a very
direct replacement for UDP when used as you indicate.

However, one goal of a transition out of UDP might be to include
integrated congestion control, for which either SCTP or DCCP would be a
better alternative.

It would be interesting to know if you've tested UDPLite through
consumer-grade NATs or across any CGNs - i.e., network devices that need
to keep transport-based state, not just firewalls.

Joe

On 7/24/2015 10:43 AM, Dave Taht wrote:
> I note that udplite is widely available, understood by bsd and linux,
> passed by at least a few firewalls, and if you don't use the reduced
> checksum feature, indistinguishable from udp... and one line of app
> code to use...
> 
> ... and widely underused. I did quite a bit of testing of it on ipv6
> paths, and even converted a few tools like mosh and netperf to use it
> for a while. Attempting to at least measure how well udplite gets
> around would be a starting place for a new protocol, and at least
> probing for the ability to use it in, say, quic might make some
> headway.
> 
> My overall objection, however, to a new connectionless protocol for
> "good udp applications" is that we had no idea how many crap udp
> applications we had at the time we wrote them.
> 
> On Fri, Jul 24, 2015 at 7:13 PM, Joe Touch <[email-masked]> wrote:

---

From: Dave Taht <[email-masked]>
Date: Fri, 24 Jul 2015 19:43:58 +0200

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: Brian Trammell <[email-masked]>
Date: Tue, 28 Jul 2015 10:27:48 +0200

Hi, Tom,

Thanks for your message. A partial reply for now, may revisit a few bits later once I've chewed through the post-IETF queue a bit...

> On 26 Jul 2015, at 04:17, Tom Herbert <[email-masked]> wrote:
> 
> Hi Brian,
> Can you provide any more context to support this statement? SCTP has
> been around a long time, and also there is SCTP-over-UDP which should
> already facilitate userspace implementation of SCTP (in fact that
> there is a publicly available userspace SCTP stack
> (https://docs.google.com/presentation/d/190sBrVsLICDn6ayni9bYTjZNRUgpOjlvUhbIVhyjfpA/edit#slide=id.p).
> When I Googled "why isn't SCTP deployed", I get
> http://stackoverflow.com/questions/1171555/why-is-sctp-not-much-used-known
> which seems to mostly imply that SCTP is not deployed because home
> routers can deal with it (at least as of 2011). I suppose the demise
> of SCTP deployment could be a classic chicken-and-the-egg problem: OS
> vendors won't invest in a protocol that can be reliably be transported
> over the Internet, and router vendors are probably unwilling invest in
> support for protocols that aren't widely being used.

First, I should have used a different tense in my statement above:

"This has more than NAT to do with why we *have* failed to deploy SCTP." We haven't deployed SCTP widely *yet*, but history isn't over yet. If I wasn't optimistic, I wouldn't be posting to a list called "stack evolution". :)

I agree with you that it's chicken-and-egg. But it would be easier to conjure chickens from the vacuum without needing to hack the kernel to do so.

As someone who was introduced to SCTP by Randall's chapter in the third edition of Stevens, I was excited about using the features it described (at the time, transport-layer multihoming and multistreaming) in my applications. I then had occasion to build a compliant IPFIX implementation in 2006 (for which SCTP is the MTI), and suddenly found myself wedged between a maze of kernel patches and userland implementations on the one side and the depths of the OpenSSL codebase trying to wedge DTLS atop SCTP on the other -- for that was the way you were supposed to do security for SCTP before we decided reversing the stack was the way to go and put SCTP on DTLS on UDP. I was not a good enough programmer to get it to work reliably. In any case, the maze of kernel patches meant I could never get the compliance guys to sign off on deploying it; I suspect there's still a fair deployed base of that particular application running IPFIX over TLS over TCP for exactly that reason.

This is admittedly a relatively specific anecdote. I'll observe that with the move to mobile this problem is getting worse, not better, though.

Or tl;dr: even an open source kernel doesn't fix all your problems with requiring kernel changes to get an app deployed on a new stack.

> In any case, for any proposed transport protocol, the questions of why
> existing protocols (e.g. SCTP, DCCP) aren't sufficient or are
> chronically non-deployable will be undoubtably raised. IMO, we should
> separate out transport protocol definition from the delivery mechanism
> intended to facilitate the short term or "transitional" goal of
> deployment on the current Internet (e.g. encapsulation in UDP). SCTP
> already demonstrates this model since it is both a native IP protocol
> (132) and can be encapsulated in UDP (RFC6951). Encapsulation of
> pretty much any IP protocol (foo-over-UDP for TCP, DCCP, etc.) is
> straightforward, and in fact GUE already defines a common and
> extensible method that allows encapsulation of any IP protocol over
> UDP without needing a port number for each protocol.

I've skimmed the draft but need to look into GUE more deeply. Is there code I can play with somewhere?

In any case I agree completely that the "evolutionary" requirement ("support userspace implementation and deployment of new transports in the Internet") should be separated from the "transitional" requirement (UDP encaps).

> As for the desire to run transport protocols in userspace, I would
> point out that transport layer stacks are *not* implemented in a
> vacuum. For instance, a fully functional and performant kernel TCP
> stack will directly interact with the IP layer protocols, driver
> layer, the device, and other sub-systems in the kernel. Interactions
> with IP protocol layer are needed to support PMTU discovery, explicit
> congestion notification, and network QoS via diff-serv. For PMTU
> discovery, we need an interface to set DF for IPv4 and to get ICMP PTB
> messages, for ECN we need to get bits from the IP header of marked
> packets, for diff-serv we need a mechanism to set DSCP bits. I would
> be surprised if support for these were not requirements of a new
> transport protocol intended for Internet scale.

I agree with you here: we need a standard kernel interface for getting PMTU and ECN information back from the IP stack into userspace, and a bidirectional interface for DSCP -- and the abstract API should be reasonably portable across platforms. These are not merely necessary for transport protocols in userspace; this information is often useful up at the application layer, as well.

> Other than that, there are probably >50 places (at least in Linux
> stack) where TCP interacts with other layers.

I'm only passingly familiar with the situation here -- I doubt any of the TCP/IP stack code from the kernel that was around the last time I dug into it (1997) is still there. But I will observe that an interface with fifty hardpoints isn't an interface, it's an arbitrary line drawn through what is essentially an integrated design.

At least for Linux, then, there would need to be a design effort to determine which of these interfaces are essential to expose, and which are essentially accidents of localized optimization decisions. But some of the particular interfaces you mention sound like exactly what would be needed to put a "userspace transport layer" atop IP in Linux, especially the "wider" interfaces (sendmmsg, recvmmsg, sendfile).

> This includes the
> routing layer (like re-routing for failing connections), saving of
> routing path metrics such as ssthresh and ICW which can be applied to
> new connections to a common destination, interactions with queuing
> layer to prioritize and optimize transmission (TCP sockets, unlike UDP
> sockets can be directly flow controlled by local egress queues). There
> is a rich set of protocol offloads in devices to provide checksum
> offload, LSO, LRO, as well as software analogues in GSO and GRO.

Protocol offloading is pretty much the opposite of what we're trying to do here -- offloads only work for ossified protocols pretty much by definition, so we'd have to give these up in the near term for new protocols.

> The
> stack interacts with packet steering mechanisms (deliver packets to
> right CPU),

This may be exposing my ignorance, but packet steering works on (a function of) the 5-tuple, no? One of the points we're conceding here is that the ports are de facto part of the network layer, and appear at fixed offsets from the end of the network header.

Thanks again, cheers,

Brian

> pacing mechanisms to mitigate the effect of packet bursts
> within a flow, has interactions with DOS mitigation mechanisms,
> interactions with memory management for network buffers. There are
> interfaces to set IP options or extension headers, appropriately set
> IPv6 flow labels. There's also newer interactions with mechanisms to
> fight buffer bloat such as TCP small queues and Byte Queue Limits
> (BQL). Kernels also implement protocol aware NAT and packet filtering
> per TCP flags and fields. For UDP specifically, there are newer socket
> APIs which can be used improve efficiency of the UDP data path
> (sendmmsg, recvmmsg). If a new transport protocol is intended to serve
> high throughput (like for serving video) then support for sendfile
> might be important
> (https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf is interesting
> reading on where kernel support is likely headed for that).
> 
> For many of these items there are already kernel interfaces that a
> user space stack could use to get of set the appropriate information
> (will vary between OSes), for some newer features new interfaces will
> need to be developed, for a few extending support into userspace might
> not be feasible. Support for some features, such as LSO or LRO, would
> require the kernel or device to understand the an encapsulated
> transport protocol as well.
> 
> Along these lines, for anyone who is planning on attending Linux
> Plumber's Conference next month we're planning a BOF to discuss kernel
> support for user space transport protocols. We are posing two
> questions: 1) is there anything we can do from the kernel side to
> address the perception that OS development and deployment impedes
> advancements in transport protocols 2) assuming user space transport
> protocols become popular, what sort of support and interfaces can we
> provide to optimize them (like for above interactions).
> 
> Thanks,
> Tom

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2015 09:07:47 -0700

Hi Brian,

Some replies inline...

> I've skimmed the draft but need to look into GUE more deeply. Is there code I can play with somewhere?
http://lwn.net/Articles/615044/ for GUE in Linux
https://lwn.net/Articles/614348/ for FOU which allows direct
encapsulation of any IP protocol over UDP (GUE and FOU share code
base)

> I'm only passingly familiar with the situation here -- I doubt any of the TCP/IP stack code from the kernel that was around the last time I dug into it (1997) is still there. But I will observe tha…
For the most part the interfaces are clean. SCTP and DCCP similarly
interact with the rest of the stack although probably not as intensely
since TCP tends to be the first focus for optimizations.

> Protocol offloading is pretty much the opposite of what we're trying to do here -- offloads only work for ossified protocols pretty much by definition, so we'd have to give these up in the near ter…
Not really, many good protocol offloads are generic mechanisms that
permit broad application. For instance, if implemented properly HW
checksum offload can work on any transport layer protocol checksum
even if device does not know how to parse the the transport or even
network layer or any used encapsulation protocols. CRC could be
similarly offloaded (like for SCTP) at least on transmit. For UDP
encapsulations, we are requesting that vendors implement offloads that
are agnostic to (the many :-) ) different encapsulation protocols (see
http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf).

> This may be exposing my ignorance, but packet steering works on (a function of) the 5-tuple, no? One of the points we're conceding here is that the ports are de facto part of the network layer, and…
The 5-tuple hash for TCP and UDP is what most devices have
implemented, however IMO there is a better solution moving forward. As
I pointed out before, not all packets contain port numbers (IP
fragments, IPIP, GRE/IP, MPLS/IP, etc.). Also, for ECMP and RSS
hardware needs to parse the transport protocol so currently we are
effectively limited from using anything other than TCP or UDP in the
DC for high packet rate applications. I have come to believe that the
real fix for these problems (as opposed to just blindly encapsulating
everything in UDP) is to use IPv6 flow labels for load balancing (a la
RFC6438). We've implemented the necessary kernel support, and are
actively requesting HW vendors to use the field as input to ECMP and
RSS (several already do). We intend to enable automatic flow label
generation by default in Linux after due diligence, so in the not too
distant future expect to see a lot of IPv6 packets on the Internet
with non-zero flow labels!

Tom