Subject: Re: [Stackevo] [tsvwg] draft-byrne-opsec-udp-advisory

From: Brian Trammell <[email-masked]>
Date: Thu, 23 Jul 2015 15:50:15 +0200

hi Cameron, all,

(Copying the IAB Stack Evolution program as well...)

> On 22 Jul 2015, at 16:55, Ca By <[email-masked]> wrote:
> 
> Thanks for the response Brian
> 
> On Wed, Jul 22, 2015 at 5:56 AM, Brian Trammell <[email-masked]> wrote:
>> hi Szilveszter, Cameron, all,
>> 
>> (First, Cameron, thanks for the draft; I'm drafting my own response thereto now).
>> 
>> I can't speak to QUIC, and note that SPUD is a prototype (in the sense of "experimental test article", not "alpha version of a future proposed standard")
>> 
>> However I will note that both SPUD and GUE use a magic number at a fixed point in the header. This magic number was in both cases chosen not to collide with valid packet content at the same offsets for a variety of known UDP-based protocols. This is of course completely useless for keeping *direct* sources of UDP attack traffic from being passed by a QUIC or SPUD-"friendly" baseline UDP rate filter: the magic number (and other details of the packet structure) are documented. So yes, useless for attack mitigation against an army of bots.
>> 
>> It does, however, do a lot for reducing *reflected* attack traffic, and I assume the majority of the garbage packets you see in operation are reflected. (Indeed, have you got data for this?)
> 
> yes.

Cool. So, if we assume that we can find a way to quickly and easily distinguish bad old UDP ("all" of which is reflected crap, for some high value of all) from UDP-as-layer-boundary-reinforcement -- a well-selected magic number being the most obvious way (to me) to do this statelessly.

>> In a reflection attack. Note that the attacker can only induce the reflectors to generate reply traffic with bit patterns within the set of possible patterns for the protocol used. The "echo" service can be used to reflect packets with valid SPUD magic, but I doubt there's much echo out there (and echo in any case has a reflection factor of 1x, making it not much more attractive that just spoofing source addresses).
>> 
>> We need a better answer for how the UDP-based encapsulations we're designing operate in a high-garbage-packet environment in general. This protection against reflection attack is a part of that answer IMO.
>> 
>> Cheers,
>> 
>> Brian
>> 
> 
> That is all good to know.  But, the genesis of draft-byrne-opsec-udp-advisory is trying to create "a light at the end of the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is largely attack traffic.  And, it is of such a scale that it causes network meltdowns.  The only acceptable path is to rate-limit the entire protocol... as unsavory as that might sound, i have been doing this for years as a last ditch effort to keep the lights on, and it works.  Without it, we careen from outage to outage.
> 
> Loading more types of legitimate traffic into transport protocol 17 makes my life harder because you are mixing important and legitimate signal (youtube!) with a sewer of destructive criminal noise (ssdp!)

Reductio ad absurdum, you could say the same thing about IPv4. :) But I get your point...

So, there are broadly two things I mean when I say UDP:

(1) Packets with a protocol identifier of 17 on them. This is what you want to filter on because most of them are crap.

(2) What happens when you open a SOCK_DGRAM socket. This service is basically "nonprivileged userspace raw sockets". When we're talking about making it possible to deploy new transport protocols, the "userspace" is IMO crucial here (and the "nonprivileged" equally so if you're talking about the new world where everything happens on devices where the user has no root).

If we had a way to get this second thing without throwing a UDP header on it, that'd be great, but whatever it is you need port numbers on it, more because you can have multiple userspace processes on each eventual endpoint than you need to get around NAT.

> I can only think of 2 paths forward.  Remove the noise from UDP (aka, upgrade the internet to stop DRDoS [home routers with SSDP, open DNS and NTP, booter services...]) or stop assuming your UDP packets will be transported at scale (aka, the ietf needs to not load new things into UDP and accept the mess that grew out of UDP... learning from it in new transport protocols that are not transport protocol 17).

I see the use of UDP here as a transition technology: that is, something that will be deployed for the next twenty years or so. The eventual goal (at least mine) is to be able to do all of this over a new protocol. The stack looks something like this:

   |  transport services API (TAPS)  |
   +---------------------------------+
   | shiny new transport protocol(s) |  encrypted
==========================================
   |   explicit cooperation layer    |  cleartext
   +-----------------+               |
   |       UDP       |               |
   +----------+------+---------------+
   |    IPv4  |           IPv6       |
   +----------+----------------------+

    ------------ time -------------->

> The former is impossible and outside the capabilities of anyone reading this email.

I'd submit that if a good-UDP / bad-UDP discriminator, as well as return routability proof and state exhaustion resistance, can be designed into future good-UDP protocols, that's a start.

> The latter is inconvenient for some yet-to-be-standardized or broadly deployed technologies, and it is within the circle of influence here at the IETF to wind down UDP.

This is tantamount to saying the future of the Internet is over TCP, which is not a particularly useful future. The inability to get new transport protocols deployed isn't just an issue with NATs, it's an issue with the kernelspace/userspace boundary in the endpoints. This has more than NAT to do with why we failed to deploy SCTP.

One thing that has changed since we failed to deploy SCTP: machines (kernel instances) are silly cheap on the server side now, so building new transports into your servers, which used to be kind of an operational pain, is actually pretty easy. Another thing that has changed: most people are connected to the Internet on machines they don't even have administrative privilege: even if they want to, they can't add new protocols to the kernels.

Something that probably hasn't: default firewall rules.

<shameless plug>But this is something we should measure. Interested people, come to HOPSRG on Friday morning.</shameless plug>

Cheers,

Brian

> Finally, i would like to touch a 3rd option.  The DRDoS situation is so bad that it clogs the pipes such that the traffic never reaches the on-premise "middle-boxes".  Thus we have a new category of hell called the "middle-clouds" which scrubs traffic in the cloud.  These services inflict at 60x surchage over what one normally pays for ISP access on a usage basis, and their marketing puffery is based around how large of a ddos they have fought off.
> 
> You can tell them by the home page buttons that say "I AM UNDER ATTACK!" that we lowly souls are compelled to click on it in desperation to keep our networks up.  This market structure means more attacks generate more $$ for those in the mitigation business.  It's a classic mafia structure where you pay for protection or get shaken down.
> 
> There is even an IETF WG called DOTS that has resigned itself to perpetually dealing with and signalling the DDoS situation to local and cloud based middle boxes.  I cannot feel good about the long term vision of DOTS being broadly deployed and relied on for internet operations.
> 
> We need a better solution than constant attacks and escalating tiers of middle-boxen.
> 
> For context, in your favorite search engine, query "network booter" and "hosted ddos protection".  These market structures are unsettling.  The IETF needs to steer us out of this, which means "cutting bait" on UDP and apply lessons-learned to new transport protocols.
> 
> Regards,
> CB
> 
> 
> 
> 
> 
> > On 22 Jul 2015, at 12:24, Szilveszter Nadas <[email-masked]> wrote:
> >
> > Hi,
> >
> >
> >
> > I am just wondering, what keeps the “DOS guys” from adopting any new wire format introduced? E.g. the wire format ofQUIC or SPUD even if those are not over UPD?
> >
> >
> >
> > Of course it might take a while before they adapt, but not that long, probably. Is it a strong enough argument that we have a few more months or a year?
> >
> >
> >
> > Cheers,
> >
> > Szilveszter
> >
> >
> >
> > From: tsvwg [mailto:[email-masked]] On Behalf Of Ca By
> > Sent: Wednesday, July 22, 2015 4:12
> > To: Joe Touch
> > Cc: [email-masked]; [email-masked]
> > Subject: Re: [tsvwg] draft-byrne-opsec-udp-advisory
> >
> >
> >
> > Thanks Joe, for providing feedback
> >
> > On Tuesday, July 21, 2015, Joe Touch <[email-masked]> wrote:
> >
> > I disagree with the recommendations as follows:
> >
> > - avoiding use of UDP as a general recommendation isn't helpful
> >
> >         UDP should never be used to subvert protections available
> >         in other transports, e.g., TCP, SCTP, or DCCP
> >
> >         UDP has its uses and in particular has no direct replacement
> >         for multicast discovery (whether using mDNS or service-specific
> >         capability discovery)
> >
> >
> >
> > Ack.
> >
> >
> >
> > - IPsec is appropriate for only some uses
> >         Notably it cannot easily be used with NAT (you'd end up
> >         needing to run IPsec over UDP)
> >
> >
> >
> > Seems the ephemeral existence of  NAT has ossified  your view of the stack
> >
> >
> >
> > - I'm not sure what you mean by "IANA assigned protocol number" for
> > things like QUIC:
> >
> >
> >
> > I mean transport protocol
> >
> >
> >
> >         if you mean port number, I disagree, as it would serve only
> >         to move the problem to the new port and only consume a new
> >         port assignment without benefit
> >
> >         if you mean transport protocol number, that won't be compatible
> >         with NATs and firewalls that translate or block non-TCP/UDP
> >         protocols
> >
> >
> >
> > Yes, NATs are a challenge for many things. The ietf has specified many things that pose a challenge to NATs.  Every month that goes by, fewer and fewer of my subscribers require NAT. Please take note of this reality.
> >
> >
> >
> >         in general, assigning new things to new numbers doesn't
> >         protect it from abuse
> >
> >
> >
> > But it does help us classify and treat traffic.  This is critical for network operation. Failure to do so is immediate doom.
> >
> >
> >
> > In the case of udp, the vast majority of transmitted udp is only ever destine for the bit bucket.  I speak from a great folks who brought us ntp, snmp, ssdp, chargen and so on  .... Did not foresee the evil that would be done
> >
> >
> >
> > I agree with the most of the advice to network operators. IMO, this
> > document would be better served focusing there.
> >
> >
> >
> > In fact, the document is focused at app and protocol developers, but for good measure i paid dues to bcp38, but it is only address the trigger of a DRDOS. The napalm is udp and its harmfully deployed protocol.
> >
> >
> >
> > The exception is to rate-limit UDP. There's no basis for doing so UNTIL
> > their resources become recipients of an attack (where they could limit
> > traffic to themselves only) or they experience network congestion. The
> > latter could be handled with preferential dropping of non-responsive
> > traffic, and need only be done when resources are limited.
> >
> >
> >
> > Joe, as a network operator, i can assure you that 100% of the time, there is service disrupting udp flows on the internet. Here is a cool visual, i am not sure it is grounded in fact .... Somebody made it
> >
> >
> >
> > http://www.digitalattackmap.com/
> >
> >
> >
> >
> >
> > I.e., large UDP flows are not themselves an attack until they are.
> > Please don't jump to conclusions and act in the absence of need.
> >
> >
> >
> > I would not dare act in the absence of need.  As a network operator performing the outlined tactics in the I-D, i have been called to action to advise the ietf that further udp development efforts will not be met well. It simply not possible to restore udp to be a first class citizen on the 'net.
> >
> >
> >
> > I am all for innovation in transport, but not with protocol 17 which is home to nearly all ddos  modes and vectors. It is simply unconscionable to pretend udp is a blank slate for transport innovation when it is in fact a sewer of downtime engendering crime.
> >
> >
> >
> > I created a visual of my concern https://twitter.com/theipv6guy/status/623535325915168768
> >
> >
> >
> >
> >
> > I see certain people grabbing on to udp because they think it is an easy way to deal with NAT. This is not particularlythe full cost of their innovation with a new transport protocol number, and thusly separating themselves from the abomination that is udp reflecting home routers.
> >
> >
> >
> >
> >
> > Best regards,
> >
> >
> >
> > Cameron
> >
> > Joe
> >
> >
> >
> >
> > On 7/20/2015 5:44 PM, Ca By wrote:
> > > Dear transport,
> > >
> > >
> > > Given the number of udp related developments, I created this I-d to
> > > express concern  about further udp work and expound on some operational
> > > practices that conflict with udp growth.
> > >
> > > Your feedback is welcome.
> > >
> > > =========
> > >
> > > A new version of I-D, draft-byrne-opsec-udp-advisory-00.txt
> > > has been successfully submitted by Cameron Byrne and posted to the
> > > IETF repository.
> > >
> > > Name:        draft-byrne-opsec-udp-advisory
> > > Revision:    00
> > > Title:        Advisory Guidelines for UDP Deployment
> > > Document date:    2015-07-20
> > > Group:        Individual Submission
> > > Pages:        5
> > > URL:
> > >            https://www.ietf.org/internet-drafts/draft-byrne-opsec-udp-advisory-00.txt
> > > Status:
> > >         https://datatracker.ietf.org/doc/draft-byrne-opsec-udp-advisory/
> > > Htmlized:
> > >       https://tools.ietf.org/html/draft-byrne-opsec-udp-advisory-00
> > >
> > >
> > > Abstract:
> > >   User Datagram Protocol (UDP) is commonly used as a volumetric attack
> > >   transport on the internet.  Some network operators experience surges
> > >   of UDP attack traffic that are multiple orders of magnitude above the
> > >   baseline traffic rate for UDP.  Application developers should be
> > >   advised that UDP is being rate-limited on a bits-per-second and
> > >   packet-per-second basis by network operators to enforce known good
> > >   baseline traffic levels for UDP. UDP has been abused to such an
> > >   extent that legitimate use may become collateral damage and
> > >   application and protocol developers should avoid using UDP as a
> > >   transport when possible.
> > >
> > >
> > >
> > >
> > >
> > > Please note that it may take a couple of minutes from the time of submission
> > > until the htmlized version and diff are available at tools.ietf.org
> > > <http://tools.ietf.org/>.
> > >
> > > The IETF Secretariat
> >
> 
>

---

From: Ca By <[email-masked]>
Date: Thu, 23 Jul 2015 22:04:06 -0700

On Thursday, July 23, 2015, Brian Trammell <[email-masked]> wrote:

> > On 22 Jul 2015, at 16:55, Ca By <[email-masked] <javascript:;>>
> wrote:
> >
> > On Wed, Jul 22, 2015 at 5:56 AM, Brian Trammell <[email-masked]
> <javascript:;>> wrote:
> >> I can't speak to QUIC, and note that SPUD is a prototype (in the sense
> of "experimental test article", not "alpha version of a future proposed
> standard")
> >>
> >> However I will note that both SPUD and GUE use a magic number at a
> fixed point in the header. This magic number was in both cases chosen not
> to collide with valid packet content at the same offsets for a variety of
> known UDP-based protocols. This is of course completely useless for keeping
> *direct* sources of UDP attack traffic from being passed by a QUIC or
> SPUD-"friendly" baseline UDP rate filter: the magic number (and other
> details of the packet structure) are documented. So yes, useless for attack
> mitigation against an army of bots.
> >>
> >> It does, however, do a lot for reducing *reflected* attack traffic, and
> I assume the majority of the garbage packets you see in operation are
> reflected. (Indeed, have you got data for this?)
> >
> > yes.
>
> Cool. So, if we assume that we can find a way to quickly and easily
> distinguish bad old UDP ("all" of which is reflected crap, for some high
> value of all) from UDP-as-layer-boundary-reinforcement -- a well-selected
> magic number being the most obvious way (to me) to do this statelessly.
>
>
I would simply call it udp2 with a new transport protocol number.

But I imagine you want to preserve some way to blast through legacy nats,
which is not a good strategic goal / constraint

>> In a reflection attack. Note that the attacker can only induce the
> reflectors to generate reply traffic with bit patterns within the set of
> possible patterns for the protocol used. The "echo" service can be used to
> reflect packets with valid SPUD magic, but I doubt there's much echo out
> there (and echo in any case has a reflection factor of 1x, making it not
> much more attractive that just spoofing source addresses).
> >>
> >> We need a better answer for how the UDP-based encapsulations we're
> designing operate in a high-garbage-packet environment in general. This
> protection against reflection attack is a part of that answer IMO.
> >>
> >> Cheers,
> >>
> >> Brian
> >>
> >
> > That is all good to know.  But, the genesis of
> draft-byrne-opsec-udp-advisory is trying to create "a light at the end of
> the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is
> largely attack traffic.  And, it is of such a scale that it causes network
> meltdowns.  The only acceptable path is to rate-limit the entire
> protocol... as unsavory as that might sound, i have been doing this for
> years as a last ditch effort to keep the lights on, and it works.  Without
> it, we careen from outage to outage.
> >
> > Loading more types of legitimate traffic into transport protocol 17
> makes my life harder because you are mixing important and legitimate signal
> (youtube!) with a sewer of destructive criminal noise (ssdp!)
>
> (1) Packets with a protocol identifier of 17 on them. This is what you
> want to filter on because most of them are crap.
>
>
Yes

> (2) What happens when you open a SOCK_DGRAM socket. This service is
> basically "nonprivileged userspace raw sockets". When we're talking about
> making it possible to deploy new transport protocols, the "userspace" is
> IMO crucial here (and the "nonprivileged" equally so if you're talking
> about the new world where everything happens on devices where the user has
> no root).
>
>
Udp2

> If we had a way to get this second thing without throwing a UDP header on
> it, that'd be great, but whatever it is you need port numbers on it, more
> because you can have multiple userspace processes on each eventual endpoint
> than you need to get around NAT.
>
>
I am optimistic about IPv6 deployment and the removal of nats and middle
boxes.  This is tied to LTE capacity  and latency improvements outpacing
middle box utility and capacity. I find it strange that the ietf is trying
now to talk to middle boxes while they are waning. YMMV.

> > I can only think of 2 paths forward.  Remove the noise from UDP (aka,
> upgrade the internet to stop DRDoS [home routers with SSDP, open DNS and
> NTP, booter services...]) or stop assuming your UDP packets will be
> transported at scale (aka, the ietf needs to not load new things into UDP
> and accept the mess that grew out of UDP... learning from it in new
> transport protocols that are not transport protocol 17).
>
> I see the use of UDP here as a transition technology: that is, something
> that will be deployed for the next twenty years or so. The eventual goal
> (at least mine) is to be able to do all of this over a new protocol. The
> stack looks something like this:
>
>
> I'd submit that if a good-UDP / bad-UDP discriminator, as well as return
> routability proof and state exhaustion resistance, can be designed into
> future good-UDP protocols, that's a start.
>
>
But it does not solve change anything in the I-D at hand, as the I-D is
tactical and operational in nature

> > The latter is inconvenient for some yet-to-be-standardized or broadly
> deployed technologies, and it is within the circle of influence here at the
> IETF to wind down UDP.
>
> This is tantamount to saying the future of the Internet is over TCP, which
> is not a particularly useful future. The inability to get new transport
> protocols deployed isn't just an issue with NATs, it's an issue with the
> kernelspace/userspace boundary in the endpoints. This has more than NAT to
> do with why we failed to deploy SCTP.
>
> TCP is not a useful future?

>From where I sit, udp is less than 5% of traffic at peak, except during
attacks when it goes north of 50%.  This is why policing udp is low risk
and high reward in real networks.

Put the good udp in IPSec. Better, put sctp on the wire or in IPSec. Xbox
is doing this with IPv6

https://www.nanog.org/sites/default/files/wed.general.palmer.xbox_.47.pdf

> One thing that has changed since we failed to deploy SCTP: machines
> (kernel instances) are silly cheap on the server side now, so building new
> transports into your servers, which used to be kind of an operational pain,
> is actually pretty easy. Another thing that has changed: most people are
> connected to the Internet on machines they don't even have administrative
> privilege: even if they want to, they can't add new protocols to the
> kernels.
>
>
Failed to deploy sctp ?!  SCTP is just needs a chance.

The good news is, people don't have admin privilege so sctp can be brought
down from on high and deployed without any intervention from people who
don't know / care

Google and Apple can push sctp to a billion devices and boom, it is good
(some assembly required)

But, they won't even deploy edns0  ...

>
> <shameless plug>But this is something we should measure. Interested
> people, come to HOPSRG on Friday morning.</shameless plug>
>
Cheers,
>
> Brian
>
> > Finally, i would like to touch a 3rd option.  The DRDoS situation is so
> bad that it clogs the pipes such that the traffic never reaches the
> on-premise "middle-boxes".  Thus we have a new category of hell called the
> "middle-clouds" which scrubs traffic in the cloud.  These services inflict
> at 60x surchage over what one normally pays for ISP access on a usage
> basis, and their marketing puffery is based around how large of a ddos they
> have fought off.
> >
> > You can tell them by the home page buttons that say "I AM UNDER ATTACK!"
> that we lowly souls are compelled to click on it in desperation to keep our
> networks up.  This market structure means more attacks generate more $$ for
> those in the mitigation business.  It's a classic mafia structure where you
> pay for protection or get shaken down.
> >
> > There is even an IETF WG called DOTS that has resigned itself to
> perpetually dealing with and signalling the DDoS situation to local and
> cloud based middle boxes.  I cannot feel good about the long term vision of
> DOTS being broadly deployed and relied on for internet operations.
> >
> > For context, in your favorite search engine, query "network booter" and
> "hosted ddos protection".  These market structures are unsettling.  The
> IETF needs to steer us out of this, which means "cutting bait" on UDP and
> apply lessons-learned to new transport protocols.
> >
> > Regards,
> > CB
> >
> >
> >
> >
> >
> > > On 22 Jul 2015, at 12:24, Szilveszter Nadas <
> [email-masked] <javascript:;>> wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > I am just wondering, what keeps the “DOS guys” from adopting any new
> wire format introduced? E.g. the wire format ofQUIC or SPUD even if those
> are not over UPD?
> > >
> > >
> > >
> > > Of course it might take a while before they adapt, but not that long,
> probably. Is it a strong enough argument that we have a few more months or
> a year?
> > >
> > >
> > >
> > > Cheers,
> > >
> > > Szilveszter
> > >
> > >
> > >
> > > From: tsvwg [mailto:[email-masked] <javascript:;>] On Behalf
> Of Ca By
> > > Cc: [email-masked] <javascript:;>;
> [email-masked] <javascript:;>
> > > On Tuesday, July 21, 2015, Joe Touch <[email-masked] <javascript:;>>
> wrote:
> > >
> > > Yes, NATs are a challenge for many things. The ietf has specified many
> things that pose a challenge to NATs.  Every month that goes by, fewer and
> fewer of my subscribers require NAT. Please take note of this reality.
> > >
> > >
> > >
> > > But it does help us classify and treat traffic.  This is critical for
> network operation. Failure to do so is immediate doom.
> > >
> > >
> > >
> > > In the case of udp, the vast majority of transmitted udp is only ever
> destine for the bit bucket.  I speak from a great folks who brought us ntp,
> snmp, ssdp, chargen and so on  .... Did not foresee the evil that would be
> done
> > >
> > >
> > >
> > > In fact, the document is focused at app and protocol developers, but
> for good measure i paid dues to bcp38, but it is only address the trigger
> of a DRDOS. The napalm is udp and its harmfully deployed protocol.
> > >
> > >
> > >
> > > Joe, as a network operator, i can assure you that 100% of the time,
> there is service disrupting udp flows on the internet. Here is a cool
> visual, i am not sure it is grounded in fact .... Somebody made it
> > >
> > >
> > >
> > > I would not dare act in the absence of need.  As a network operator
> performing the outlined tactics in the I-D, i have been called to action to
> advise the ietf that further udp development efforts will not be met well.
> It simply not possible to restore udp to be a first class citizen on the
> 'net.
> > >
> > >
> > >
> > > I am all for innovation in transport, but not with protocol 17 which
> is home to nearly all ddos  modes and vectors. It is simply unconscionable
> to pretend udp is a blank slate for transport innovation when it is in fact
> a sewer of downtime engendering crime.
> > >
> > >
> > >
> > > I created a visual of my concern
> https://twitter.com/theipv6guy/status/623535325915168768
> > >
> > >
> > >
> > >
> > >
> > > I see certain people grabbing on to udp because they think it is an
> easy way to deal with NAT. This is not particularlythe full cost of their
> innovation with a new transport protocol number, and thusly separating
> themselves from the abomination that is udp reflecting home routers.
> > >
> > >
> > >
> > >
> > >
> > > Best regards,
> > >
> > >
> > >
> > > Cameron
> > >
> > > Joe
> > >
> > >
> > >
> > >

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 27 Jul 2015 13:30:54 +0200

Hi CB,

> Am 24.07.2015 um 07:04 schrieb Ca By <[email-masked]>:
> 
> Put the good udp in IPSec. Better, put sctp on the wire or in IPSec. Xbox is doing this with IPv6 
> 

Please note that they encapsulate IPsec/IPv6 over UDP over IPv4 for IPV4 support. Yet another example where UDP is used.

Mirja

---

From: Tom Herbert <[email-masked]>
Date: Sat, 25 Jul 2015 22:17:44 -0400

> This is tantamount to saying the future of the Internet is over TCP, which is not a particularly useful future. The inability to get new transport protocols deployed isn't just an issue with NATs,…

Hi Brian,

Can you provide any more context to support this statement? SCTP has
been around a long time, and also there is SCTP-over-UDP which should
already facilitate userspace implementation of SCTP (in fact that
there is a publicly available userspace SCTP stack
(https://docs.google.com/presentation/d/190sBrVsLICDn6ayni9bYTjZNRUgpOjlvUhbIVhyjfpA/edit#slide=id.p).
When I Googled "why isn't SCTP deployed", I get
http://stackoverflow.com/questions/1171555/why-is-sctp-not-much-used-known
which seems to mostly imply that SCTP is not deployed because home
routers can deal with it (at least as of 2011). I suppose the demise
of SCTP deployment could be a classic chicken-and-the-egg problem: OS
vendors won't invest in a protocol that can be reliably be transported
over the Internet, and router vendors are probably unwilling invest in
support for protocols that aren't widely being used.

In any case, for any proposed transport protocol, the questions of why
existing protocols (e.g. SCTP, DCCP) aren't sufficient or are
chronically non-deployable will be undoubtably raised. IMO, we should
separate out transport protocol definition from the delivery mechanism
intended to facilitate the short term or "transitional" goal of
deployment on the current Internet (e.g. encapsulation in UDP). SCTP
already demonstrates this model since it is both a native IP protocol
(132) and can be encapsulated in UDP (RFC6951). Encapsulation of
pretty much any IP protocol (foo-over-UDP for TCP, DCCP, etc.) is
straightforward, and in fact GUE already defines a common and
extensible method that allows encapsulation of any IP protocol over
UDP without needing a port number for each protocol.

As for the desire to run transport protocols in userspace, I would
point out that transport layer stacks are *not* implemented in a
vacuum. For instance, a fully functional and performant kernel TCP
stack will directly interact with the IP layer protocols, driver
layer, the device, and other sub-systems in the kernel. Interactions
with IP protocol layer are needed to support PMTU discovery, explicit
congestion notification, and network QoS via diff-serv. For PMTU
discovery, we need an interface to set DF for IPv4 and to get ICMP PTB
messages, for ECN we need to get bits from the IP header of marked
packets, for diff-serv we need a mechanism to set DSCP bits. I would
be surprised if support for these were not requirements of a new
transport protocol intended for Internet scale.

Other than that, there are probably >50 places (at least in Linux
stack) where TCP interacts with other layers. This includes the
routing layer (like re-routing for failing connections), saving of
routing path metrics such as ssthresh and ICW which can be applied to
new connections to a common destination, interactions with queuing
layer to prioritize and optimize transmission (TCP sockets, unlike UDP
sockets can be directly flow controlled by local egress queues). There
is a rich set of protocol offloads in devices to provide checksum
offload, LSO, LRO, as well as software analogues in GSO and GRO. The
stack interacts with packet steering mechanisms (deliver packets to
right CPU), pacing mechanisms to mitigate the effect of packet bursts
within a flow, has interactions with DOS mitigation mechanisms,
interactions with memory management for network buffers. There are
interfaces to set IP options or extension headers, appropriately set
IPv6 flow labels. There's also newer interactions with mechanisms to
fight buffer bloat such as TCP small queues and Byte Queue Limits
(BQL). Kernels also implement protocol aware NAT and packet filtering
per TCP flags and fields. For UDP specifically, there are newer socket
APIs which can be used improve efficiency of the UDP data path
(sendmmsg, recvmmsg). If a new transport protocol is intended to serve
high throughput (like for serving video) then support for sendfile
might be important
(https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf is interesting
reading on where kernel support is likely headed for that).

For many of these items there are already kernel interfaces that a
user space stack could use to get of set the appropriate information
(will vary between OSes), for some newer features new interfaces will
need to be developed, for a few extending support into userspace might
not be feasible. Support for some features, such as LSO or LRO, would
require the kernel or device to understand the an encapsulated
transport protocol as well.

Along these lines, for anyone who is planning on attending Linux
Plumber's Conference next month we're planning a BOF to discuss kernel
support for user space transport protocols. We are posing two
questions: 1) is there anything we can do from the kernel side to
address the perception that OS development and deployment impedes
advancements in transport protocols 2) assuming user space transport
protocols become popular, what sort of support and interfaces can we
provide to optimize them (like for above interactions).

Thanks,
Tom