Subject: UDP source ports for HTTP/3 and QUIC

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Jul 2021 10:20:39 +1000

[ bringing this up on both lists because it's not yet clear what the right scope is ]

It's not uncommon for servers to block certain UDP source ports to avoid being overwhelmed by certain reflection attacks. In particular:

* 53 - DNS
* 123 - NTP
* 1900 - SSDP
* 5353 - mDNS
* 11211 - memcached

... among other candidates.

See, eg., <https://blog.cloudflare.com/reflections-on-reflections/>. This isn't done to avoid protocol vulnerabilities as such -- it's to avoid volumetric attacks (usually DDoS). 

If a client chooses source ports from the ephemeral port range, this shouldn't be an issue. However, some implementations (or deployments) extend the source port range "downwards" to avoid exhaustion:

* On Linux, it's pretty common to tune to extend the ephemeral port range, e.g., <https://ma.ttias.be/linux-increase-ip_local_port_range-tcp-port-range/>

* On some versions of Windows, ports as low as 1025 are used: <https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/service-overview-and-network-port-requirements>

* FreeBSD has used ports as low as 1024 in the past; their documentation is... confused right now. <https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/netinet/in.h#L272> (I don't run a FreeBSD box to check this)

* NAT and CGNAT are a thing. <https://www.rfc-editor.org/rfc/rfc4787.html#section-4.2.1> recommends that source ports in the range 1024-65535 be reassigned within that range, which can cause collisions. 

Overall, the chance of collision (i.e., a client or a NAT choosing a blocked source port) is quite low, but at Internet scale it'll happen, leading to the client needing to open a new connection, thereby adding perceived latency. 

At a minimum, it seems like a good thing for client and NAT developers to be aware that some ports are likely to be blocked, so that they can avoid them if they care about perceived performance. This e-mail might be enough for that immediate purpose.

My question is whether there's a need for more coordination -- e.g., a very short RFC outlining such ports, to help bring this to folks' attention (especially in the NAT crowd)?

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 15 Jul 2021 11:20:19 +1000

Thanks for raising this Mark.  (Limiting my reply to QUIC, because I think this is more specific to that group.)

On Thu, Jul 15, 2021, at 10:20, Mark Nottingham wrote:
> Overall, the chance of collision (i.e., a client or a NAT choosing a 
> blocked source port) is quite low, but at Internet scale it'll happen, 
> leading to the client needing to open a new connection, thereby adding 
> perceived latency. 

What is going to happen in many cases is that QUIC will be disabled in addition to having high latency for that connection.
 
> My question is whether there's a need for more coordination -- e.g., a 
> very short RFC outlining such ports, to help bring this to folks' 
> attention (especially in the NAT crowd)?

So this isn't easy.  I think that it is good you raise this either way.  From the client side, some greater determinism around this would be good, for the above reasons if nothing else.  We don't currently have a source port blocklist and we will need to add that.

Because this isn't entirely under our control as a client, us knowing doesn't really provide a complete solution.  To that end, I am inclined to say that an informational RFC is a good idea.  Ideally, NAT and cgNAT will avoid these ports also and an RFC does tend to reach people.

HOWEVER, the challenge there is that the exact set of port numbers could be volatile.  That memcached is a problem is well known and so we can reasonably exclude it for the foreseeable future, but we don't know if other protocols might become a source of DoS.  We might hope that new protocols adopt designs with more rigorous DoS prevention inherent to them (like QUIC), making this less likely, but we can't guarantee it.  Baking a list into an RFC seems like it might need careful consideration.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2021 04:39:11 +0200

On Thu, Jul 15, 2021 at 11:20:19AM +1000, Martin Thomson wrote:
> What is going to happen in many cases is that QUIC will be disabled in
> addition to having high latency for that connection.
>  
> So this isn't easy.  I think that it is good you raise this either way.  From
> the client side, some greater determinism around this would be good, for the
> above reasons if nothing else.  We don't currently have a source port
> blocklist and we will need to add that.
> 
> Because this isn't entirely under our control as a client, us knowing doesn't
> really provide a complete solution.  To that end, I am inclined to say that
> an informational RFC is a good idea.  Ideally, NAT and cgNAT will avoid these
> ports also and an RFC does tend to reach people.

I don't imagine a single second that those random unprivileged ports will be
avoided. They were randomly picked and will be difficult to exclude from NAT
ranges. By the time implementors have a solution to exclude them, they will
probably not be used anymore and others will happen.

I think that we must at least specify that privileged ports (<1024) must
always be blocked because under no circumstances valid HTTP traffic should
be emitted from such ports which are normally reserved for services. And
for the rest we should rather caution that many UDP services are highly
vulnerable to reflection attacks because there's no such thing as a
distinction from a request packet and a response packet like in TCP,
and that along QUIC's life new popular services will occasionally pop up
and will cause trouble.

The most important for implementors is to be able to quickly validate a
QUIC packet and make sure that no response is ever sent if it cannot be
decoded. This will only lead to CPU usage and nothing more. Sadly, if
we'd had a short protocol identifier at the very beginning of each
packet (e.g. even a 8-bit or 16-bit field), it would have become extremely
difficult to make other protocols match it in their responses and it
wouldn't have been an issue at all. But we don't have this so the packets
will at least have to be parsed.

Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 15 Jul 2021 10:56:28 +0200

It is perhaps worth noting that due to QUIC (optionally) having unique connection identifiers, it is feasible to have many connections on the same source port. Therefore that could be a recommendation in cases where some source ports might be blocked.

As others have pointed out, I would suspect an RFC with a port list would quickly become outdated.

Mikkel

> On 15 Jul 2021, at 02.20, Mark Nottingham <[email-masked]> wrote:
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2021 11:29:37 +0200

On Thu, Jul 15, 2021 at 10:56:28AM +0200, Mikkel Fahnøe Jørgensen wrote:
> It is perhaps worth noting that due to QUIC (optionally) having unique
> connection identifiers, it is feasible to have many connections on the same
> source port. Therefore that could be a recommendation in cases where some
> source ports might be blocked.

I think that this is an excellent idea! The simple fact that this is
being discussed precisely is because the source port serves no purpose
here other than being compatible with UDP. So basically we could have
a recommendation that each application preferably uses a single socket
and source port for outgoing communication. This will also lower the
stress on source port allocation (and recycling) as well as the need
for file descriptors.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Thu, 15 Jul 2021 11:53:46 +0200

> Am 15.07.2021 um 11:29 schrieb Willy Tarreau <[email-masked]>:
> 

Yikes, all QUIC packets will potentially arrive a the same child listener. Let's go all the way and defined a fixed source port. ^^

- Stefan

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2021 12:01:49 +0200

On Thu, Jul 15, 2021 at 11:53:46AM +0200, Stefan Eissing wrote:
> Yikes, all QUIC packets will potentially arrive a the same child listener.

That's a good point, indeed. But we could suggest that for low network
resource usage (i.e. browsers), reusing sockets that proved to work fine
is recommended.

> Let's go all the way and defined a fixed source port. ^^

:-)

Willy

---

From: Töma Gavrichenkov <[email-masked]>
Date: Thu, 15 Jul 2021 14:14:19 +0300

Peace,

So basically we could have
> and source port for outgoing communication.

> Let's go all the way and defined a fixed source port. ^^

This would irretrievably break ECMP/UCMP 5-tuple hashes pretty much
everywhere in transit.  The closer to the endpoint the worse.

(I understand you were just joking but wanted to point that out just in
case someone takes this seriously)

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Thu, 15 Jul 2021 14:36:13 +0300

Peace,

> As others have pointed out, I would suspect an RFC with a port list would

Speaking generally of lists of a content too dynamic and too host-specific
to hardcode in RFCs, there once was a habit of putting them into DNS
records.

--
Töma

---

From: Nick Banks <[email-masked]>
Date: Thu, 15 Jul 2021 13:40:03 +0000

You should definitely not make any assumptions around having unique source ports for QUIC connections. MsQuic already supports sharing the local port and is looking to make it a default behavior (for at least some scenarios) to avoid the fairly common "port exhaustion" problems we see with TCP. This doesn't necessarily mean all connections would be on the same source port, but only a few ports might be used for all connections.

Thanks,
- Nick

Sent from Outlook<http://aka.ms/weboutlook>

________________________________
From: QUIC <[email-masked]> on behalf of Töma Gavrichenkov <[email-masked]>
Sent: Thursday, July 15, 2021 6:36 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: Re: UDP source ports for HTTP/3 and QUIC

Peace,

On Thu, Jul 15, 2021, 11:57 AM Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
As others have pointed out, I would suspect an RFC with a port list would quickly become outdated.

Speaking generally of lists of a content too dynamic and too host-specific to hardcode in RFCs, there once was a habit of putting them into DNS records.

--
Töma

---

From: David Schinazi <[email-masked]>
Date: Thu, 15 Jul 2021 10:10:20 -0700

Chrome doesn't plan on having multiple connections share a UDP source port.
In part because sharing would require the overhead of client connection
IDs, but mainly the fact that switching away from the
one-port-per-connection model we have today is work towards solving a
problem that we don't have.

Back to Mark's original email, I think there's value in writing an RFC to
document this behavior - that way clients can have code that requests a
different source port from the OS if it initially assigns one that is
likely to be blocked by servers. And while modifying NATs is a long game,
RFCs do tend to move the needle over time.

David

On Thu, Jul 15, 2021 at 6:40 AM Nick Banks <nibanks=
[email-masked]> wrote:

> You should definitely not make any assumptions around having unique source
> ports for QUIC connections. MsQuic already supports sharing the local port
> and is looking to make it a default behavior (for at least some scenarios)
> to avoid the fairly common "port exhaustion" problems we see with TCP. This
> doesn't necessarily mean all connections would be on the same source port,
> but only a few ports might be used for all connections.
>
> Thanks,
> - Nick
>
> Sent from Outlook <http://aka.ms/weboutlook>
> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Töma Gavrichenkov <
> [email-masked]>
> *Sent:* Thursday, July 15, 2021 6:36 AM
> *To:* Mikkel Fahnøe Jørgensen <[email-masked]>
> *Cc:* Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; HTTP
> Working Group <[email-masked]>
> *Subject:* Re: UDP source ports for HTTP/3 and QUIC
>
> Peace,
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Thu, 15 Jul 2021 22:03:22 +0300

Peace,

On Thu, Jul 15, 2021, 4:40 PM Nick Banks <[email-masked]> wrote:

> ports for QUIC connections.
>

Ephemeral port randomization-based traffic balancing is not what I assume,
it's what, maybe, half the Internet connectivity providers do for a
decade.  RFC 6438 kinda sums it up, and I urge you to read it.

To distribute the load across multiple paths, an Internet router needs an
entropy source providing a random integer stable for a connection (if you
want to avoid delays and retries), generally different for different
connections (if you want to avoid seeing all the connections congested on a
single paths), available for reading in every packet.  Now, that's the
ephemeral source port.

Compared to a protocol X which ditches the ephemeral port, TCP would
probably work faster and more reliable along a lot of the network paths,
because it won't be always taking paths already congested by the protocol X.

That, of course, assumes the protocol X is popular. Many VPN protocols also
suffer from that but their general (negligible) Internet-scale adoption
state kinda saves the day.  For the HTTP protocol that would be different.

This doesn't necessarily mean all connections would be on the same source
> port, but only a few ports might be used for all connections.
>

*That* *might* work but the fixed source port won't.

--
Töma

---

From: Erik Nygren <[email-masked]>
Date: Thu, 15 Jul 2021 19:25:00 -0400

On Wed, Jul 14, 2021 at 8:21 PM Mark Nottingham <[email-masked]> wrote:

> It's not uncommon for servers to block certain UDP source ports to avoid
> being overwhelmed by certain reflection attacks. In particular:
>
> * 53 - DNS
> * 123 - NTP
> * 1900 - SSDP
> * 5353 - mDNS
> * 11211 - memcached
>
> See, eg., <https://blog.cloudflare.com/reflections-on-reflections/>. This
> isn't done to avoid protocol vulnerabilities as such -- it's to avoid
> volumetric attacks (usually DDoS).
>
>

Closely related, the Fetch spec's "bad port" list is fairly TCP-specific
and could likely
use additions for some of these.  I opened
https://github.com/whatwg/fetch/issues/1268
to track that.

    Erik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 16 Jul 2021 10:27:20 +1000

Adding source ports to Fetch to complement their list of "bad" destination ports is interesting, but probably not sufficient -- NAT and CGNAT vendors tend not to read that document.

If we believe both that this should be written down and that more reflection-capable applications are going to emerge, we should probably put this information in a registry.

It strikes me that it could be done by adding a column to the port registry <https://www.iana.org/assignments/service-names-port-numbers/>. In other words, TSVWG. Thoughts?

Cheers,

> On 16 Jul 2021, at 9:25 am, Erik Nygren <[email-masked]> wrote:
> 
> 
> Closely related, the Fetch spec's "bad port" list is fairly TCP-specific and could likely
> use additions for some of these.  I opened https://github.com/whatwg/fetch/issues/1268
> to track that.
> 
>     Erik
> 
>

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Jul 2021 03:40:10 +0200

Inline

On Thu, Jul 15, 2021 at 11:29:37AM +0200, Willy Tarreau wrote:
> here other than being compatible with UDP.

Huh ?? The UDP source port serves as the UDP destination port for
return packets. Otherwise how do you redirect the packet to he
right socket ? 

> and source port for outgoing communication. 

Sure, but different UDP source port (socket) for different applications.

> This will also lower the

Yes, especially for systems with many simultaneous connections,
likely servers/responders.

Unless scale is a problem its somewhat hard to say whats better or worse.
Using separate sockets/port-numbers even within a sinle app allows
to distinguish the connections easier with the usual suspsects such
as per-socket kernel diagnostics, network tools such a IPfix or the
like. If i where to implement a stack for clients i'd certainly
wanted to offer the option of whether to allocate separate sockets
(source ports) per connection or not - and see what users do with it.

Toerless

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2021 08:06:44 +0200

> return packets. Otherwise how do you redirect the packet to he

That's exactly what I meant by "compatible with UDP", i.e. a datagram
oriented, bidirectional transport protocol. The source port is solely
used to know what socket to deliver the return traffic to.

> Sure, but different UDP source port (socket) for different applications.

Sure :-)

> Unless scale is a problem its somewhat hard to say whats better or worse.

Stefan made a good point about the problem that might result, with
inbound load balancing between multiple listeners (typically what's
achieved by L3 switches doing L3+L4 hash between multiple servers,
and operating systems hashing the source+destination port to pick a
different listening socket). Thus a suggestion might be to possibly
save resources by using a small amount of sockets, with "small" left
to the appreciation of the implementation.

> like.

Sure, but at the application level it does require to check the
connection ID anyway.

> If i where to implement a stack for clients i'd certainly

Yep, at least as a debug option to help with captures or to trace
file descriptors activities.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 16 Jul 2021 06:34:32 +0000

--------

> to the appreciation of the implementation.

We should run the question "few or many UDP ports?" past some some
friendly 100G and 400G device driver maintainers.

The NICs I have been working with for the ESO ELT project all
included UDP ports in the hash they used to decide which CPU core
to deliver packets/interrupts to and we had to spread across UDP
ports to or all the traffic would hit one single core.

I dont know if QUIC has registered with the NIC designers and device
drivers writers yet, but given the opacity of QUIC packets, it is
very hard to see what else than the UDP port they can feed into
their hash, so I expect the answer to be "many".

But it would best to ask, and find out how important they think it
will be, and what "many" means for them.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2021 09:16:40 +0200

> their hash, so I expect the answer to be "many".

That's true but queues are not the only parameter, as using many
sockets also results in a massive cache miss ratio and many more
syscalls. Even with TCP it's quite visible that using a few hundred
connections tends to be incredibly more efficient at 100Gbps than
using tens of thousands.

Ideally we want all queues to be used with a good balance while
limiting the amount of operations to transport these data.

Probably that some wording evocating that possibility offered by
the protocol and a suggestion to take various factors into account
(L4 LB, NIC queues, listeners, syscalls etc) is sufficient to let
implementors ask the relevant people for advice in their context.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 16 Jul 2021 09:51:01 +0200

> Am 16.07.2021 um 09:16 schrieb Willy Tarreau <[email-masked]>:
> 

It seems a good idea to actively pull in people knowledgable in 
these implementation and hear their recommendations/expectations.

As far as my limited knowledge goes, having a QUIC application
share a single source port for many QUIC connections does not
pose any problems for servers.

The concern is more about NAT (and esp. CGNAT) to coalesce
many Source:address+port into a few NAT-address+ports,
using QUIC connection IDs. On the extreme, once these NATs
use connection IDs, they could theoretically live with a single
NAT port. That would not work well with any UDP load balancing.

The advice for QUIC aware NAT implementations could then be
something like (IANAUBE):
- distribute QUIC connections across a minimum of N ports
- The N ports would ideally be random or a complete interval
  (they should not all give the same number module a prime)
- distribute new NATed connections in some random fashion
  or round-robin across your port pool.

- Stefan

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Jul 2021 23:53:11 +0200

> to the appreciation of the implementation.

That would be an example getting more resources on a load-balanced
server farm when you use multiple 5-tuple flows to the same
(cluster) server by reson of client UDP port. 

The same (more/different) resource allocation could also occur for
network bandwidth along the path when there is any type of per-flow
traffic operation, even if heuristically, such as attempts for
bandwidth fairness. Or even alternative paths choosen when there
is 5-tuple load-spliting along the path.

We have been around this block pre-HTTP2.0, using multiple
sockets to increase performance. But any HTTP2.0 fixes did
not change the network or server-farm aspects.

> Sure, but at the application level it does require to check the

This again would be on the network side.

Cheers
    Toerless

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 17 Jul 2021 10:28:09 +1000

I appreciate the enthusiasm for the tangential here :) but to return the original topic -- I realised that this is obliquely discussed in the applicability draft:
  https://quicwg.org/ops-drafts/draft-ietf-quic-applicability.html#name-port-selection-and-applicat

So at a minimum, it seems like we should expand upon that text to make this issue more clear. I'll try to start a PR.

If I read the thread correctly, people seem to think (somewhat pessimistically, but realistically) that protocols susceptible to reflection will continue to be created, so hardcoding a list into the RFC isn't workable.

Instead, I'm currently thinking the best approach will be to:

1. Expand the applicability document as per above, using examples from the list I gave, since they're already pervasive
2. Start a discussion in the TSVWG about the possibility of adding a new column to the port registry to capture this information 

Make sense?

---

From: Töma Gavrichenkov <[email-masked]>
Date: Wed, 28 Jul 2021 00:01:54 +0300

Peace,

On Sat, Jul 17, 2021, 3:28 AM Mark Nottingham <[email-masked]> wrote:

> I appreciate the enthusiasm for the tangential here :) but to return the
> original topic -- I realised that this is obliquely discussed in the
> applicability draft:
>
> So at a minimum, it seems like we should expand upon that text to make
> this issue more clear. I'll try to start a PR.
>
> If I read the thread correctly, people seem to think (somewhat
> pessimistically, but realistically) that protocols susceptible to
> reflection will continue to be created, so hardcoding a list into the RFC
> isn't workable.
>
> 1. Expand the applicability document as per above, using examples from the
> list I gave, since they're already pervasive
> 2. Start a discussion in the TSVWG about the possibility of adding a new
> column to the port registry to capture this information
>

What if we add a new key to the DNS SVCB record so that an HTTP/3-enabled
service could indicate the source port ranges it does [not] want or is
[not] able to see?

The HTTP/3 servers would probably be using SVCB anyway for a number of
applications.

--
Töma