Subject: DoS-ing QUIC servers is too easy

From: [email-masked]
Date: Fri, 19 Jul 2019 15:14:09 +0000

Problem:
QUIC servers, like just about all other servers, are far too susceptible to DoS attacks.  Integral to the client-server architecture is the fact that servers must listen to all potential clients on the network(s) they serve – whether the connection is from a legitimate client or not. This was fine when IT controlled the network, but the scope of the Internet changed all that. Servers listening to all communications enables resource DoS attacks. Resource DoS attacks on QUIC servers are amplified by the processing requirements of public/private key cryptography which an attacker a can induce the server to do without burdening the attacker.  In addition, servers responding to all queries can leak information that adversaries can later use to attack them.

Requirement:
Provide a way that QUIC servers can easily, with very low processing requirement, tell if a connection is from a legitimate client.

Brent Bliger
Director-Technology Verizon SDP
[email-masked]<mailto:[email-masked]>

---

From: Frederick Kautz <[email-masked]>
Date: Fri, 19 Jul 2019 08:26:56 -0700

What techniques are common in standard TCP/UDP stacks for both kernel and
userspace networking scenarios?

On Fri, Jul 19, 2019, 08:16 <brent.bilger=[email-masked]>
wrote:

> Problem:
> QUIC servers, like just about all other servers, are far too susceptible
> to DoS attacks.  Integral to the client-server architecture is the
> fact that servers must listen to all potential clients on the network(s)
> they serve – whether the connection is from a legitimate client or
> not. This was fine when IT controlled the network, but the scope of the
> Internet changed all that. Servers listening to all communications enables
> resource DoS attacks. Resource DoS attacks on QUIC servers are amplified by
> the processing requirements of public/private key cryptography which an
> attacker a can induce the server to do without burdening the attacker.  In
> addition, servers responding to all queries can leak information that
> adversaries can later use to attack them.
>
> Requirement:
> Provide a way that QUIC servers can easily, with very low processing
> requirement, tell if a connection is from a legitimate client.
>
>
>
> Brent Bliger
> [email-masked]

---

From: Ian Swett <[email-masked]>
Date: Fri, 19 Jul 2019 11:31:26 -0400

> requirement, tell if a connection is from a legitimate client.

QUIC's Retry token is designed for this purpose.  I believe that should be
sufficient and make QUIC equivalent to TCP.

Also, QUIC requires much larger Initial packet sizes than a TCP SYN,
increasing the amount of work an attacker has to expend.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 19 Jul 2019 17:53:23 +0200

> userspace networking scenarios?

For TCP, typically SYN cookies, which can easily be offloaded very low
into the stack or even on external devices so that multi-10G at line
rate is easily achieved on commodity hardware.

Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 19 Jul 2019 08:59:36 -0700

QUIC v1 does have some mitigations such as amplification attack prevention
and mostly avoiding disconnect on bad input.

If there are multiple server endpoints, a load balancer SHOULD choose an
arbitrary endpoint that is not dependent on client input. This prevents
targeting a single server during the expensive public key handshake.

An endpoint can also issue a RETRY request. This forces the client to
confirm that traffic can be routed back to it before more resources are
invested in the handshake. Outside of QUIC this can be used to learn about
bad addresses since the number of source IP’s are now limited to actual
attackers, so rate limiting can be applied.

If the client can be controlled, such as in a client library on a phone
handset, or an IoT device, there are more options. These would in principle
amount to a new custom QUIC version, but it is not strictly necessary:

When a connection is established, a random OCID (original connection ID) is
sent to what ever endpoint care to answer when traffic is sent towards a
server address as stated before. The OCID is NOT used to route traffic to
prevent attacks. However, if client generates the OCID not randomly but
from some shared key crypto scheme, then a load balancer can quickly filter
out invalid traffic, except for replay attacks. If a time factor is also
encoded, such replays can be limited.

I guess the folks at CloudFlare can speak a lot more about that.

But a common practice is to route traffic through a gateway service.
Normally that traffic just passes through to the endpoint or cluster of
endpoints. This means the DNS points to the gateway, shall we say,
midpoints.

If an attack is detected, the gateway starts to create a maze of new
midpoints with suitably updated DNS records that can be disorienting to an
attacker that is trying to lock on to one single endpoint in order to
overload it. The gateway can then forward traffic to the endpoints as
before since the attack is becoming diluted and possibly ending blind.

At the same time the gateway may start to learn about the source of the
attack and filter out traffic more or less brutally for the greater good,
for example taking out an entire country or continent. Since DDoS attacks
are distributed, this can be difficult to achieve at a granular level, if
at all. At least the gateway service can drop excess traffic instead of
overloading resources such as auto-scalers or triggering crashes due to
memory exhaustion.

Machine learning probably pays a part in this. Such attack prevention
mechanisms are far to complex to handle at the individual endpoint level.

That said, it would be possible to place more emphasis on client
authentication, and there is support in TLS and QUIC, but it is not
practical for general browser deployment, and for attack prevention the
detection may come to late, if only because it has a cost to validate the
clients.

Mikkel

On 19 July 2019 at 17.27.23, Frederick Kautz ([email-masked]) wrote:

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 19 Jul 2019 18:04:32 +0200

Hi Ian,

> sufficient and make QUIC equivalent to TCP.

The important difference is that only the QUIC stack can process this,
so the packet has to pass all the way through the whole stack to userland,
the protection cannot be offloaded to lower layers nor to an external
device. This remains a pretty big concern to me.

> increasing the amount of work an attacker has to expend.

Sure but not that much. Even with an initial packet as large as 200 bytes
you still feed the whole stack at 5 million packets per second on a single
10G port. This is a lot to pass through the whole stack, not counting the
potential crypto that is associated with it nor building the response, of
course. But we'll see.

Willy

---

From: Ian Swett <[email-masked]>
Date: Fri, 19 Jul 2019 13:47:19 -0400

> device. This remains a pretty big concern to me.

There's a proposed draft(draft-duke-quic-load-balancers
<https://datatracker.ietf.org/doc/draft-duke-quic-load-balancers/>) on QUIC
load balancers, so I don't think the above is necessarily true.

> potential crypto that is associated with it nor building the response, of
The minimum is 1200 bytes FYI.

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 19 Jul 2019 14:10:29 -0400

> device. This remains a pretty big concern to me.

The design only requires cooperation between the device that generates the Retry and the eventual endpoint.  That doesn't require heavy processing before offload.  As Ian says, there is work going on to formalize a protocol between the two, but this could be an extremely simple process if you want to keep the two at arms length.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 20 Jul 2019 03:07:12 +0300

Peace,

On Fri, Jul 19, 2019 at 6:31 PM Ian Swett
<ianswett=[email-masked]> wrote:
> QUIC's Retry token is designed for this purpose..
> I believe that should be sufficient and make QUIC
> equivalent to TCP..

For a server, yes.

However, mitigating a Server Hello DDoS attack towards a _client_ is
very complicated (if even possible) with QUIC, as opposed to TCP with
SYN/ACK floods.
We've discussed that during the hackathon in Prague (I don't remember
if you were there or not).  Possible solution with respect to the
current QUIC threat model and architecture probably has something to
do with the likes of potential DOTS integration or something like
that.

> Also, QUIC requires much larger Initial packet sizes than a TCP SYN, increasing the amount of work an attacker has to expend.

No, there's no increase.

What we see in practice are combined UDP+SYN flood incidents where
proper SYN packet rates (w/o fast open) are roughly equal to UDP
packet rates but the bandwidth consumed by SYN segments is an order of
magnitude lower than the UDP bandwidth (without amplification).
Therefore, currently the bottleneck for DDoSers is the packet rate,
not the bandwidth, and the initial packet size growth won't help here.

--
Töma

---

From: William Herrin <[email-masked]>
Date: Fri, 19 Jul 2019 17:57:18 -0700

> do with the likes of potential DOTS integration or something like

Would you elaborate on this? Considering that the quic client employs an
ephemeral UDP port, what is the quic-specific threat model here? I.e. how
does it differ from any other client with a handful of open ephemeral UDP
ports?

Thanks,
Bill Herrin

-- 
William Herrin
[email-masked]
https://bill.herrin.us/

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 20 Jul 2019 07:06:47 +0200

Hi Martin!

On Fri, Jul 19, 2019 at 02:10:29PM -0400, Martin Thomson wrote:
> The design only requires cooperation between the device that generates the
> Retry and the eventual endpoint.  That doesn't require heavy processing
> before offload.  As Ian says, there is work going on to formalize a protocol
> between the two, but this could be an extremely simple process if you want to
> keep the two at arms length.

I've read Martin's proposal when he sent it a few months ago. But honestly
I think the proposal tries hard to work around some protocol weaknesses
that were purposely created to prevent linkability, and I remain convinced
that all of this is not heading in the right direction.

I think that another approach would be to let the server ask enforce some
verifiable computational cost on the client. For example, an initial packet
could contain a 32-bit nonce and a 32-bit hash of the header+nonce. The
server would demand that the hash starts with a given number of zeroes
(i.e. the hash is inferior to a limit). The default value could for example
be 8 zeroes, meaning the client on average needs to try 256 different nonces
before finding a valid hash. For the server, this means that when an initial
packet comes, it just has to check the hash. If it doesn't match, no need to
parse the packet nor to even reply. If the server is under heavy load or
attack, it could send back a retry packet indicating how many zeroes it
expects in the hash (e.g. try again with at least 12 zeroes, or 4096 tries).
Just like an MTU, the expected number of zeroes for a server could be saved
locally for future connections.

And what is nice with such a method is that it becomes easy to deal with
DDoS : all a front equipment has to do is to:
  - check incoming hashes and drop packets which do not match ;
  - take a fingerprint of the headers or just the hash itself, and
    keep a packet counter for such fingerprint over a period, and drop
    those that are too frequent (so that precomputed hashes cannot be
    used to form an attack)
  - optionally respond with a retry packet informing about the expected
    hash value, or just let valid packet with insufficient hashes pass
    through since there will be much less of them (typically 1/256 for
    random packets).

A very inexpensive hash for small messages should be used for this, it
doesn't even need to be very secure, it just needs not to be reversible
(i.e. not a CRC32). Possibly that even a MAC algorithm like Poly1305
could be a good candidate for example as it's inexpensive for small
devices and would be cheap on servers to validate messages as they
arrive. What matters is that verifying small messages is extremely
fast, much faster than the time it takes to serialize a packet.

The default number of leading zeroes would just need to be calculated
so that there is no impact on smallest devices used for browsing nor
on proxies. On the server side, gateways could be configured to use a
much lower value in agreement with the servers. And similarly a server
working well could advertise in responses "no need to spend your time
hashing too much, I'm fine with X zeroes".

But typically if a regular 4-core hacked hosted server on the net cannot
compute more than a few tens of thousands packets per second, we'll
greatly reduce the risk of DDoS compared even to TCP today (which has
become easy to deal with).

Just my two cents,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 20 Jul 2019 07:08:56 +0200

Hi Ian,

> load balancers, so I don't think the above is necessarily true.

Yes, as mentioned in my other message I've already seen this one.

> The minimum is 1200 bytes FYI.

My understanding was that it was the minimum supported so that it should
be the maximum used by clients. i.e. a valid initial packet can be much
smaller than this. But I could be mistaken as I've had a hard time keeping
up with the progress made on the protocol.

Thanks,
Willy

---

From: Watson Ladd <[email-masked]>
Date: Fri, 19 Jul 2019 22:53:37 -0700

> locally for future connections.

There is an implicit assumption here that special devices are needed
to deal with DDoS attacks. That assumption is wrong. See
https://blog.cloudflare.com/meet-gatebot-a-bot-that-allows-us-to-sleep/
for an approach that is far more capital efficient. You can drop
oodles and oodles of packets in eBPF filters. If you want to go to
kernel bypass you might be able to drop even more, and line rate FPGA
is doable at 40Gbits. Cloudflare doesn't do that, but I've talked to
some organizations that have for some other services.

Now let's consider the cost of responding to a QUIC ServerHello. It is
one signature and one ECDHE exchange. One core can do 500,000 of those
a second. A beefy server can devote two cores to the task of handling
the flood of a million packets per second, and the other 22 to other
miscellany.

Are you sure there is a problem here?

Sincerely,
Watson Ladd

---

From: Watson Ladd <[email-masked]>
Date: Fri, 19 Jul 2019 23:09:16 -0700

> the flood of a million packets per second, and the other 22 to other

Unfortunately I was off by three orders of magnitude (prefix
confusion).  A core can only do 7,000 of these per second, so you
would need some more servers to handle the million packet flood with
complete responses (vs. filtering the attacking machines). The
conclusion that commodity hardware can filter DDoS beyond that rate I
do stand by.

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 20 Jul 2019 09:37:00 +0300

On Sat, Jul 20, 2019 at 3:57 AM William Herrin <[email-masked]> wrote:
> Would you elaborate on this? Considering that the quic client employs
> an ephemeral UDP port, what is the quic-specific threat model here?

That's a big assumption already.  Another (implicit) assumption is
that the ephemeral port generation algorithm built into a device
implements randomness properly (which is especially complicated in
IoT).

https://tools.ietf.org/html/draft-gont-numeric-ids-generation-04

But anyway, ephemeral port number consists of just 15 bits and is
prone to brute force.  In TCP, we had initial sequence number which
could've been changed by a device near the border of the network,
along with the port, to ensure those are random enough, and that's
already 47 bits.  In QUIC, I don't see any way to reliably protect the
network border from such incidents without just blackholing the
device.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 20 Jul 2019 09:51:16 +0300

On Sat, Jul 20, 2019 at 9:09 AM Watson Ladd <[email-masked]> wrote:
> A core can only do 7,000 of these per second, so you
> complete responses (vs. filtering the attacking machines).

You would need multiple servers anyway, because the world has seen
incidents over 1 Tbps (and even 400GbE is still just not there by the
way).  For instance, SYN flood attack seen by Cloudflare in April 2018
topped 942 Gbps with approximately 1000-octet packets (which is also
an evidence bandwidth isn't a bottleneck for SYN floods, packet rate
is).

https://blog.cloudflare.com/the-rise-of-multivector-amplifications/

You cannot handle that in a single machine.

--
Töma

---

From: Watson Ladd <[email-masked]>
Date: Fri, 19 Jul 2019 23:57:53 -0700

> network border from such incidents without just blackholing the

Isn't the Connection ID just such a thing? Looking at
https://tools.ietf.org/html/draft-ietf-quic-transport-22#section-7.2
in conjunction with 17 the server sends a packet with the destination
ID that the client told it to use.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 20 Jul 2019 10:06:01 +0300

On Sat, Jul 20, 2019 at 9:58 AM Watson Ladd <[email-masked]> wrote:
> Isn't the Connection ID just such a thing?

No. Could be zero.

Also, the issue with predictable IDs is still there.

--
Töma

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 20 Jul 2019 00:26:11 -0700

On 20 July 2019 at 07.07.06, Willy Tarreau ([email-masked]) wrote:

before finding a valid hash.

You need to make that hash a hmac or similar with a pre-shared key, or an
extra roundtrip, otherwise you just use a precomputed table.  With a
roundtrip this would be an extension to Retry, with a pre-shared key it is
somewhat similar to my earlier email where you sign the OCID, but here you
don’t need proof-of-work, only proof-of-ownership (of the key).

Mikkel

---

From: [email-masked]
Date: Sat, 20 Jul 2019 15:36:11 +0000

On Jul 20, 2019, at 12:26 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

On 20 July 2019 at 07.07.06, Willy Tarreau ([email-masked]<mailto:[email-masked]>) wrote:

You need to make that hash a hmac or similar with a pre-shared key, or an extra roundtrip, otherwise you just use a precomputed table.  With a roundtrip this would be an extension to Retry, with a pre-shared key it is somewhat similar to my earlier email where you sign the OCID, but here you don’t need proof-of-work, only proof-of-ownership (of the key).

Mikkel

A possible approach to defeating this:

One of the nice things about UDP is that the server can consume the first data packet before replying to the Client (unlike TCP where the server must SYN/ACK before getting the first data packet). Therefore, it is possible to have a unique-to-the-client, pre-authentication (or pre-authorization) token in the first packet. The presence of such a token would identify that the client is authorized to connect to the QUIC server. The token could possibly be provided to the client by an authorization server before the session. Or, in an enterprise environment, it could be possible that a one time on-boarding could provide a lifetime of tokens.

The token could be a One Time Password such as described in RFC 4226, which uses a computationally efficient algorithm, or could be some other type of token. In some ways, one could think of this mechanism like SAML Single Sign On in that the client is pre-authenticated to connect to the server - just like the pre-authentication token could be used to pre-authorize a QUIC connection.

From the standpoint of the QUIC server, this pre-authentication token could allow a server to limit the resources it applies to the connection. If the token is not valid, it immediately drops the connection without executing resource-demanding cryptography. Further, the server becomes almost black to the network. Scans of the network would not be acknowledged by this QUIC server if the initial packet did not contain a legitimate pre-authentication token.

This approach could be subject to an eavesdropping attack if the token were static or did not prevent replay attacks, so these issues should be considered when deciding on how tokens should be created.

Brent Bliger

---

From: Watson Ladd <[email-masked]>
Date: Sat, 20 Jul 2019 10:30:26 -0700

> You cannot handle that in a single machine.

Of course not. But I don't understand the magic properties of this box
that is supposed to be doing the filtering. What actually happened is
anycast spread that attack geographically, and then ECMP spread it
over machines, and then the attacking machines were blackholed with
firewall rules determined by statistical analysis of incoming packets.
The changes to the protocol being discussed are not going to reduce
the damage 1 Tbps does to a site with a smaller connection: you are
going to need multiple links, you are going to need to have rules to
efficiently drop the attack traffic.

What is the setup where these issues are a problem? I get that you
might not want to share, but it helps think about solutions if we know
more about the problem.

As for predictability: don't use devices that can't make random numbers.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 20 Jul 2019 21:14:39 +0300

Peace,

On Sat, Jul 20, 2019 at 8:30 PM Watson Ladd <[email-masked]> wrote:
> that is supposed to be doing the filtering.

Yes, the properties of a filtering box are hardly relevant.  Why?

> What actually happened is

Statistical analysis is generally useless against SYN flood.
It was useful in the 942 Gbps case just because the attacker was dumb
enough to try to multiply the bandwidth by adding payload to a SYN
segment, thus exposing it to a simple packet length filter.

With QUIC, an attacker won't be able to make such a mistake.

> As for predictability: don't use devices that can't make random numbers.

1) Is this when we ban RIOT from participating in IETF hackathons?

https://github.com/RIOT-OS/RIOT/blob/637bbdd9856c14cb8c56a6191253363d53f54a9f/sys/random/random.c#L42
https://github.com/RIOT-OS/RIOT/blob/637bbdd9856c14cb8c56a6191253363d53f54a9f/sys/include/random.h#L37

2) This is not how companies choose MRI devices or any other
industrial IoT stuff.  I don't even think they *could* evaluate such a
criteria, but even if they could, they wouldn't.

3) Even worse, DDoS mitigation is widely believed to be a
responsibility of an access network, and I can't imagine circumstances
under which that ISP would be asked by their client if the latter
should order an expensive device or not.

--
Töma

---

From: Watson Ladd <[email-masked]>
Date: Sat, 20 Jul 2019 12:22:20 -0700

> Yes, the properties of a filtering box are hardly relevant.  Why?

It's relevant because unless I understand the setup and the
constraints, I don't understand the problem. Clearly commodity servers
can drop packets en masse. But for some reason this isn't a solution.
There are constraints here that make run lots of load balancers going
back to a few servers also not a solution. Is the ask that we need a
way to filter packets associated with flows that can be interpreted by
boxes without any setup between boxes and server?

> With QUIC, an attacker won't be able to make such a mistake.

How would your solution (which you claim works for TCP but not QUIC)
handle a SYN flood?

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 21 Jul 2019 04:21:33 +0300

Peace,

> can drop packets en masse. But for some reason this isn't a solution.

Err...?  It *is*.
(Of course, if you don't need the HTTPS private key to generate Retry.)

Special hardware can also drop packets en masse.  Sometimes it's just
the same commodity in a nice designer box.  What's the difference?

> How would your solution (which you claim works for TCP but not QUIC)

It'll do that via retries.  SYN flood isn't a big issue here, SYN/ACK flood is.

That being said, with SYN flood, we could've been dropping anything
bigger than a proper short SYN, while with QUIC, we'll have to handle
20x more traffic for the same purpose.  Like I said before, the
assumption that raising the initial packet size up to 1200 octets
helps to reduce the footprint of DDoS attacks doesn't happen to be
true.

--
Töma

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 22 Jul 2019 08:23:17 +0200

Hi,

> that is supposed to be doing the filtering.

There is no magic involved. It's just that the application servers are
often unsuited to such a task because of the amount of work involved
through the whole stack, their sizing which matches application needs
more than network bandwidth, the fact that very commonly nowadays they
run in VM and that the hypervisor itself is unable to cope with line-rate
attacks etc. In practice you need to filter packet-based attacks at the
edge and prevent them from entering the network. For some content providers
who use ECMP traffic distribution, this means having a fast packet processing
layer on the front hosts themselves (either using XDP, or by dedicating a
set of NICs and CPU cores to DPDK, or running alternatives), or just doing
it on the front load-balancers since most of the time they're very close
to the edge and they need to protect themselves.

> efficiently drop the attack traffic.

Absolutely. However 1 Tbps attacks are not that common. But certain
sites I know deal with around 10G several times a day. And while
some attacks are trivial to handle *when you have the bandwidth* (i.e.
all those that can quickly be evicted based on stateful filtering,
thus SYN-ACK, ACK, NTP/DNS amplification, etc), others are technically
simple but cost a lot (typically SYN floods). They cost a lot because
when packets appear normal you have to resort to solutions involving
responding SYN cookies, or randomly dropping and waiting for retranmits.
When you decide to send SYN cookies, it means that for 10G of traffic
inbound, you generate 10G outbound and that is damaging to your useful
trafic. When you randomly drop and wait for retransmits, it is damaging
for the user's experience.

Here what I'm seeing with QUIC is :
  - with the ability to migrate connection to a different IP address,
    existing stateful filtering will not work very well anymore until
    the connection ID is taken into account by equipments. In practice
    what will happen is that stateful filtering will continue to be used
    and only enforced during attacks. This way only users migrating during
    an attack will be affected and that will probably be acceptable to
    most sites.

  - if the only way to slow down attacks on the server side is to
    build a retry packet, this is exactly equivalent to SYN cookies, and
    it does not slow attacks down, it only allows to sort the good from
    the bad at a high cost.

Thus my suggestion to offset the packet production cost on the client in
order to limit to reasonable values. If even the most basic smartphone is
limited to 1000 connections per second, it will already be a lot. If a
hacked server participating to a botnet can do 10k connections per second,
that's trivial to deal with on the server side. Right now a gigabit connected
server can produce 1.488 million packets per second. Just take 10 of such
bots and you force the target server to 1) be able to cope with 10G of
SYN flood, and 2) produce 10G of parasitic SYN-ACK reducing the room for
useful trafic.

If we consider that a default limit to ~10k conn/s for a large client is
something acceptable to deal with during attacks, some devices will be
affected, these are intermediaries. But there are mainly two categories of
intermediaries, the server-side ones (e.g. haproxy, varnish), which are
very close to the servers or at least under the same authority, so it is
possible to configure a lower communication cost between them (i.e. in
the equipment configuration the difficulty just has to be lowered, knowing
that the server is OK with this). The other main category are on the client
side (e.g. Squid). These ones would probably learn from the most used servers
the accepted difficulty and will probably coalesce a number of connections,
thus not being too much impacted by this either.

That doesn't remove anything from the principle of the retry packet, it
just makes the retry packet needed only when the incoming one is considered
too cheap to compute. And doing this definitely allows servers to generate
the retry packets themselves without requiring that a smart front box does
all the job, because a stupid box can be configured to automatically drop
all packets with a difficulty below the default limit, and deliver only
other ones to the servers. The servers can then decide to adjust their
behaviour based on their load. But having to send a retry packet for
100% incoming packets or for 1% of them definitely is not the same job.

Regarding the risk of precomputed tables, I already thought about it and
I consider it's reasonably easy to deal with: the front filtering device
just has to implement per-signature counters and drop the too frequent
ones when under attack. Keep in mind that dealing with an attack doesn't
mean you want a perfect service and that nobody notices the problem. It
just means you're using statistical tools to maintain the highest ratio
of good traffic so that your activity is not affected by the attack, and
the attacker gives up and moves to another target.

I'm not claiming all of this is trivial, I'm just suggesting that we
don't reproduce some of the designs of the past which in a hostile
environment cause some trouble, and that we also consider modern
techniques that allow to offset costs between the attacker and the
victim. And I really think that forcing computation on the attacker is
effective. I've been doing this in the past using javascript to efficiently
slow down attacking browsers and you see that you can quickly turn an
armada of infected machines trying to assault you into an armada of
electric heaters. It may not be the nicest thing for the environment
but multiplying the size of server farms to deal with small attacks
is even worse.

Just my two cents,
Willy

---

From: [email-masked]
Date: Mon, 22 Jul 2019 14:38:53 +0000

On Jul 19, 2019, at 8:31 AM, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>> wrote:

On Fri, Jul 19, 2019 at 11:16 AM <brent.bilger=[email-masked]<mailto:[email-masked]>> wrote:
Problem:
QUIC servers, like just about all other servers, are far too susceptible to DoS attacks.  Integral to the client-server architecture is the fact that servers must listen to all potential clients on the network(s) they serve – whether the connection is from a legitimate client or not. This was fine when IT controlled the network, but the scope of the Internet changed all that. Servers listening to all communications enables resource DoS attacks. Resource DoS attacks on QUIC servers are amplified by the processing requirements of public/private key cryptography which an attacker a can induce the server to do without burdening the attacker.  In addition, servers responding to all queries can leak information that adversaries can later use to attack them.

Requirement:

QUIC's Retry token is designed for this purpose..  I believe that should be sufficient and make QUIC equivalent to TCP..

But we can make QUIC much, much better than TCP because the first packet of the UDP connection that the server receives can pre-authenticate the client. Therefore, the server could: 1) allocate very little resource to the client to check that the client is authorized to connect, 2) eliminate the process intensive crypto for unauthorized clients, and 3) reject the connection immediately instead of waiting for some timeout.

Yes, so we could optionally make use of some of that larger packet size for the pre-authentication.

Brent

Brent Bliger

Brent Bliger

---

From: Lars Eggert <[email-masked]>
Date: Mon, 22 Jul 2019 11:01:53 -0400

Hi,

> But we can make QUIC much, much better than TCP because the first packet of the UDP connection that the server receives can pre-authenticate the client. Therefore, the server could: 1) allocate ver…

how is the current Retry token mechanism not fulfilling this goal?

Thanks,
Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 22 Jul 2019 08:19:15 -0700

The Retry token forces the infrastructure to send data instead of black
holing.

Why don’t we look into optionally signing long OCIDs that can be used if
under attack and otherwise just serve as dumb randomness.

The 0-RTT token could also be used in case of an attack.

Mikkel

On 22 July 2019 at 17.02.22, Lars Eggert ([email-masked]) wrote:

Hi,

On 2019-7-22, at 10:38, brent.bilger=[email-masked] wrote:
> But we can make QUIC much, much better than TCP because the first packet
of the UDP connection that the server receives can pre-authenticate the
client. Therefore, the server could: 1) allocate very little resource to
the client to check that the client is authorized to connect, 2) eliminate
the process intensive crypto for unauthorized clients, and 3) reject the
connection immediately instead of waiting for some timeout.

Thanks,
Lars

---

From: Roberto Peon <[email-masked]>
Date: Mon, 22 Jul 2019 17:10:43 +0000

When you’re talking about pre-authentication, are you referring to the 0-rtt stuff?
If so, then I agree-- biasing traffic to clients that have successfully connected in the past is exactly what every SYN-flood-like attack mitigation does at its core (though ‘past’ may be pretty recent in the case of syn-cookies).
-=R

From: QUIC <[email-masked]> on behalf of "brent.bilger=[email-masked]" <brent.bilger=[email-masked]>
Date: Monday, July 22, 2019 at 10:40 AM
To: Ian Swett <ianswett=[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [E] Re: DoS-ing QUIC servers is too easy

Requirement:

Brent

Brent Bliger

Brent Bliger

---

From: [email-masked]
Date: Tue, 23 Jul 2019 00:10:47 +0000

On Jul 22, 2019, at 8:01 AM, Lars Eggert <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

On 2019-7-22, at 10:38, brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]> wrote:
But we can make QUIC much, much better than TCP because the first packet of the UDP connection that the server receives can pre-authenticate the client. Therefore, the server could: 1) allocate very little resource to the client to check that the client is authorized to connect, 2) eliminate the process intensive crypto for unauthorized clients, and 3) reject the connection immediately instead of waiting for some timeout.

If every server always requested a Retry token, it would be able to validate that the source ID is not being spoofed. However,

  1.  It adds a round trip time to the connection (a stated goal is to eliminate this)
  2.  it does not prove that the client is authorized to access the client,
  3.  it requires that the server maintain a resource allocation for the client until the client replies (which maybe never),
  4.  it enables scanning the internet to find QUIC servers, and
  5.  it leaks information about the server - information that may be used by an adversary to attack the server (e.g., that it is a certain Windows server that is vulnerable to some future Eternal Blue vulnerability).

Regards,
Brent

Thanks,
Lars

Brent Bliger

---

From: [email-masked]
Date: Tue, 23 Jul 2019 00:15:07 +0000

On Jul 22, 2019, at 8:19 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

The Retry token forces the infrastructure to send data instead of black holing.

Why don’t we look into optionally signing long OCIDs that can be used if under attack and otherwise just serve as dumb randomness.

Yes, and with a slightly modified token we can eliminate the attack by an unauthorized user on the server on the first packet eliminating the need to spend resources on clients that will eventually be rejected.

Brent

Mikkel

On 22 July 2019 at 17.02.22, Lars Eggert ([email-masked]<mailto:[email-masked]>) wrote:

Hi,

Thanks,
Lars

Brent Bliger

---

From: [email-masked]
Date: Tue, 23 Jul 2019 00:28:40 +0000

On Jul 22, 2019, at 10:10 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:

0-RTT is good stuff. It reduces time for a device that’s already known to be authorized to connect to the server.

Part of the problem is to stop resource DoS. The other part is about information leakage.

Let’s assume a time when all website use QUIC and let's take an adversary who has just learned about a vulnerability in Wordpress. The adversary wants to scan the Internet finding Wordpress sites. Now, if the QUIC connection required pre-authentication, then the server would be black to the connection and the adversary could not determine if the site uses Wordpress. But, as it stands today, in this future time, the adversary scans the Internet, connects to the server, and is handed a Wordpress login screen. Now the adversary knows where to attack.

In the enterprise it is a little different. Companies implement reverse proxies to try to protect servers from unauthorized users. But what if the servers required a pre-authentication token. Then, reverse proxies would not be needed.

Brent

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of "brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>" <brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>>
Date: Monday, July 22, 2019 at 10:40 AM
To: Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [E] Re: DoS-ing QUIC servers is too easy

Requirement:

Brent

Brent Bliger

Brent Bliger

Brent Bliger

---

From: Watson Ladd <[email-masked]>
Date: Mon, 22 Jul 2019 21:16:02 -0700

On Mon, Jul 22, 2019 at 5:29 PM
<brent.bilger=[email-masked]> wrote:
>
>
>
> On Jul 22, 2019, at 10:10 AM, Roberto Peon <[email-masked]> wrote:
>

If connection requires preauthentication then no one can see the site, no?

> In the enterprise it is a little different. Companies implement reverse proxies to try to protect servers from unauthorized users. But what if the servers required a pre-authentication token. Then,…

So I think there are two different things here, or maybe I'm misunderstanding:
1: We have servers that have a limited number of clients that will
ever want to connect to them, and want to enable other devices to
filter the traffic, so need some mechanism that enables a middlebox to
filter.
2: We have servers generally available to the public and are concerned
that the QUIC version of a SYN flood can't be dealt with (despite
evidence to the contrary).

I don't know much about how Cloudflare (my place of employment)  deals
with 2 but hopefully some people who do/implementors can speak up
about how much of a problem it is. But 1 is I think very much a
nongoal: middlebox ossification is part of the reason we have QUIC.

Sincerely,
Watson Ladd

---

From: [email-masked]
Date: Tue, 23 Jul 2019 12:21:05 +0000

On Jul 22, 2019, at 9:16 PM, Watson Ladd <[email-masked]<mailto:[email-masked]>> wrote:

On Mon, Jul 22, 2019 at 5:29 PM
<brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>> wrote:

No middle box! Pre-authentication makes servers self-defending. Lack of pre-authentication is why we have middle boxes.

Companies and website owners have to pay for anti-DoS services from the likes of Verizon and others BECAUSE servers listen to all clients. With pre-authentication, the need to have an anti-DoS service would be greatly reduced because servers would not respond to unauthorized clients.

And the point about information leakage is important, too.

Brent

Sincerely,
Watson Ladd

Brent Bliger

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Jul 2019 13:26:21 +0000

I'm trying to understand how anything was pre-authenticated without suffering from all of the same problems.

How would such pre-authentication be done in practice, especially in scenarios where the server doesn't have a whitelist of clients, and that service desires traffic from non-malicious clients?

-=R

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: [email-masked] <[email-masked]>
Sent: Tuesday, July 23, 2019 8:21:05 AM
To: Watson Ladd <[email-masked]>
Cc: brent.bilger=[email-masked] <brent.bilger=[email-masked]>; Roberto Peon <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; Ian Swett <ianswett=[email-masked]>
Subject: Re: [E] DoS-ing QUIC servers is too easy

Brent

Sincerely,
Watson Ladd

Brent Bliger

---

From: [email-masked]
Date: Tue, 23 Jul 2019 13:58:49 +0000

There are lots of possible scenarios, so please just consider these to be examples. Let’s divide the world of servers in to 1) enterprise business-to-business websites, 2) websites meant for consumers where authentication is desirable, and 3) websites that will want to allow their users to be anonymous, therefore, pre-authentication would not apply.

Enterprise business-to-business websites could use something similar to the Software Defined Perimeter (SDP) Single Packet Authorization (SPA) protocol created by the Cloud Security Alliance. It uses the fact that clients are on-boarded to the system in a one time event to provide the clients with a shared secret key. The key is used in a counter-based HMAC calculation. Since only an authorized client can calculate the proper HMAC, the HMAC can be used for pre-authentication. Thus, the pre-authentication token could look something like this:
Client->Server: ClientID, Counter, HMAC[ClientID || Counter, Ks] where Ks is a shared secret key that both the client and server have.
On the server side, the server would recalculate the HMAC and compare it to the one provided by the client.
Note that this information is sent from the client to the server in clear text (i.e., not encrypted).
In practice, there would probably be more fields, but the important part is that the one-time on boarding enables a lifetime of pre-authentication tokens.

Websites meant for consumers would probably be handled differently - specifically, no on-boarding. That means there would be some kind of token service - similar to OAuth, but operating prior to the TCP equivalent of SYN/ACK. Perhaps CloudFlair or Google or Microsoft could invent something like this and offer it as a service???

Again, there will be other websites that will not want pre-authentication because they want to allow anonymous users to their site.

Brent

On Jul 23, 2019, at 6:26 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:

-=R

Get Outlook for Android<https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwMF-g&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=W-O0t0zwhgXHP8ujph6uz3rH0MtuaLwEr_7gE2p1X5o&m=CV-HIhbRVp-argiNrVu_DK7rONk5xt9L5r0Efmw170I&s=WXhjQ5cGfCrw6BSXdq5jydKEbOF2Pu3tjL2QZWXz6eY&e=>

________________________________
From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, July 23, 2019 8:21:05 AM
To: Watson Ladd <[email-masked]<mailto:[email-masked]>>
Cc: brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]> <brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>; Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>>
Subject: Re: [E] DoS-ing QUIC servers is too easy

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Jul 2019 14:39:30 +0000

I’d assume #1 is less likely to experience crippling SYN-based DoS attacks, and could otherwise be accomplished by limiting access in IP-space (imperfect, but we’re looking for “good-enough”, here).

.. which means that I think we should consider 2+3 only, really.

For #2 and #3, which I suspect have significant overlaps (sites may provide services both to logged-in users and users who aren’t logged in), the pre-authorization either:
  a) can be DoS’d itself.
  b) can be used to do a DoS in the future.

In case b, as an attacker, you’d get or wait for the client to authenticate, then use that authentication in a replay attack, e.g. by replicating the auth token across many hosts.
In cases where you are disallowed to replicate the auth token, e.g. because it is tied to a particular IP, you can help solve the replay attack however you’ll be biasing traffic to users whose IP is stable and who are logged in/auth’d.
In the pre-shared key scenario that was described, if any one client leaks the shared-secret, then it becomes not very useful and certainly doesn’t protect against a DoS. I’d guess that the probability of the loss of a symmetric key is proportional to the breadth of the distribution of the key.

What I’m trying to point out is that this doesn’t really seem to solve the generic DoS attack.
You’ll still need to do some heuristics/rules based upon whatever is observable, at a minimum the 5-tuple.
Getting more than the 5-tuple to be visible will take more work by the server, true, but given other design constraints, I don’t see how that is avoidable.

-=R

From: "[email-masked]" <[email-masked]>
Date: Tuesday, July 23, 2019 at 9:59 AM
To: Roberto Peon <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, Watson Ladd <[email-masked]>, "brent.bilger=[email-masked]" <brent.bilger=[email-masked]>, "[email-masked]" <[email-masked]>, Ian Swett <ianswett=[email-masked]>
Subject: Re: [E] DoS-ing QUIC servers is too easy

Brent

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 07:53:38 -0700

It is not that easy to steal a token and replay, but it can happen.

If you add a time factor such an attack will have limited value. In fact a
token need not be valid for more than about 10 minutes to account for
skewed clocks.

On 23 July 2019 at 16.41.09, Roberto Peon ([email-masked]) wrote:

I’d assume #1 is less likely to experience crippling SYN-based DoS attacks,
and could otherwise be accomplished by limiting access in IP-space
(imperfect, but we’re looking for “good-enough”, here).

For #2 and #3, which I suspect have significant overlaps (sites may provide
services both to logged-in users and users who aren’t logged in), the
pre-authorization either:
  a) can be DoS’d itself.
  b) can be used to do a DoS in the future.

In case b, as an attacker, you’d get or wait for the client to
authenticate, then use that authentication in a replay attack, e.g. by
replicating the auth token across many hosts.
In cases where you are disallowed to replicate the auth token, e.g. because
it is tied to a particular IP, you can help solve the replay attack however
you’ll be biasing traffic to users whose IP is stable and who are logged
in/auth’d.
In the pre-shared key scenario that was described, if any one client leaks
the shared-secret, then it becomes not very useful and certainly doesn’t
protect against a DoS. I’d guess that the probability of the loss of a
symmetric key is proportional to the breadth of the distribution of the key.

What I’m trying to point out is that this doesn’t really seem to solve the
generic DoS attack.
You’ll still need to do some heuristics/rules based upon whatever is
observable, at a minimum the 5-tuple.
Getting more than the 5-tuple to be visible will take more work by the
server, true, but given other design constraints, I don’t see how that is
avoidable.

-=R

*From: *"[email-masked]" <[email-masked]>
*Date: *Tuesday, July 23, 2019 at 9:59 AM
*To: *Roberto Peon <[email-masked]>
*Cc: *"[email-masked]" <[email-masked]>, Watson Ladd <
[email-masked]>, "brent.bilger=[email-masked]"
<brent.bilger=[email-masked]>, "[email-masked]" <[email-masked]>,
Ian Swett <ianswett=[email-masked]>
*Subject: *Re: [E] DoS-ing QUIC servers is too easy

There are lots of possible scenarios, so please just consider these to be
examples. Let’s divide the world of servers in to 1) enterprise
business-to-business websites, 2) websites meant for consumers where
authentication is desirable, and 3) websites that will want to allow their
users to be anonymous, therefore, pre-authentication would not apply.

Enterprise business-to-business websites could use something similar to the
Software Defined Perimeter (SDP) Single Packet Authorization (SPA) protocol
created by the Cloud Security Alliance. It uses the fact that clients are
on-boarded to the system in a one time event to provide the clients with a
shared secret key. The key is used in a counter-based HMAC calculation.
Since only an authorized client can calculate the proper HMAC, the HMAC can
be used for pre-authentication. Thus, the pre-authentication token could
look something like this:

Client->Server: ClientID, Counter, HMAC[ClientID || Counter, Ks] where Ks
is a shared secret key that both the client and server have.

On the server side, the server would recalculate the HMAC and compare it to
the one provided by the client.

Note that this information is sent from the client to the server in clear
text (i.e., not encrypted).

In practice, there would probably be more fields, but the important part is
that the one-time on boarding enables a lifetime of pre-authentication
tokens.

Websites meant for consumers would probably be handled differently -
specifically, no on-boarding. That means there would be some kind of token
service - similar to OAuth, but operating prior to the TCP equivalent of
SYN/ACK. Perhaps CloudFlair or Google or Microsoft could invent something
like this and offer it as a service???

Again, there will be other websites that will not want pre-authentication
because they want to allow anonymous users to their site.

Brent

On Jul 23, 2019, at 6:26 AM, Roberto Peon <[email-masked]> wrote:

I'm trying to understand how anything was pre-authenticated without
suffering from all of the same problems.

How would such pre-authentication be done in practice, especially in
scenarios where the server doesn't have a whitelist of clients, and that
service desires traffic from non-malicious clients?

-=R

Get Outlook for Android
<https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwMF-g&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=W-O0t0zwhgXHP8ujph6uz3rH0MtuaLwEr_7gE2p1X5o&m=CV-HIhbRVp-argiNrVu_DK7rONk5xt9L5r0Efmw170I&s=WXhjQ5cGfCrw6BSXdq5jydKEbOF2Pu3tjL2QZWXz6eY&e=>

------------------------------

*From:* [email-masked] <[email-masked]>
*Sent:* Tuesday, July 23, 2019 8:21:05 AM
*To:* Watson Ladd <[email-masked]>
*Cc:* brent.bilger=[email-masked] <
brent.bilger=[email-masked]>; Roberto Peon <[email-masked]>;
[email-masked] <[email-masked]>; [email-masked] <
[email-masked]>; Ian Swett <ianswett=[email-masked]>
*Subject:* Re: [E] DoS-ing QUIC servers is too easy

On Jul 22, 2019, at 9:16 PM, Watson Ladd <[email-masked]> wrote:

When you’re talking about pre-authentication, are you referring to the
0-rtt stuff?

0-RTT is good stuff. It reduces time for a device that’s already known to
be authorized to connect to the server.

Part of the problem is to stop resource DoS. The other part is about
information leakage.

Let’s assume a time when all website use QUIC and let's take an adversary
who has just learned about a vulnerability in Wordpress. The adversary
wants to scan the Internet finding Wordpress sites. Now, if the QUIC
connection required pre-authentication, then the server would be black to
the connection and the adversary could not determine if the site uses
Wordpress. But, as it stands today, in this future time, the adversary
scans the Internet, connects to the server, and is handed a Wordpress login
screen. Now the adversary knows where to attack.

In the enterprise it is a little different. Companies implement reverse
proxies to try to protect servers from unauthorized users. But what if the
servers required a pre-authentication token. Then, reverse proxies would
not be needed.

So I think there are two different things here, or maybe I'm
misunderstanding:
1: We have servers that have a limited number of clients that will
ever want to connect to them, and want to enable other devices to
filter the traffic, so need some mechanism that enables a middlebox to
filter.

No middle box! Pre-authentication makes servers self-defending. Lack of
pre-authentication is why we have middle boxes.

Companies and website owners have to pay for anti-DoS services from the
likes of Verizon and others BECAUSE servers listen to all clients. With
pre-authentication, the need to have an anti-DoS service would be greatly
reduced because servers would not respond to unauthorized clients.

Brent

Sincerely,
Watson Ladd

Brent Bliger

[email-masked]

Brent Bliger

[email-masked]

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Jul 2019 15:01:16 +0000

With short timeouts for the auth-token you have a requirement to authenticate often, which means that a significant percentage of the population of clients/users will be unable to connect when the authentication endpoint is DoS’d.

If, ultimately, we keep needing to look at 5-tuples to solve the DoS, then why have the additional complexity of an additional service?
-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tuesday, July 23, 2019 at 10:54 AM
To: Roberto Peon <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: Watson Ladd <[email-masked]>, "[email-masked]" <[email-masked]>, "brent.bilger=[email-masked]" <brent.bilger=[email-masked]>, Ian Swett <ianswett=[email-masked]>
Subject: Re: [E] DoS-ing QUIC servers is too easy

If you add a time factor such an attack will have limited value. In fact a token need not be valid for more than about 10 minutes to account for skewed clocks.

On 23 July 2019 at 16.41.09, Roberto Peon ([email-masked]<mailto:[email-masked]>) wrote:
I’d assume #1 is less likely to experience crippling SYN-based DoS attacks, and could otherwise be accomplished by limiting access in IP-space (imperfect, but we’re looking for “good-enough”, here).

-=R

From: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, July 23, 2019 at 9:59 AM
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Watson Ladd <[email-masked]<mailto:[email-masked]>>, "brent.bilger=[email-masked]<mailto:[email-masked]>" <brent.bilger=[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>>
Subject: Re: [E] DoS-ing QUIC servers is too easy

Brent

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 08:05:33 -0700

No that is not the point.

You send a token with the initial packet (which is large anyway). It is not
thought to be a high quality authentication token - for that you can use
TLS client auth. But you need to pre-qualify before wasting handshake
resources.

It is intended to get priority over other packets on a load balancer the
experience high load, and only for initial packets. After that, the load
balancer can track existing sessions either by remembering them, or by
algorithmically checking the server assigned CID on subsequent packets.

A load balancer may choose to drop or down prioritize intial packets that
do not have a signed token.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 23 July 2019 at 17.01.26, Roberto Peon ([email-masked]) wrote:

With short timeouts for the auth-token you have a requirement to
authenticate often, which means that a significant percentage of the
population of clients/users will be unable to connect when the
authentication endpoint is DoS’d.

If, ultimately, we keep needing to look at 5-tuples to solve the DoS, then
why have the additional complexity of an additional service?
-=R

*From: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Date: *Tuesday, July 23, 2019 at 10:54 AM
*To: *Roberto Peon <[email-masked]>, "[email-masked]" <
[email-masked]>
*Cc: *Watson Ladd <[email-masked]>, "[email-masked]" <[email-masked]>,
"brent.bilger=[email-masked]" <brent.bilger=
[email-masked]>, Ian Swett <ianswett=
[email-masked]>
*Subject: *Re: [E] DoS-ing QUIC servers is too easy

-=R

Brent

-=R

Get Outlook for Android

Brent

Sincerely,
Watson Ladd

Brent Bliger

[email-masked]

Brent Bliger

[email-masked]

---

From: [email-masked]
Date: Tue, 23 Jul 2019 15:08:13 +0000

On Jul 23, 2019, at 7:39 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:

Not so, enterprise is where the attraction is today. There’s DoS on sites that are “moving to the cloud”, but also there is information leakage derived from just connecting to the server via TCP/IP. The leakage often determines the type of server and often what applications cold be running on it. Take not petra. Scanning the internet and finding a windows server the malware attempted the eternal blue vulnerability. If those servers were QUIC with pre-authentication the attack would have been defeated. And the same on the internal network. The Zero Trust model is the new buzz term in security. The approach recommended is to isolate servers and require pre-authorization prior to enabling connectivity.

A common  philosophy of people in the security industry is, “put all your eggs in one basket and protect the basket really, really well.” Now I understand this is a contrary to the philosophy of networking people where everything is about distributed redundancy, but there are very good reasons for the security philosophy.

I wasn’t very complete in providing the details of Single Packet Authorization. You may want to read the specification for the full details, but the quick of it is that Single Packet Authorization is based on RFC 4226 which is Hashed One Time Password with replay prevention via a counter that is incremented each time it is used. And the other part you should know is that each client would have its own secret key such that stealing a key is the compromise of just one client.

Again, the above is just an example. If the RFC contained the concept of pre-authentication, there would be plenty of really good ideas on the topic.

Brent

-=R

Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Watson Ladd <[email-masked]<mailto:[email-masked]>>, "brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>" <brent.bilger=[email-masked]<mailto:brent.bilger=[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>>

Brent

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 08:14:46 -0700

And, if you don’t like load balances doing the job, the endpoint can do it
too.

The endpoint already checks the connection ID for non-initial packets, so
it can drop bad traffic very fast, except for the handshake. A signed token
would allow it to avoid sending Retry or initiate handshake unless it is
lightly loaded.

On 23 July 2019 at 17.05.33, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Kind Regards,
Mikkel Fahnøe Jørgensen

-=R

Brent

-=R

Get Outlook for Android

Brent

Sincerely,
Watson Ladd

Brent Bliger

[email-masked]

Brent Bliger

[email-masked]

---

From: [email-masked]
Date: Tue, 23 Jul 2019 15:19:03 +0000

On Jul 23, 2019, at 7:53 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

Yes, it requires compromising a client and stealing the key or just using the key from a back door. In practice the latter may be advantageous to the adversary because the source of the attack will be from a normal location and because future device authorizations may include a fingerprint of the device - a fingerprint that would fail on the stolen key. But look at what we’ve done to the unauthorized adversary with the pre-authentication token - defeated resource DoS, information leakage, all vulnerabilities of the server, all configuration errors of the applications of the server. And it augments TLS to defeat man-in-the-middle attacks.

As noted on a different posting, Single Packet Authorization is based on RFC 4226 which has both a counter and a time based approach. Personally, I prefer a counter.

But again, this is all irrelevant because I’m not suggesting any one pre-authentication. I’m just using an example to show how it can stop multiple types of attacks.

Brent

-=R

Brent

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 08:30:50 -0700

I believe that we largely agree, but a counter is not very good from a
distributed perspective. I’d consider it a non-starter.

I do have a proposal for QUIC v2 about preventing replay attacks on 0-RTT
that involves something similar - but 0-RTT is sort of guaranteed to land
on the same server based on routing on said token.

An initial packet is guaranteed to not land on the same server because
otherwise an attacker could target that single server.

Placing the counter in the load balancer infrastructure also isn’t good
because it requires coordination which is a vary scarce resource in high
volume areas.

But a counter isn’t necessary as you can just share some comparatively weak
keys and hash a timestamp. It will be good enough, and otherwise you can
use stronger 0-RTT tokens for return visits. The keys can be rotated
periodically and shared to an extend, and so forth.

On 23 July 2019 at 17.19.06, [email-masked] (
[email-masked]) wrote:

As noted on a different posting, Single Packet Authorization is based on
RFC 4226 which has both a counter and a time based approach. Personally, I
prefer a counter.

But again, this is all irrelevant because I’m not suggesting any one
pre-authentication. I’m just using an example to show how it can stop
multiple types of attacks.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Jul 2019 15:36:17 +0000

If you’re in a “small” deployment that is exposed to the internet, then you’re going to be dead regardless of the tokens because the links will be full of useless packets if someone decides to do an attack at any reasonable scale.

If you’re in a “large” deployment that is exposed to the internet, then a counter-based implementation that requires any state at servers (to verify the counter) is likely to be very expensive as it either requires an RPC for each incoming packet-to-be-authenticated, or it requires that the packet be routed to a specific server, which makes the individual servers DoSable.

If you have a “large” deployment and use a time-based implementation, then you have the goldilocks-like problem of choosing the correct TTL, and that is going to be difficult as the DoS effectiveness becomes directly (inversely) related to the TTL length as TTL length determines the amount of the population that is no longer authenticated. A short TTL means very little benefit can be ascribed by having the token because few clients will remain authenticated.

The combination of time and counter doesn’t negate any of the disadvantages of either.

I don’t ultimately see a way out of having to heuristically examine 5-tuples.
-=R

From: "[email-masked]" <[email-masked]>
Date: Tuesday, July 23, 2019 at 11:19 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Roberto Peon <[email-masked]>, "[email-masked]" <[email-masked]>, Watson Ladd <[email-masked]>, "[email-masked]" <[email-masked]>, "brent.bilger=[email-masked]" <brent.bilger=[email-masked]>, Ian Swett <ianswett=[email-masked]>
Subject: Re: [E] DoS-ing QUIC servers is too easy

Brent

-=R

Brent

Brent

Sincerely,
Watson Ladd

Brent Bliger

Brent Bliger

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 08:45:54 -0700

I agree that only the time based approach is workable. But again, it is not
about keeping clients authenticated. It is only to allow the initial packet
to survive end to end, once. An since timers are inaccurate, you need some
slack. That opens for an attack. But that attack assumes that an earlier
token has been intercepted within the last few minutes, and if that is
possible, you have a compromised machine that can do this all day long. So
it is no better than no token. It is not perfect. But even then, you can
filter on high volume tokens - it will kill a few clients that generate
tokens the same way, but that is it.

However, you would not normally have a compromised client, and if you do,
well, you have bigger problems. And if somehow the 10minute pre-auth token
was stolen without any means to regenerated, that is useless, so nobody
would bother to raise such an attack.

On 23 July 2019 at 17.36.21, Roberto Peon ([email-masked]) wrote:

If you’re in a “small” deployment that is exposed to the internet, then
you’re going to be dead regardless of the tokens because the links will be
full of useless packets if someone decides to do an attack at any
reasonable scale.

If you’re in a “large” deployment that is exposed to the internet, then a
counter-based implementation that requires any state at servers (to verify
the counter) is likely to be very expensive as it either requires an RPC
for each incoming packet-to-be-authenticated, or it requires that the
packet be routed to a specific server, which makes the individual servers
DoSable.

If you have a “large” deployment and use a time-based implementation, then
you have the goldilocks-like problem of choosing the correct TTL, and that
is going to be difficult as the DoS effectiveness becomes directly
(inversely) related to the TTL length as TTL length determines the amount
of the population that is no longer authenticated. A short TTL means very
little benefit can be ascribed by having the token because few clients will
remain authenticated.

The combination of time and counter doesn’t negate any of the disadvantages
of either.

I don’t ultimately see a way out of having to heuristically examine
5-tuples.
-=R

*From: *"[email-masked]" <[email-masked]>
*Date: *Tuesday, July 23, 2019 at 11:19 AM
*To: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Cc: *Roberto Peon <[email-masked]>, "[email-masked]" <
[email-masked]>, Watson Ladd <[email-masked]>, "
[email-masked]" <[email-masked]>, "brent.bilger=[email-masked]"
<brent.bilger=[email-masked]>, Ian Swett <ianswett=
[email-masked]>
*Subject: *Re: [E] DoS-ing QUIC servers is too easy

On Jul 23, 2019, at 7:53 AM, Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

Yes, it requires compromising a client and stealing the key or just using
the key from a back door. In practice the latter may be advantageous to the
adversary because the source of the attack will be from a normal location
and because future device authorizations may include a fingerprint of the
device - a fingerprint that would fail on the stolen key. But look at what
we’ve done to the unauthorized adversary with the pre-authentication token
- defeated resource DoS, information leakage, all vulnerabilities of the
server, all configuration errors of the applications of the server. And it
augments TLS to defeat man-in-the-middle attacks.

Brent

-=R

Brent

-=R

Get Outlook for Android

Brent

Sincerely,
Watson Ladd

Brent Bliger

[email-masked]

Brent Bliger

[email-masked]

Brent Bliger

[email-masked]

---

From: [email-masked]
Date: Tue, 23 Jul 2019 16:03:48 +0000

Again, I’m not sure it matters at this point because we’re not recommending a method for creating a pre-authentication token, but I am curious about counter vs. timer. In counter-based HOTP (hashed one time password), the counter value is sent as part of the token. Since the client is storing the secret shared key, presumably, it would keep the counter is the same general location in long term memory. Since the counter changes every time, the OTP changes every time. What is it about the above that make it difficult to distribute in an enterprise environment?

Brent

On Jul 23, 2019, at 8:30 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

I believe that we largely agree, but a counter is not very good from a distributed perspective. I’d consider it a non-starter.

I do have a proposal for QUIC v2 about preventing replay attacks on 0-RTT that involves something similar - but 0-RTT is sort of guaranteed to land on the same server based on routing on said token.

An initial packet is guaranteed to not land on the same server because otherwise an attacker could target that single server.

Placing the counter in the load balancer infrastructure also isn’t good because it requires coordination which is a vary scarce resource in high volume areas.

But a counter isn’t necessary as you can just share some comparatively weak keys and hash a timestamp. It will be good enough, and otherwise you can use stronger 0-RTT tokens for return visits. The keys can be rotated periodically and shared to an extend, and so forth.

On 23 July 2019 at 17.19.06, [email-masked]<mailto:[email-masked]> ([email-masked]<mailto:[email-masked]>) wrote:

Brent Bliger

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Jul 2019 09:30:21 -0700

We are talking about an architecture where you have K load balancers that
routes traffic to N endpoints. It is assumed that the load balancers are
trusted by the the N endpoints. Sometimes the endpoints can coordinate with
the load balancers, but not always (e.g. public cloud computing).

Initially the endpoint is chosen at random (sort of). It must be. If the
client can decide, an attacker can send all traffic to that end point.
After that, obviously traffic should land on the same endpoint throughout
the connection.

Since we are discussing a token for the initial packet, it can pass through
any of the K load balancers and land on any of the N endpoints.

Where should the counter be stored? It must be server side (i.e. either
endpoint or load balancer) in order to validate the token. And of course
also client side.

Aside from problems with clients and servers crashing and persistent
recovery, which is a whole problem on its own, we now have KxN places to
have a counter. This is reduced to N if we decide to store at the end
points, or K if the load balancers take on the job. Let's just say K.

How do you increment the counter or check it? You add a separate service
that all K load balancers talk to. This is either a single point of
failure, and a very slow bottleneck with network latency, or it is a
distributed service like etcd.

Both of these coordination approaches are used for load balancers, but that
happens at an entirely different time scale. If for super high speed
distributed coordination servers, you typically cannot go faster than a
single machine. They also typically need to have persistent crash safe
storage.

This is way way too slow for high volume traffic. Especially under attack.

Now, even if it was fast enough it would be vulnerable to single point of
failure and getting out of sync.

A lot of cryptography and security these days work really hard to avoid
having such shared state. Public key cryptography is a prominent example,
and we could use that to authenticate tokens - only - it is also too slow -
in fact - slow crypto is largely what we are trying to avoid with the token
in the first place.

What is not too slow is a pre-shared secret with a timestamp. It is on the
other hand not strong enough for proper authentication, but it is good
enough for our purposes. Each of the KxN locations can independently check
the token with low overhead.

On 23 July 2019 at 18.03.51, [email-masked] (
[email-masked]) wrote:

Again, I’m not sure it matters at this point because we’re not recommending
a method for creating a pre-authentication token, but I am curious about
counter vs. timer. In counter-based HOTP (hashed one time password), the
counter value is sent as part of the token. Since the client is storing the
secret shared key, presumably, it would keep the counter is the same
general location in long term memory. Since the counter changes every time,
the OTP changes every time. What is it about the above that make it
difficult to distribute in an enterprise environment?

Brent

On Jul 23, 2019, at 8:30 AM, Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

Brent Bliger

---

From: [email-masked]
Date: Tue, 23 Jul 2019 18:39:35 +0000

In counter-based HOTP, the client has the counter, not the server. When the client wants to connect to the server, the client increments the counter and creates the OTP (from the hash the counter with the secret key). The client then sends the Counter and the OTP to the server. It would also send its client ID. Therefor all K servers will get the counter and don't need some other service to provide it.

Now, you may ask about the shared key. One way to handle this is to give the servers a master key, Km, and use it to generate the client specific key, Ks. One way to do that is Ks = HMAC[ClientID, Km]. Therefore, the on-boarding of the client would involve generating a random client ID and the shared key from it, Ks = HMAC[ClientID, Km]. Then, all K servers would get the counter from the client and generate the client specific shared key from the master key. Note that HMAC is a computationally efficient algorithm.

The above was all made up. Again, the goal is to get the pre-authentication token in the spec, not design it.

Regards,
Brent

On Jul 23, 2019, at 9:30 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

We are talking about an architecture where you have K load balancers that routes traffic to N endpoints. It is assumed that the load balancers are trusted by the the N endpoints. Sometimes the endpoints can coordinate with the load balancers, but not always (e.g. public cloud computing).

Initially the endpoint is chosen at random (sort of). It must be. If the client can decide, an attacker can send all traffic to that end point. After that, obviously traffic should land on the same endpoint throughout the connection.

Since we are discussing a token for the initial packet, it can pass through any of the K load balancers and land on any of the N endpoints.

Where should the counter be stored? It must be server side (i.e. either endpoint or load balancer) in order to validate the token. And of course also client side.

Aside from problems with clients and servers crashing and persistent recovery, which is a whole problem on its own, we now have KxN places to have a counter. This is reduced to N if we decide to store at the end points, or K if the load balancers take on the job. Let's just say K.

How do you increment the counter or check it? You add a separate service that all K load balancers talk to. This is either a single point of failure, and a very slow bottleneck with network latency, or it is a distributed service like etcd.

Both of these coordination approaches are used for load balancers, but that happens at an entirely different time scale. If for super high speed distributed coordination servers, you typically cannot go faster than a single machine. They also typically need to have persistent crash safe storage.

Now, even if it was fast enough it would be vulnerable to single point of failure and getting out of sync.

A lot of cryptography and security these days work really hard to avoid having such shared state. Public key cryptography is a prominent example, and we could use that to authenticate tokens - only - it is also too slow - in fact - slow crypto is largely what we are trying to avoid with the token in the first place.

What is not too slow is a pre-shared secret with a timestamp. It is on the other hand not strong enough for proper authentication, but it is good enough for our purposes. Each of the KxN locations can independently check the token with low overhead.

On 23 July 2019 at 18.03.51, [email-masked]<mailto:[email-masked]> ([email-masked]<mailto:[email-masked]>) wrote:

Brent

Brent Bliger

Brent Bliger