Subject: Back to work

From: "Martin Thomson" <[email-masked]>
Date: Tue, 27 Oct 2020 17:34:31 +1100

Nick opened https://github.com/quicwg/base-drafts/issues/4257 last week.

Amplification attacks based on spoofed path migrations seem like they might need some thought.

We have two options we're debating and it seems like a good candidate for broader discussion.

# Background

We currently recommend that you send kMinimumWindow each kInitialRtt to an unvalidated peer address after migration.  This is a different limit to that we use during the handshake and not absolutely bounded.  Thus, if an attacker can spoof migration the spoofed address might receive path validation probes at this rate until the path validation fails.

If our recommendations for validation are followed, this means that you could have a relatively small packet (21 bytes is our minimum, but then we have to add overheads and a connection ID to be realistic, so maybe 51 bytes) to get 1200 times the number of probes sent during path validation (which might between 3 and maybe 9 depending on how you interpret the text).

The same connection can be used to attack as many different hosts as the server is willing to probe from the same connection.  Whether this is in parallel or series doesn't matter so much, we're most interested in the number of bytes involved.

That seems bad, but you have to remember that the attacker has performed a full handshake: all the crypto and round trips necessary to get this going.  That's a non-trivial cost; even if the attacker doesn't bother with retransmissions and timers, it's still a fairly large commitment to mount the attack.

Rough calculations show that 3 probes toward a single target isn't that interesting for an attacker.  If you count a minimal handshake in their costs, they get less than 3x amplification if there are 3 probes and nearly 8x for 9 probes.

If an attacker has multiple targets and considers attacking all of them equally worthwhile, then then can amortize the setup cost across those targets.  Then it depends on how many targets they have and how many paths the server is willing to probe.  Attacker advantage then goes up asymptotically toward the maximum advantage, which is about 25 times the number of probes sent in response to the single packet.

Kazuho also points out that the attacker is mounting a denial of service on themselves as part of this.  Migration causes a probe on the existing, validated path.  Those can be answered readily to limit the number of probes, but congesting their downlink in this way might limit their ability to create additional connections.

# Option 1: 3x limit

We could make the 3x limit for unvalidated paths a blanket requirement.

This is principled, but complicates the process if validating a path.  If you haven't received much data on that path, then you will be unable to send a fully padded probe and so validation only establishes that the recipient is present; then you have to probe again to determine that the path supports the 1200 byte MTU.

This is also risky.  The last two changes we made in this area produced nasty regressions.  This issue is a direct result of changes; without the requirement to pad packets containing PATH_CHALLENGE, this would not be an issue.

# Option 2: Document the exposure and let implementations manage this

The existing rate limiting does have a limited ability to limit amplification.  25x amplification isn't anything to celebrate, but you might say that the setup cost and other inherent limitations mean that it is rarely that bad.  If the number of probes and the number of failed path validations are both limited, then this could be entirely manageable.  For instance, 3 probes, a connection ID of 8 bytes and 4 path validations provide just 11.6x advantage.

Recommending that implementations limit the number probes and the number of probed paths might then be sufficient.

Note: The rate limiting we currently have only helps if servers limit the number of active connections in some way.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Tue, 27 Oct 2020 10:52:22 +0300

Peace,

On Tue, Oct 27, 2020 at 9:35 AM Martin Thomson <[email-masked]> wrote:
> 25x amplification isn't anything to celebrate, but you might
> say that the setup cost and other inherent limitations mean
> that it is rarely that bad.

This is a price an attacker only needs to pay once.  If this code
leaks to one of the myriad Mirai forks available on Github, the cost
would be effectively zero then.

> Recommending that implementations limit the number
> probes and the number of probed paths might then be
> sufficient.

_Recommendations_ never worked well for DDoS prevention in the past.
I think Section 11.3 of RFC 7252 and what happened next is the most
recent example of that.

--
Töma

---

From: Ian Swett <[email-masked]>
Date: Tue, 27 Oct 2020 09:12:28 -0400

Thanks for summarizing this issue. I think the above discussion is about
immediate migration and repeated immediate migrations, but I also wonder if
we've introduced a single packet amplification attack by requiring
PATH_RESPONSEs be padded on new paths without a requirement on the size of
PATH_CHALLENGE(see first item)?

*Validating a new path*
If one receives only a PATH_CHALLENGE on a new path, then the server
responds with a full-sized PATH_RESPONSE.  This seems safe.  If a
non-padded PATH_CHALLENGE is received on a new path, then the peer is
supposed to send a fully padded PATH_RESPONSE on the path, which could be
>20x larger.  I'm not sure if we care about this, but I wanted to point it
out.

*Immediately migrating to a new path*
I think we should remove the text about allowing kMinimumWindow each
kInitialRtt after migration and change it to the 3x limit.  I'm actually
surprised the text about 2*kInitialWindow still exists, since recovery
says *"Until
the server has validated the client's address on the path, the amount of
data it can send is limited to three times the amount of data received, as
specified in Section 8.1 of {{QUIC-TRANSPORT}}.*".

In order to not get deadlocked by the 3x factor, I think we should change
the newly added MUSTs to only apply to path validation prior to migration,
not the peer responding to migration.

My reasoning is that if a peer migrates prior to validating the path, it
means it's either unintentional or they have no other choice, so the
migrating peer has implicitly decided that validating PathMTU is not a
prerequisite to migrating.

Thanks, Ian

---

From: Nick Banks <[email-masked]>
Date: Tue, 27 Oct 2020 15:54:53 +0000

I think Ian’s recommendation is a good one. It seems well balanced and in line with the existing principles.

Also, on the topic of factoring the cost of the handshake in here, I don’t see that as a significant hurdle. Once you have a connection, you can do minimal work to keep the normal path alive and continue to use it as the amplification vector. Then there’s no absolute limit to the number of times it can be used for a path validation attempt, AFAIK.

Thanks,
- Nick

From: QUIC <[email-masked]> On Behalf Of Ian Swett
Sent: Tuesday, October 27, 2020 6:12 AM
To: Töma Gavrichenkov <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Back to work

Thanks for summarizing this issue. I think the above discussion is about immediate migration and repeated immediate migrations, but I also wonder if we've introduced a single packet amplification attack by requiring PATH_RESPONSEs be padded on new paths without a requirement on the size of PATH_CHALLENGE(see first item)?

Validating a new path
If one receives only a PATH_CHALLENGE on a new path, then the server responds with a full-sized PATH_RESPONSE.  This seems safe.  If a non-padded PATH_CHALLENGE is received on a new path, then the peer is supposed to send a fully padded PATH_RESPONSE on the path, which could be >20x larger.  I'm not sure if we care about this, but I wanted to point it out.

Immediately migrating to a new path
I think we should remove the text about allowing kMinimumWindow each kInitialRtt after migration and change it to the 3x limit.  I'm actually surprised the text about 2*kInitialWindow still exists, since recovery says "Until the server has validated the client's address on the path, the amount of data it can send is limited to three times the amount of data received, as specified in Section 8.1 of {{QUIC-TRANSPORT}}.".

In order to not get deadlocked by the 3x factor, I think we should change the newly added MUSTs to only apply to path validation prior to migration, not the peer responding to migration.

My reasoning is that if a peer migrates prior to validating the path, it means it's either unintentional or they have no other choice, so the migrating peer has implicitly decided that validating PathMTU is not a prerequisite to migrating.

Thanks, Ian

On Tue, Oct 27, 2020 at 3:52 AM Töma Gavrichenkov <[email-masked]<mailto:[email-masked]>> wrote:
Peace,

--
Töma

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 27 Oct 2020 17:04:28 +0000

On Tue, 2020-10-27 at 09:12 -0400, Ian Swett wrote:
> responds with a full-sized PATH_RESPONSE.  This seems safe.  If a non-padded
> PATH_CHALLENGE is received on a new path, then the peer is supposed to send a
> fully padded PATH_RESPONSE on the path, which could be >20x larger.  I'm not
> sure if we care about this, but I wanted to point it out. 
> 
> surprised the text about 2*kInitialWindow still exists, since recovery says
> "Until the server has validated the client's address on the path, the amount
> of data it can send is limited to three times the amount of data received, as
> specified in Section 8.1 of {{QUIC-TRANSPORT}}.".
> 
> In order to not get deadlocked by the 3x factor, I think we should change the
> newly added MUSTs to only apply to path validation prior to migration, not the
> peer responding to migration.
> 
> My reasoning is that if a peer migrates prior to validating the path, it means
> it's either unintentional or they have no other choice, so the migrating peer
> has implicitly decided that validating PathMTU is not a prerequisite to
> migrating.

So some quesitons and ideas as I think it is relevant to resolve this as best as
possible.

So isn't this recreating the issue that if the client initiates a migration to a
new path that is not QUIC compatible, by responding with a minimal size packet
and completing the migration and then if the server performs the path
verification with 1200 bytes UDP payload it fails. Thus maintaining a broken
path. 

So is there need for the non pre-validated path migration case that one need
need to do a two step process where one will ACK with minimal packet while
initiating path validation. If path validatation fails then maybe one need to
close down the connection as the migration ended up on a path that was unable to
support QUIC. The question here is how to avoid the DoS attack this may open up
if an attack rewrites source address of packets. 

So Maybe the path validation needs to be a two step process. First a return
routability over the new path to verify that it is bi-directional. When that has
been verified one does a test with minimal MTU to prove it to be QUIC
compatible. This might even be done with application data if there is some that
are available to send. 

But, I think that one needs to work through the criterias for when the QUIC
connection is shut down under the conditions that the path available is not
supporting 1200 bytes. Also do we end up in a situation where the client needs
to do the second step itself towards the server to verify the path so that it
can determine if it needs to try another path if this one doesn't work? 

Cheers

Magnus

---

From: Martin Duke <[email-masked]>
Date: Tue, 27 Oct 2020 13:13:13 -0700

PATH_CHALLENGE MUST be padded as well, so I don't see that as an
amplification vector. It might be useful to add language to allow
PATH_CHALLENGE to be ignored if not padded (SHOULD ignore?)

In migration, the path usually should have already been pre-validated so
there is no attack vector once that happens.

But Ian's proposal to use the 3x limit is not so great in the NAT rebinding
case. If a small packet causes this, data transfer is going to plummet, and
we can't simultaneously verify the path and the MTU.

On Tue, Oct 27, 2020 at 10:04 AM Magnus Westerlund <magnus.westerlund=
[email-masked]> wrote:

> non-padded
> says
> means

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 28 Oct 2020 08:17:23 +0900

> PATH_CHALLENGE to be ignored if not padded (SHOULD ignore?)

The problem here is that an attacker might establish a connection, then
send small-sized datagrams containing PATH_CHALLENGE frames. Therefore, if
we are to have a stringent amplification limit for path migration (BTW, I
like Ian's proposal), we should better block this attack vector too.

So maybe something like "MUST NOT send PATH_RESPONSE frames in response to
PATH_CHALLENGE frame carried by a datagram that is less than 1200 bytes."

> But Ian's proposal to use the 3x limit is not so great in the NAT
> rebinding case. If a small packet causes this, data transfer is going to
> plummet, and we can't simultaneously verify the path and the MTU.
>

-- 
Kazuho Oku

---

From: Martin Duke <[email-masked]>
Date: Tue, 27 Oct 2020 16:23:31 -0700

Again the spec says PATH_CHALLENGE MUST be padded to 1200

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 28 Oct 2020 08:49:52 +0900

> Again the spec says PATH_CHALLENGE MUST be padded to 1200

The specification says that. At the same time the receiver is *not*
required to enforce that.

To be specific, the receiver is not allowed to error-close the connection,
because the size of datagrams are never authenticated. See #4253. At the
moment, section 14[1] permits *either* of the following two behaviors when
receiving a small-sized datagram containing PATH_CHALLENGE: a) respond with
a full-sized PATH_RESPONSE, b) discard that packet.

But as stated, we need a MUST that somehow prohibits the endpoint from
sending PATH_RESPONSE, if we want the rule to be stringent.

That might sound like we will be enforcing b. But that's going to be
problematic, due to the approach requiring implementations to do frame
processing then discard the packet. That's going to be complicated for some
QUIC stacks, because they make changes to the connection state (e.g.,
adding the packet number to ack queue) before processing the frames.

Fortunately, we have another option, and that is to state that endpoints
MUST not respond with PATH_RESPONSEs under these conditions. By stating as
such, the receiver can either discard the packet, or process the packet as
usual but refrain from sending PATH_RESPONSE.

[1]
https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-14-8

>> So maybe something like "MUST NOT send PATH_RESPONSE frames in response
>> to PATH_CHALLENGE frame carried by a datagram that is less than 1200 bytes."
>>
>>

-- 
Kazuho Oku

---

From: Eric Kinnear <[email-masked]>
Date: Tue, 27 Oct 2020 19:41:07 -0700

> On Oct 27, 2020, at 4:49 PM, Kazuho Oku <[email-masked]> wrote:
> 
> 
> 
> 2020年10月28日(水) 8:23 Martin Duke <[email-masked] <mailto:[email-masked]>>:
> The specification says that. At the same time the receiver is *not* required to enforce that.
> 
> To be specific, the receiver is not allowed to error-close the connection, because the size of datagrams are never authenticated. See #4253. At the moment, section 14[1] permits *either* of the following two behaviors when receiving a small-sized datagram containing PATH_CHALLENGE: a) respond with a full-sized PATH_RESPONSE, b) discard that packet.
> 
> But as stated, we need a MUST that somehow prohibits the endpoint from sending PATH_RESPONSE, if we want the rule to be stringent.
> 
> That might sound like we will be enforcing b. But that's going to be problematic, due to the approach requiring implementations to do frame processing then discard the packet. That's going to be complicated for some QUIC stacks, because they make changes to the connection state (e.g., adding the packet number to ack queue) before processing the frames.
> 
> Fortunately, we have another option, and that is to state that endpoints MUST not respond with PATH_RESPONSEs under these conditions. By stating as such, the receiver can either discard the packet, or process the packet as usual but refrain from sending PATH_RESPONSE. 
> 
That seems like a not-unreasonable option, Kazuho. It certainly seems like it might close some loopholes that haven’t yet been fully explored around sizing of PATH_CHALLENGE and PATH_RESPONSE.
I’d also note that at one point we lost the 3x limit for path validation. We used to match the handshake, and when the pacing discussion came in we wound up with the X bytes / initial RTT duration.

It is important to be able to migrate immediately without validating, not just because an endpoint might be discovering/bringing up a new path at the same time as the old one is bad, but due to NAT or other routing changes — we can’t attempt to guarantee that the 4-tuple will stay the same and also expect to survive NAT rebinding. 

One of the situations that we’ve been using as an example attack is someone who is able to cause you to toggle between arbitrary ports on a NAT for each packet (if you go back and forth between two, ideally you’d just end up validating both and then be cheerfully willing to alternate the data between the two of them on a sub-RTT basis, probably with somewhat restricted flight sizes, but let’s not get back into multipath here).

Had a few more thoughts after reading through this thread and re-reading the relevant sections of the document, but will split those out into a separate message as they got rather long.

Thanks,
Eric

> [1] https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-14-8 <https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-14-8>
> 
> 
> On Tue, Oct 27, 2020 at 4:17 PM Kazuho Oku <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 2020年10月28日(水) 5:13 Martin Duke <[email-masked] <mailto:[email-masked]>>:
> PATH_CHALLENGE MUST be padded as well, so I don't see that as an amplification vector. It might be useful to add language to allow PATH_CHALLENGE to be ignored if not padded (SHOULD ignore?)
> 
> The problem here is that an attacker might establish a connection, then send small-sized datagrams containing PATH_CHALLENGE frames. Therefore, if we are to have a stringent amplification limit for path migration (BTW, I like Ian's proposal), we should better block this attack vector too.
> 
> So maybe something like "MUST NOT send PATH_RESPONSE frames in response to PATH_CHALLENGE frame carried by a datagram that is less than 1200 bytes."
> 
> 
> In migration, the path usually should have already been pre-validated so there is no attack vector once that happens.
> 
> But Ian's proposal to use the 3x limit is not so great in the NAT rebinding case. If a small packet causes this, data transfer is going to plummet, and we can't simultaneously verify the path and the MTU.
> 
> On Tue, Oct 27, 2020 at 10:04 AM Magnus Westerlund <magnus.westerlund=[email-masked] <mailto:[email-masked]>> wrote:

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 28 Oct 2020 13:41:18 +1100

Thanks to everyone for the feedback.

I've written up a draft pull request here: https://github.com/quicwg/base-drafts/pull/4264

This does something like what Magnus suggests below.  It's not pretty, because in some very common cases path validation could take twice as long, and it's more complicated, but I think that it is at least principled.

On Wed, Oct 28, 2020, at 04:04, Magnus Westerlund wrote:
> Attachments:
> * smime.p7s

---

From: Eric Kinnear <[email-masked]>
Date: Tue, 27 Oct 2020 19:54:25 -0700

This is an interesting PR, and likely accomplishes the goals at the moment.
I do really like how we’ve kept some bidirectionally of the approach and the padding can stay as is. 

Just thinking things through a little bit: 
(This is all discussed below by Ian/Magnus/Martin/Kazuho, and others, just restating so we have it in one place)

At any point, either endpoint can choose to send a PATH_CHALLENGE.
The presence of a PATH_CHALLENGE always evokes a PATH_RESPONSE.

Therefore, we assume that in order to restrict folks from being able to spoof a source address when sending a PATH_CHALLENGE and attack the real owner of that source address with the PATH_RESPONSE, we need to make the PATH_CHALLENGE very large as well. 

However, there’s another situation where PATH_CHALLENGE is sent, and that's whenever we receive a non-probing packet that arrives on a new path without any prior validation, and we send that PATH_CHALLENGE on both the old and the new path.

This is where we haven’t fully plugged the amplification hole, since an attacker can use any other, smaller datagram to cause the other endpoint to generate full-size datagrams containing PATH_CHALLENGE. This wasn’t previously a huge issue since PATH_CHALLENGE wasn’t meaningfully larger than the smallest packet you’d otherwise be able to send (slash the per-packet costs were potentially higher than the cost of the data inside that packet).

——— 

One other approach we could take here would be to restrict ourselves to only covering the cases where you’re actively generating a PATH_CHALLENGE to validate a new path, not responding to a new non-probing packet on an unvalidated path. 

In other words: 
Only the client needs to pad PATH_CHALLENGE and any response to a padded PATH_CHALLENGE should also be padded. That also fits nicely into the unidirectionality of path validation as it stands today.

The other option that we haven’t discussed much is if we’d rather live with the previous pre-padding problem and remove the padding. 
My initial inclination was to avoid this, but actually we’d be returning to a state where the main risk was that the path wasn’t MTU compatible and any implementation migrating is likely already dealing with cases where packets aren’t going through on a path in at least one direction. So, the natural responses to path validation failures (for MTU reasons or otherwise), if you map them all out, generally result in the “correct” behavior. We could then say “any endpoint using a new path is encouraged to do PMTUD or otherwise be careful that the path may not work in at least one direction” and leave it at that.

——— 

Overall, I suspect we’re probably headed in the right direction by making the 3x limit more universal, although it does seem like it introduces some really interesting cases to code around, and that limit and double path validation might be more painful than just checking for “am I client, therefore I should pad” which is annoying because it has a client/server distinction but does likely cause less churn and risk for later fallout.

Thanks,
Eric

> On Oct 27, 2020, at 7:41 PM, Martin Thomson <[email-masked]> wrote:
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 27 Oct 2020 23:18:27 -0700

Rather than a race to the top with padding, would it be possible to do the
opposite:

Force challenges and responses to occur in their packets and also UDP
datagrams. This prevents other traffic until a path is confirmed.

The initial handshake has several concerns with padding:

- amplification attack mitigation
- PMTU discovery
- reply capacity for completing handshake

Since new paths do not need a handshake, there is less need for large
replies. Of course there is the PMTU issue still.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 28 October 2020 at 03.55.46, Eric Kinnear (
ekinnear=[email-masked]) wrote:

This is an interesting PR, and likely accomplishes the goals at the moment.
I do really like how we’ve kept some bidirectionally of the approach and
the padding can stay as is.

Just thinking things through a little bit:
(This is all discussed below by Ian/Magnus/Martin/Kazuho, and others, just
restating so we have it in one place)

Therefore, we assume that in order to restrict folks from being able to
spoof a source address when sending a PATH_CHALLENGE and attack the real
owner of that source address with the PATH_RESPONSE, we need to make the
PATH_CHALLENGE very large as well.

However, there’s another situation where PATH_CHALLENGE is sent, and that's
whenever we receive a non-probing packet that arrives on a new path without
any prior validation, and we send that PATH_CHALLENGE on both the old and
the new path.

This is where we haven’t fully plugged the amplification hole, since an
attacker can use *any other, smaller datagram* to cause the other endpoint
to generate full-size datagrams containing PATH_CHALLENGE. This wasn’t
previously a huge issue since PATH_CHALLENGE wasn’t meaningfully larger
than the smallest packet you’d otherwise be able to send (slash the
per-packet costs were potentially higher than the cost of the data inside
that packet).

———

One other approach we could take here would be to restrict ourselves to
only covering the cases where you’re actively generating a PATH_CHALLENGE
to validate a new path, not responding to a new non-probing packet on an
unvalidated path.

In other words:
Only the client needs to pad PATH_CHALLENGE and any response to a padded
PATH_CHALLENGE should also be padded. That also fits nicely into the
unidirectionality of path validation as it stands today.

The other option that we haven’t discussed much is if we’d rather live with
the previous pre-padding problem and remove the padding.
My initial inclination was to avoid this, but actually we’d be returning to
a state where the main risk was that the path wasn’t MTU compatible and any
implementation migrating is likely already dealing with cases where packets
aren’t going through on a path in at least one direction. So, the natural
responses to path validation failures (for MTU reasons or otherwise), if
you map them all out, generally result in the “correct” behavior. We could
then say “any endpoint using a new path is encouraged to do PMTUD or
otherwise be careful that the path may not work in at least one direction”
and leave it at that.

———

Overall, I suspect we’re probably headed in the right direction by making
the 3x limit more universal, although it does seem like it introduces some
really interesting cases to code around, and that limit and double path
validation might be more painful than just checking for “am I client,
therefore I should pad” which is annoying because it has a client/server
distinction but does likely cause less churn and risk for later fallout.

Thanks,
Eric

I've written up a draft pull request here:
https://github.com/quicwg/base-drafts/pull/4264

This does something like what Magnus suggests below.  It's not pretty,
because in some very common cases path validation could take twice as long,
and it's more complicated, but I think that it is at least principled.

Validating a new path
PATH_CHALLENGE is received on a new path, then the peer is supposed to send
a

of data it can send is limited to three times the amount of data received,
as

In order to not get deadlocked by the 3x factor, I think we should change
the
newly added MUSTs to only apply to path validation prior to migration, not
the
peer responding to migration.

My reasoning is that if a peer migrates prior to validating the path, it
means
it's either unintentional or they have no other choice, so the migrating
peer
has implicitly decided that validating PathMTU is not a prerequisite to
migrating.

So some quesitons and ideas as I think it is relevant to resolve this as
best as
possible.

So isn't this recreating the issue that if the client initiates a migration
to a
new path that is not QUIC compatible, by responding with a minimal size
packet
and completing the migration and then if the server performs the path
verification with 1200 bytes UDP payload it fails. Thus maintaining a broken
path.

So is there need for the non pre-validated path migration case that one need
need to do a two step process where one will ACK with minimal packet while
initiating path validation. If path validatation fails then maybe one need
to
close down the connection as the migration ended up on a path that was
unable to
support QUIC. The question here is how to avoid the DoS attack this may
open up
if an attack rewrites source address of packets.

So Maybe the path validation needs to be a two step process. First a return
routability over the new path to verify that it is bi-directional. When
that has
been verified one does a test with minimal MTU to prove it to be QUIC
compatible. This might even be done with application data if there is some
that
are available to send.

But, I think that one needs to work through the criterias for when the QUIC
connection is shut down under the conditions that the path available is not
supporting 1200 bytes. Also do we end up in a situation where the client
needs
to do the second step itself towards the server to verify the path so that
it
can determine if it needs to try another path if this one doesn't work?

Cheers

Magnus

Attachments:
* smime.p7s

---

From: Ian Swett <[email-masked]>
Date: Wed, 28 Oct 2020 16:03:52 -0400

I'll note that this problem is created/worsened by the fact that the
congestion controller is reset.  If it was not reset, you'd be limited by
the existing congestion controller.

That would allow you to build up a big window and direct it at another
path, but creating a larger window is more work on top of completing the
handshake.

NAT rebinds don't require resetting the congestion controller if my memory
is correct, so I don't believe they don't need to be covered by this new
amplification factor.

Ian

On Wed, Oct 28, 2020 at 2:18 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> However, there’s another situation where PATH_CHALLENGE is sent, and
> that's whenever we receive a non-probing packet that arrives on a new path
> without any prior validation, and we send that PATH_CHALLENGE on both the
> old and the new path.
>
> attacker can use *any other, smaller datagram* to cause the other
> endpoint to generate full-size datagrams containing PATH_CHALLENGE. This
> wasn’t previously a huge issue since PATH_CHALLENGE wasn’t meaningfully
> larger than the smallest packet you’d otherwise be able to send (slash the
> The other option that we haven’t discussed much is if we’d rather live
> with the previous pre-padding problem and remove the padding.
> My initial inclination was to avoid this, but actually we’d be returning
> to a state where the main risk was that the path wasn’t MTU compatible and
> any implementation migrating is likely already dealing with cases where
> packets aren’t going through on a path in at least one direction. So, the
> natural responses to path validation failures (for MTU reasons or
> otherwise), if you map them all out, generally result in the “correct”
> behavior. We could then say “any endpoint using a new path is encouraged to
> do PMTUD or otherwise be careful that the path may not work in at least one
> direction” and leave it at that.
>
> ———
>

---

From: Martin Duke <[email-masked]>
Date: Wed, 28 Oct 2020 13:27:40 -0700

NAT doesn't require reset, but there are times when it's wise, particularly
when the NAT is closer to the server than the client. (e.g. a NAT might
signal a mobility event on the clientside by changing its port)

On Wed, Oct 28, 2020 at 1:04 PM Ian Swett <ianswett=
[email-masked]> wrote:

> On Wed, Oct 28, 2020 at 2:18 AM Mikkel Fahnøe Jørgensen <
> [email-masked]> wrote:
>
>> Rather than a race to the top with padding, would it be possible to do
>> the opposite:
>>
>> (This is all discussed below by Ian/Magnus/Martin/Kazuho, and others,
>> just restating so we have it in one place)
>>

---

From: Ian Swett <[email-masked]>
Date: Wed, 28 Oct 2020 17:27:10 -0400

Agreed.  I'm proposing we require either not resetting(under the assumption
the path hasn't meaningfully changed) or applying the 3x limit, but not
both.  This might be a result of Martin's current PR, but if so we should
probably make the point and the reasoning more explicit.

On Wed, Oct 28, 2020 at 4:28 PM Martin Duke <[email-masked]> wrote:

> NAT doesn't require reset, but there are times when it's wise,
> particularly when the NAT is closer to the server than the client. (e.g. a
> NAT might signal a mobility event on the clientside by changing its port)
>
>> NAT rebinds don't require resetting the congestion controller if my
>> memory is correct, so I don't believe they don't need to be covered by this
>> new amplification factor.
>>
>> Ian
>>
>>> Overall, I suspect we’re probably headed in the right direction by
>>> making the 3x limit more universal, although it does seem like it
>>> introduces some really interesting cases to code around, and that limit and
>>> double path validation might be more painful than just checking for “am I
>>> client, therefore I should pad” which is annoying because it has a
>>> client/server distinction but does likely cause less churn and risk for
>>> later fallout.
>>>
>>> Thanks,
>>> Eric
>>>
>>>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 28 Oct 2020 17:17:58 -0700

I like the universality of the 3x limit. The reasoning applies broadly and
there's no reason to separately reason about how a server responds to new
addresses, be it at the start of a connection or mid-connection. Overall, I
have a few minor suggestions to make, but I'm happy with the way the PR is
headed.

Ian -- I don't understand what resetting has to do with this. Can you
explain a bit more? (We have the min window now in the spec, but the
reasoning of using that for an amplification limit seems arbitrary)

On Wed, Oct 28, 2020 at 2:27 PM Ian Swett <ianswett=
[email-masked]> wrote:

> Agreed.  I'm proposing we require either not resetting(under the
> assumption the path hasn't meaningfully changed) or applying the 3x limit,
> but not both.  This might be a result of Martin's current PR, but if so we
> should probably make the point and the reasoning more explicit.
>
>>>> Only the client needs to pad PATH_CHALLENGE and any response to a
>>>> padded PATH_CHALLENGE should also be padded. That also fits nicely into the
>>>> My initial inclination was to avoid this, but actually we’d be
>>>> returning to a state where the main risk was that the path wasn’t MTU
>>>> compatible and any implementation migrating is likely already dealing with
>>>> cases where packets aren’t going through on a path in at least one
>>>> direction. So, the natural responses to path validation failures (for MTU
>>>> reasons or otherwise), if you map them all out, generally result in the
>>>> “correct” behavior. We could then say “any endpoint using a new path is
>>>> encouraged to do PMTUD or otherwise be careful that the path may not work
>>>> in at least one direction” and leave it at that.
>>>>
>>>> ———
>>>>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 28 Oct 2020 17:41:04 -0700

On 10/28/2020 5:17 PM, Jana Iyengar wrote:
> I like the universality of the 3x limit. The reasoning applies broadly
> and there's no reason to separately reason about how a server responds
> to new addresses, be it at the start of a connection or
> mid-connection. Overall, I have a few minor suggestions to make, but
> I'm happy with the way the PR is headed.

I think that we can devise proper solutions for "organized migrations",
such as requiring both receipt and acknowledgement of a long enough
packet before validating a path. I am not going to belabor that. But NAT
traversal is a can of worms, and some of the norms are going to bite us.

I order to support NAT traversal, we request servers to take action if
they see a packet arriving from a new address of the client. This opens
a huge hole through which enterprising MOTS or "specially crafted
clients" can harass servers. We have an OK defense: send a challenge to
both old and new IP to force "proof of address ownership". That defense
is OK because it uses minimal length packets, and thus does not cause
too much amplification. I think we should leave it at that, and not
force the server to send large packets. There is a need to then validate
that the path can carry traffic, which will happen naturally when the
server tries to send a large packet. But if the server only sends these
large packets after address ownership has been verified, things ought to
be quite good.

-- Christian Huitema

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 28 Oct 2020 19:13:30 -0700

I was originally shying a bit away from that option, Christian, but the more I think about it the less painful it seems: back out the padding requirement, add in a note that “after a new path has been validated, endpoints can/should/must do PMTUD to validate the MTU on that path”. 

The natural responses that an endpoint goes through during path validation (or abrupt migration) leave it with either a working or failing path when trying to migrate; this was always the case and is no more or less the case due to MTU concerns. 

Alternately, you also note that we might want to avoid forcing the server specifically to send large packets — in my opinion, that would be another equally valid way to tackle this: require the padding only for PATH_CHALLENGES sent from the client and any PATH_RESPONSE to a padded challenge.
That’s really nice in a number of ways, especially for the MOTS cases where I can make the server think that every new packet is on a new path, etc.

So… lots of words to say: I very much agree with what you’re saying. :) 

It seems like these would be good options to use if we think that there’s something we don’t like about the 3x limit (perhaps how it interacts with NAT rebinding). Otherwise, the 3x limit is nice and general.

How worried are we about the 3x limit?

Thanks,
Eric

> On Oct 28, 2020, at 5:41 PM, Christian Huitema <[email-masked]> wrote:
> 
>

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 29 Oct 2020 16:17:01 +1100

Hi Eric,

I don't know what you mean by: "How worried are we about the 3x limit?".

I got the impression that Christian's request was that we avoid padding where it is unnecessary.

The two things we need to balance here are:
* Amplification (the reason we have a 3x limit)
* Odd failures when things only work sometimes (the reason we require padding of probes)

My belief is that the proposed PR #4264 achieves this balance in a minimal fashion.

Yes, there are two things you need to verify for a new path, but for the most part, probes can be padded and so will verify both.  Both that the address is live and that the path supports a reasonable MTU.  These are separated into two separate checks if checking the latter would violate the amplification limit.

Padding packets sent on either path is sub-optimal if you already know that the MTU is OK, so I'd be happy to create a carve-out for that.  It seems like a fine optimization, but the space can be used for things other than PADDING frames, so I don't see it as critical.  My migration implementation (almost finished, thanks) will include other frames in these packets where possible.

Cheers,
Martin

On Thu, Oct 29, 2020, at 13:13, Eric Kinnear wrote:
> I was originally shying a bit away from that option, Christian, but the 
> more I think about it the less painful it seems: back out the padding 
> requirement, add in a note that “after a new path has been validated, 
> endpoints can/should/must do PMTUD to validate the MTU on that path”. 
> 
> The natural responses that an endpoint goes through during path 
> validation (or abrupt migration) leave it with either a working or 
> failing path when trying to migrate; this was always the case and is no 
> more or less the case due to MTU concerns. 
> 
> Alternately, you also note that we might want to avoid forcing the 
> server specifically to send large packets — in my opinion, that would 
> be another equally valid way to tackle this: require the padding only 
> for PATH_CHALLENGES sent from the client and any PATH_RESPONSE to a 
> padded challenge.
> That’s really nice in a number of ways, especially for the MOTS cases 
> where I can make the server think that every new packet is on a new 
> path, etc.
> 
> It seems like these would be good options to use if we think that 
> there’s something we *don’t* like about the 3x limit (perhaps how it 
> interacts with NAT rebinding). Otherwise, the 3x limit is nice and 
> general.
>

---

From: Eric Kinnear <[email-masked]>
Date: Thu, 29 Oct 2020 15:07:06 -0700

Hi Martin, 

> On Oct 28, 2020, at 10:17 PM, Martin Thomson <[email-masked]> wrote:
> 
> Hi Eric,
> 

The one place where it seems that the proposed change (which is otherwise very nice, general, and fairly clean) could introduce some issues is around NAT rebinding — consider what could be a fairly common case: 

Client is fetching a decently large web resource from a server over QUIC. NAT rebinds and so the server sees un-padded QUIC packets arriving on a different port. We’ve said that the server does not need to reset the CC state if it has strong reason to believe that it’s likely the same path, and in this case the fact that the address did not change and it was solely the port is sufficient for that server implementation to match that condition.

With the 3x amplification limit, the server can only send 3x the bytes that are coming in — normally, this is fine because an endpoint is also getting padded PATH_CHALLENGE packets from the remote endpoint. However, in this case the client isn’t aware that it has migrated (this is also the case with a number of other MOTS attacks), and so is just sending a steady stream of small packets containing ACKs.

> I got the impression that Christian's request was that we avoid padding where it is unnecessary.

This case described above is what I thought Christian was describing in the final paragraph of his message, but it’s quite likely I’m wrong there. :) 

When we enter into this situation, we’re now getting fairly minimally-sized packets for at least the first RTT until the client responds with a padded PATH_RESPONSE to our (maybe not padded due to the 3x limit) PATH_CHALLENGE.

After that first RTT, the fact that we’re getting back a padded PATH_RESPONSE doesn’t matter for path validation because the path is already considered valid and we’re free to continue sending full-sized flights as allowed by our not-reset congestion controller.

And of course, I completely agree with you that we can use the actual data we’re trying to deliver for that MTU validation rather than wasting space with PADDING frames.

I’m not bothered by a requirement to do another PATH_CHALLENGE to validate MTU (wiring that up anywhere would work for PMTUD, path validation seems fine) afterward, the potential problem is simply the larger-than-before reduction in delivery rate.

The reason I ask “how worried” we are about that is because it’s a bit of an edge case, things were never going to be perfectly smooth during a NAT rebinding, etc. This happens to be one specific case that’s almost the most common of the unintentional migration cases, but even here we don’t usually expect a NAT rebinding in the middle of an active flow that’s trying to send significant amounts of data in each flight.

So, it seems to me that a completely reasonable consensus here would be “yeah, that’s a thing, but we don’t think that will matter in practice”. :) 

> Padding packets sent on either path is sub-optimal if you already know that the MTU is OK, so I'd be happy to create a carve-out for that.  It seems like a fine optimization, but the space can be u…

This totally makes sense, I think the amplification issue introduced by the padding requirement could be sorted out if we only applied that padding requirement to the one of two path validation situations that need it (i.e. the one where the client is sending PATH_CHALLENGEs, if we’re requiring the PATH_RESPONSE to come back on the same path). So just noting that this is one area where there may be further fallout and since we were discussing reducing change and risk of fallout it seemed like it would be good to consider options that reduce the scope of the previous change, rather than building on it.

If we look at that and say, “yeah, but this approach is unlikely to cause real-world fallout and the 3x limit is more general and clean”, I’m totally in agreement with that and have no issue with #4264.

Thanks,
Eric

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 30 Oct 2020 09:53:16 +1100

Thanks for clarifying Eric,

On Fri, Oct 30, 2020, at 09:07, Eric Kinnear wrote:
> Client is fetching a decently large web resource from a server over 
> QUIC. NAT rebinds and so the server sees un-padded QUIC packets 
> arriving on a different port. 

Ah, so I had in my mind that the server would be able to treat the new address as validated if only the port changed.  Then it wouldn't be obligated to perform address validation or limit its sending.

This is not what the draft says.  The same-port exception only applies to the congestion controller.  (https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-9.4-2)

I guess that my implementation will be off-spec in this regard.  I have no intention of splitting the logic so that a port rebinding retains congestion control and RTT state, but not address validation state.

The question then is whether this is worth permitting in the spec.  ?

---

From: Christian Huitema <[email-masked]>
Date: Thu, 29 Oct 2020 16:42:20 -0700

On 10/29/2020 3:53 PM, Martin Thomson wrote:
> The question then is whether this is worth permitting in the spec.

First, let's look at Eric's scenario, a client that is fetching a big
load of data, and sending back a steady stream of ACKs. Under these
circumstances, NAT rebinding ought to be fairly rare. NATs don't
actually "rebind". They create state on first packet from client, and
discard it according to some resource management algorithm. In the
proposed scenario, the NAT sees that the port is in use and it will take
some extra pressure or some software bugs to discard the NAT state. If
that happen, incoming packets from the server will be lost until the
client sends a packet, the NAT installs some new binding state, and the
server decides to send on the new path. Bottom line, we are talking
about an error scenario, which we should not try to optimize it too much.

My concern is the opposite. Over-optimization of the NAT rebinding
scenario creates a significant attack surface. If sending a tiny packet
triggers a series of fully padded path challenges, attackers can use
that creatively. I would rather **not* optimize the NAT rebinding
scenario for performance, and look for robustness instead. On the server
side, send short challenges without padding to assess address ownership
without opening a DDOS avenue. Worry about path MTU later.

That means doing thing differently for a regular migration and for a NAT
rebinding. A regular migration happens starts the client sends a full
size packet, with a not yet seen CID, and containing a path challenge.
Responding to that with a full size response makes sense. But if the
server receives a short packet, with an already used CID, and without a
path challenge, that's probably a NAT rebinding. Responding with full
size challenge packets is counter-productive.

-- Christian Huitema

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 30 Oct 2020 11:09:05 +1100

> size challenge packets is counter-productive.

Ahh, that seems sensible.  Conveniently, the current PR results in what you describe, so I have less work to do :)

---

From: Eric Kinnear <[email-masked]>
Date: Thu, 29 Oct 2020 18:27:52 -0700

Agreed, I think that’s where the “client needs to pad PATH_CHALLENGE and any PATH_RESPONSE needs to be padded if the challenge was padded” came from. 

To the point about it being an error case — that totally makes sense, and as Christian points out, that is also significantly more likely under some of the attacks than someone being both idle and transferring lots of data at the same time. 

I’m pretty sure we did previously have text that allowed the server to treat the new address as validated if only the port changed, but we took it out to help with some of the MOTS attacks. Also fully agreed that it would be really nice to not split the logic across retaining some things (CC/RTT) some of the time, but not others (Path, MTU validation). 

All this said, I suspect these are all edge cases enough that the more conservative PR (as currently written) would be totally sufficient. 
My current feeling is that if we wanted to carve it out such that a small packet from an attacker on the side (or an unintentional migration) didn’t generate MAX(full_size_packet, 3x_what_came_in_from_the_attacker) sized packets, that would be neat, but not absolutely necessary.

Thanks,
Eric

> On Oct 29, 2020, at 5:09 PM, Martin Thomson <[email-masked]> wrote:
>

---

From: Ian Swett <[email-masked]>
Date: Fri, 30 Oct 2020 23:08:20 -0400

Thanks for all the awesome discussion.

It sounds like we're landing on text that requires the 3x limit be enforced
until path validation succeeds, even if it's believed it's NAT rebinding.
I don't think that's a huge performance hit for the reasons stated above,
but it does add some complexity for some implementations, so I'm not sure
if it'll be widely enforced or not?

To answer Jana's question(far above) about resetting the congestion
controller, my thinking is the following.  In cases when the server
believes it's NAT migration, it does not reset the congestion controller
and this limits the potential attack to the congestion window built up
prior to the migration.  Additionally, the server has to believe it's a NAT
migration, which makes the attack unpredictable or useless in the presence
of modern NATs and Firewalls.

So what I'd argue for is that NAT migrations MAY be treated as validated
paths, but the congestion controller MUST NOT be reset in that case.  I
think this fits what MT said their implementation currently did and what I
believe ours does today as well.

Thanks, Ian

On Thu, Oct 29, 2020 at 9:28 PM Eric Kinnear <ekinnear=
[email-masked]> wrote:

> Agreed, I think that’s where the “client needs to pad PATH_CHALLENGE and
> any PATH_RESPONSE needs to be padded if the challenge was padded” came
> from.
>
> To the point about it being an error case — that totally makes sense, and
> as Christian points out, that is also significantly more likely under some
> of the attacks than someone being both idle and transferring lots of data
> at the same time.
>
> I’m pretty sure we did previously have text that allowed the server to
> treat the new address as validated if only the port changed, but we took it
> out to help with some of the MOTS attacks. Also fully agreed that it would
> be really nice to not split the logic across retaining some things (CC/RTT)
> some of the time, but not others (Path, MTU validation).
>
> All this said, I suspect these are all edge cases enough that the more
> conservative PR (as currently written) would be totally sufficient.
> My current feeling is that if we wanted to carve it out such that a small
> packet from an attacker on the side (or an unintentional migration) didn’t
> generate MAX(full_size_packet, 3x_what_came_in_from_the_attacker) sized
> packets, that would be neat, but not absolutely necessary.
>
> Thanks,
> Eric
>
>
> > Ahh, that seems sensible.  Conveniently, the current PR results in what
> you describe, so I have less work to do :)
>
>

---

From: Martin Duke <[email-masked]>
Date: Sat, 31 Oct 2020 13:41:33 -0700

I want to carefully think through the various scenarios here, two "normal"
ones and two attack ones. But the summary is that the proposal to always
use the amp limit is good, if we add a coupe of more recommendations for
clients (in bold below)

1) Normal NAT rebinding. The connection is idle for a long period. A good
server will have reset its congestion controller due to the long idle
period.

The client then sends a packet (if the server does first, it'll probably
disappear).  Whether or not that packet is small, the server is going to be
limited by the amplification limit rather than congestion control. One
alternative is to kill the connection after an idle period and restart with
0RTT, but that's only superior because the client has to send a bunch of
bytes. The recommendation here, I think, is that *clients SHOULD send one
or more full-size datagrams when restarting after a long idle period*, *and
possibly reset its PMTU assumptions. *[I can't remember what DPDLPMTUD says
about idle periods]

If the client does this, the server will be able to validate address and
MTU in one RTT. Otherwise, it's going to have to take two. But the
recommendation is sufficient to mitigate the impact of the 3x limit in NAT
rebinding cases.

2) Spoofed NAT rebinding. An attacker rewrites source addresses on someone
else's packets, or opens a valid connection and then switches the source
address to the victim. The amplification limit will prevent anything bad
from happening here, then PATH_CHALLENGE will fail and we're done.

3) Normal Migration. The client suddenly changes the CIDs and source IP.
(a) If the path is pre-validated, there is no issue. PATH_CHALLENGE and
PATH_RESPONSE SHOULD have been padded to handle MTU and address validation
simultaneously.
(b) If the path is not-prevalidated, the client SHOULD pad PATH_CHALLENGE.
This will be more restrictive than init-cwnd, unless *the client sends 3
datagrams or so, padded pings if necessary.*

4) Spoofed Migration: the attacker opens a connection, then sends packets
with new CIDs and the victim's IP. It is impossible for this to be
pre-validated. Because the attacker has to send 1 byte for every 3 the
victim gets, this is safe.

I'll propose some text in review.

On Fri, Oct 30, 2020 at 8:08 PM Ian Swett <ianswett=
[email-masked]> wrote:

> It sounds like we're landing on text that requires the 3x limit be
> enforced until path validation succeeds, even if it's believed it's NAT
> rebinding.  I don't think that's a huge performance hit for the reasons
> stated above, but it does add some complexity for some implementations, so
> I'm not sure if it'll be widely enforced or not?
>