Subject: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)

From: Adam Roach via Datatracker <[email-masked]>
Date: Tue, 21 Jan 2020 21:26:47 -0800

Adam Roach has entered the following ballot position for
draft-ietf-intarea-provisioning-domains-10: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-intarea-provisioning-domains/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Thanks to the authors and working group for their work on this document.  I
have one major concern about the ability for this mechanism to be abused to
form DDoS attacks, described below. Unfortunately, while I have identified the
attack, I don't have an easy solution to propose that mitigates it satisfactorily.

I also have a handful of mostly editorial comments on the document.

---------------------------------------------------------------------------

§6:

I was expecting to see a discussion of the DDoS attack that may result from a
large network (or a rogue host on such a network) sending out a PvD ID
containing the hostname of a victim machine, and setting the "H" flag.

Since the messages used to trigger these HTTP connections are extremely
lightweight, unauthenticated UDP messages, and the resulting HTTP connections
require the exchange of a significant number of packets in addition to a
number of cryptographic operations, this is a very high ratio amplification
attack, both in terms of network and CPU resources.

Given that the delay setting comes from the network instead of being
independently computed by the host, such an attack could be honed to be
particularly devastating.  Although it isn't a complete mitigation, one
approach to consider would be moving computation of the delay upper bound to
the host, or specifying a minimum upper bound of several minutes (where a
smaller value will cause the host to use this minimum upper bound).

Regardless of how this is ultimately handled, I think this is a pretty severe
risk that needs addressing in the document prior to publication.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

>  This document also introduces a mechanism for hosts to retrieve
>  optional additional information related to a specific PvD by means of
>  an HTTP over TLS query using an URI derived from the PvD ID.

Nit: "...a URI..."

---------------------------------------------------------------------------

§3.4.1:

>  This is intended to
>  resolve backward compatibility issues with rare deployments choosing
>  to assign addresses with DHCPv6 while not sending any matching PIO.

It seems that this set of circumstances could also arise due to a
misconfiguration of DHCPv6. If this is expected to be only rarely
intended, perhaps some oprationational/implementation guidance to log
a warning or otherwise alert the operator would be helpful.

---------------------------------------------------------------------------

§4.1:

>  HTTP requests and responses for PvD additional information use the
>  "application/pvd+json" media type (see Section 8).  Clients SHOULD
>  include this media type as an Accept header in their GET requests,

Nit: "...Accept header field..."

---------------------------------------------------------------------------

§4.1:

>  If the HTTP
>  status of the answer is between 200 and 299, inclusive, the host MAY
>  get a file containing a single JSON object.

This is very confusing phrasing. The sentence -- and the use of a normative
"MAY" in particular -- indicates that the host is given permission to take
some additional action that "gets" a JSON object from somewhere. I think it's
intending to say that a 200-class HTTP response will contain such an object.

Consider rephrasing.

---------------------------------------------------------------------------

§4.3:

>  Private-use or experimental keys MAY be used in the JSON dictionary.
>  In order to avoid such keys colliding with IANA registry keys,
>  implementers or vendors defining private-use or experimental keys
>  MUST create sub-dictionaries, where the sub-dictionary is added into
>  the top-level JSON dictionary with a key of the format "vendor-*"
>  where the "*" is replaced by the implementer's or vendor's
>  identifier.  For example, keys specific to the FooBar organization
>  could use "vendor-foobar".  Upon receiving such a sub-dictionary,
>  host MUST ignore this sub-dictionary if it is unknown.  When the
>  vendor or implementer is part of an IANA URN namespace [URN], the URN
>  namespace SHOULD be used rather than the "vendor-*" format.

This is kind of a minor nit, but this paragraph is a bit confusing.  It
starts off with a less-preferred convention ("vendor-*") and discusses
it as if it were the only way to do things; and then it throws in a
SHOULD-strength different encoding at the end as a surprise twist.
I would suggest reworking the paragraph so that the preferred encoding
(URNs) are mentioned first, as a SHOULD-strength statement, followed by
the less-preferred "vendor-*" as a fallback.

---------------------------------------------------------------------------

§4.3:

>  +------------+-----------------+-----------+------------------------+
>  | JSON key   | Description     | Type      | Example                |
>  +------------+-----------------+-----------+------------------------+
>  | identifier | PvD ID FQDN     | String    | "pvd.example.com."     |

...

>  {
>    "identifier": "cafe.example.com",
>    "expires": "2017-07-23T06:00:00Z",
>    "prefixes": ["2001:db8:1::/48", "2001:db8:4::/48"],
>  }

I'm concerned about the variation in the identifier field alternately
containing and not containing the terminal dot of the FQDN. If the
intention that these are to be equivalent, it would probably head off
some implementation incompatibilities if the document were to say so
explicitly.

---------------------------------------------------------------------------

§7:

>  without leaking identity information, SHOULD make use of an IPv6
>  Privacy Address and SHOULD NOT include any privacy sensitive data,
>  such as User Agent header or HTTP cookie, while performing the HTTP

Nit: "...User-Agent header field..."
             ^             ^^^^^

---

From: Warren Kumari <[email-masked]>
Date: Wed, 22 Jan 2020 09:54:23 -0500

> https://datatracker.ietf.org/doc/draft-ietf-intarea-provisioning-domains/

Doh!

Yes, unless we are missing something, this *does* sound like an issue
-- I like your minimum upper bound idea... but I'm also picturing
something where the victim sends back some bad PvD info which makes
the clients fall off the network, thereby stopping the attack :-P

> risk that needs addressing in the document prior to publication.

Yah.
W

> ....

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 07:52:28 -0800

> On Jan 22, 2020, at 6:54 AM, Warren Kumari <[email-masked]> wrote:
> 

There should certainly be a minimum time between fetches, which we can specify here.

Beyond this, I think there's actually a relatively straightforward solution that's latent in the text but needs to be called out in the security considerations.

The privacy considerations note that the Additional Information server should limit access to valid clients (and if it checks the client public address, this can prevent the NAT66 trick):

Network operators SHOULD restrict access to PvD Additional
Information to only expose it to hosts that are connected to the local
network... [this] can be implemented by
whitelisting access from the addresses and prefixes that the router provides
for the PvD, which will match the prefixes contained in the PvD Additional
Information.

In addition, the client is told to "abandon" requests and consider that a PvD has no additional info if it receives HTTP 400 errors:

If the HTTP status of the answer is greater than or equal to 400
the host MUST abandon and consider that there is no additional PvD
information.

My thought is to specify that the server rejects clients with a 400-level if they are not indeed from its own PvD; and similarly to specify that clients don't retry and effectively blacklist the PvD Additional Info in this case, or if they don't get valid JSON back.

Thoughts?

Thanks!
Tommy

---

From: Ted Lemon <[email-masked]>
Date: Wed, 22 Jan 2020 11:07:30 -0500

> for the PvD, which will match the prefixes contained in the PvD Additional

But does this help with the problem Adam is talking about?  The attack is coming from the RA. A rogue RA will not be in control of the network operator. So the mitigation has to be on the recipient of the RA, I think. 

So your suggestion to abandon the query of the host gets a bad answer sounds okay-ish, but probably we could do better by randomizing the query time on the client and the like. 

Beyond that, is there any way to limit the scope of the query so that the attack isn’t useful?  My first instinct about this is that the attack isn’t very useful because it requires an attacker on the local net, but we’ve seen the power of the Mira attack; this would be a significant increase in these attack’s effectiveness. 

Requiring the http listener to be local could make this pretty useless as an attack. It’s inconvenient, but probably doable. Is there a reason not to do it?

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 08:44:07 -0800

As I understand the described attack, we are concerned with a case in which:

- A rogue RA is sent by the attacker to one or more clients
- The clients are being used to amplify by sending HTTP requests to a victim server
- The HTTP server receives a high load of attack requests from unwilling clients

The mitigations fall into two general categories: (1) ensuring that clients can be less easily tricked into being complicit in the attack; (2) and having the HTTP server protect itself.

The client adding a limit to its fetching frequency addresses part of (1). For the rest, there are two cases: either the host being pointed to is indeed a PvD Additional Information server (but not one working with the attacker), or else is some other server that has no idea about PvDs. If the case is the latter, trying to request the well-known PvD information won't work and will generate an error. That will let clients know that something's up and that they should stop trying to frequently fetch this file. If the case is the former, we can give advice to PvD Additional Information server to make sure it generates errors in cases that seem fishy. The client already validates that its RA prefixes match the additional info; and if the server validates that the client is coming from the expected prefix it manages, it can detect cases in which the attacker is trying to point load at it from other networks.

This validation isn't *exactly* the same as requiring that the HTTP server is local, but it's similar, and gets the same benefits of limiting the scope. The main thing it allows is the HTTP server to be a bit behind the local network, but still be aware of the expected prefixes, etc.

Thanks,
Tommy

> On Jan 22, 2020, at 8:07 AM, Ted Lemon <[email-masked]> wrote:
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 10:51:04 -0600

> for the PvD, which will match the prefixes contained in the PvD Additional

To be clear, I'm far less concerned about DDoS attacks on PvD Additional 
Information servers [1]  than I am about attacks on general-purpose web 
servers.

> My thought is to specify that the server rejects clients with a 400-level if they are not indeed from its own PvD; and similarly to specify that clients don't retry and effectively blacklist the Pv…

Durable blocklisting (e.g., blocks that survive reboots) of specific PvD 
IDs on 400-class errors certainly helps in some cases (non-PvD servers 
will likely return 401, 403, or 410 for PvD requests), but it falls down 
in the fairly common case of web servers pointed to by wildcard DNS 
records. For example, an attacker could flood github.io by first sending 
out a PvD id of "1.github.io" and then, milliseconds later, sending out 
a PvD id of "2.github.io", and so on, indefinitely.

So, a few additional thoughts to throw into the hopper -- all of which 
have some drawbacks -- would be to consider:

  * Instead of blocklisting individual PvD IDs, blocklist all hosts
    under a corresponding eTLD [2]
  * If a PvD Additional Information server returns a 400-class response,
    persistently block the sender of the RA (I don't know offhand how
    one might do this in a way that can't be evaded, though. Source IP
    address clearly won't work, and source MAC is probably too easy to
    change as well)
  * Add a mechanism to the RA interaction that requires at least as much
    work on the attacker's part (both in terms of network usage and
    computational complexity) as the HTTPS connections themselves, thus
    eliminating the amplification (think challenge/response
    proof-of-work approaches here).
  * Add a large, global (i.e., not per-PvD ID) throttle (on the order of
    minutes or hours) between acting on received PvD RAs
  * Require clients to check in DNS (or some other highly scalable
    database) that the PvD Additional Information server has opted in to
    being a PvD Additional Information server. As an example of one
    possible way to do this: for the PvD ID foo.example.com, clients
    would have to first check for the presence of a DNS TXT record
    _pvd.foo.example.com. (See RFC 8552 if you choose to do something
    like this).

/a

____
[1] The optional nature of the information they serve makes them 
non-critical
[2] I know you know what this is, but for the benefit of others, see 
https://en.wikipedia.org/wiki/Public_Suffix_List

---

From: Ted Lemon <[email-masked]>
Date: Wed, 22 Jan 2020 11:53:21 -0500

> The mitigations fall into two general categories: (1) ensuring that clients can be less easily tricked into being complicit in the attack; (2) and having the HTTP server protect itself.

For a DDoS attack, if you get to (2) you are already somewhat dead.

> This validation isn't *exactly* the same as requiring that the HTTP server is local, but it's similar, and gets the same benefits of limiting the scope. The main thing it allows is the HTTP server…

If the client has to contact the server to see that it’s an invalid server, then you have a DDoS attack.   I think if you do what you suggest, then the scope of the DDoS attack will be limited, to the extent that the server can at least say “you got the wrong number” and the client can then cache the info in the RA as “bogus” for some period of time, so that the DDoS is relatively brief.  But the server still needs to be able to answer in order for this to work.

It would be nice if there were a way for the client to validate the server information before contacting it.   I’m not convinced there’s a way to do that, but maybe some kind of DNSSEC info that has to be fetched first?   Of course, that could also be used as an attack vector, since the RA can include an RDNSS option.

Adam, do you think that randomizing the client connect time is enough to mitigate a Mira-style attack, when accompanied with the other mitigations Tommy has suggested?

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 10:53:34 -0600

On 1/22/20 10:51, Adam Roach wrote:
> non-PvD servers will likely return 401, 403, or 410 for PvD requests

Sorry, I meant 403, 404, or 410, but it doesn't really change the point. :)

/a

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 11:01:21 -0600

On 1/22/20 10:51, Adam Roach wrote:
> Instead of blocklisting individual PvD IDs, blocklist all hosts under 
> a corresponding eTLD [2]

Sorry, going too fast this morning. Make that eTLD+1. And now I'm going 
to go get more coffee.

/a

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 09:22:45 -0800

Hi Adam,

As we discuss the main DDoS attack case, I did want to also reply to the other comments (see inline).

I'm keeping pending changes available here, to be published after the telechat: https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>

Thanks!
Tommy

> On Jan 21, 2020, at 9:26 PM, Adam Roach via Datatracker <[email-masked]> wrote:
> 

Fixed!

Added the following sentence at the end of the paragraph:

 Implementations are suggested to flag or log
such scenarios as errors to help detect misconfigurations.

Fixed

This is similar to Alissa's comment. The new text is:

If the HTTP status of
the answer is between 200 and 299, inclusive, the response is expected to
be a single JSON object.

This is also similar to Alissa's comment. I've updated the URN text, but also reordered the text. The paragraph now reads:

Private-use or experimental keys MAY be used in the JSON
dictionary. In order to avoid such keys colliding with IANA registry
keys, implementers or vendors defining private-use or experimental
keys MUST create sub-dictionaries. If a set of PvD Additional Information keys
are defined by an organization that has a Formal URN Namespace {{URN}},
the URN namespace SHOULD be used as the top-level JSON key for
the sub-dictionary. For other private uses, the sub-dictionary key
SHOULD follow the format of "vendor-\*", where the "\*" is replaced by the
implementer's or vendor's identifier. For example, keys specific to the FooBar
organization could use "vendor-foobar". If a host receives a sub-dictionary with
an unknown key, the host MUST ignore the contents of the sub-dictionary.
> some implementation incompatibilities if the document were to say so

I've updated the examples to all use a trailing dot.

Fixed!

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 13:17:18 -0800

Hi Adam,

Thanks again for bringing this up! I've updated our text to include mitigations for this attack. It can be found here (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25), but here's an overview of the proposed text:

In Section 4.1, I've added two new paragraphs. The first describes time limits on fetching PvD info:

In addition to adding a random delay when fetching Additional Information, hosts
MUST enforce a minimum time between requesting Additional Information
for a given PvD on the same network. This minimum time is RECOMMENDED
to be 10 seconds, in order to avoid hosts causing a denial-of-service on the
PvD server. Hosts also MUST limit the number of requests that are made to
different PvD Additional Information servers on the same network within a short
period of time. A RECOMMENDED value is to issue no more than five PvD
Additional Information requests in total on a given network within 10 seconds.
For more discussion, see {{security}}.

The second also makes clear the behavior to take in case of failure, which will be the case for non-PvD web servers:

If the request for PvD Additional Information fails due to a TLS error,
an HTTP error, or because the retrieved file does not contain valid PvD JSON,
hosts MUST close any connection used to fetch the PvD Additional Information,
and MUST NOT request the information for that PvD ID again for the duration
of the local network attachment. For more discussion, see {{security}}.

In addition, I added text to the Security Considerations:

An attacker generating RAs on a local network can use the H-flag and the PvD ID
to cause hosts on the network to make requests for PvD Additional Information
from servers. This can become a denial-of-service attack if not mitigated. To mitigate
this attack, hosts MUST limit the rate at which they fetch a particular PvD's
Additional Information, limit the rate at which they fetch any PvD Additional
Information on a given local network, and stop making requests to any PvD ID
that does not respond with valid JSON. Details are provided in {{retr}}. This attack
can be targeted at generic web servers, in which case the host behavior of stopping
requesting for any server that doesn't behave like a PvD Additional Information server
is critical. For cases in which an attacker is pointing hosts at a valid PvD Additional
Information server (but one that is not actually associated with the local network),
the server SHOULD reject any requests that do not originate from the expected IPv6
prefix as described in {{serverop}}.

The existing text referenced here about server behavior is:

The server providing the JSON files SHOULD also check whether the
client address is contained by the prefixes listed in the additional
information, and SHOULD return a 403 response code if there is no
match.

Let me know if this addresses your concerns!

Best,
Tommy

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 15:51:02 -0600

Thanks! The new text is good, but I don't think it's sufficient. I have 
two remaining concerns in particular:

  * The mitigation for wildcarded web hosts appears inadequate,
    especially given:
  * The mechanism clearly anticipates a scale where it can generate
    *single* short torrential burst sufficient to knock an average
    server over (hence the random delay mechanism for fetching data over
    HTTP). Given that fact, simple rate-limiting will never be enough if
    a single tight burst of traffic can be orchestrated.

The more I think about it, the more I believe the TXT-based opt-in 
solution I proposed in my earlier email is a reasonable approach to 
protect general-purpose web servers from PvD-client-based attacks.

One further comment inline below.

/a

On 1/22/20 15:17, Tommy Pauly wrote:
> Hi Adam,
>
> Thanks again for bringing this up! I've updated our text to include 
> mitigations for this attack. It can be found here 
> (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25), but here's an 
> overview of the proposed text:
>
> The second also makes clear the behavior to take in case of failure, 
> which will be the case for non-PvD web servers:
>
>
>     from servers. This can become a denial-of-service attack if not
>     mitigated.
>

This doesn't really convey the amplification involved, which I think is 
highly relevant.

>     To mitigate
>> On Jan 21, 2020, at 9:26 PM, Adam Roach via Datatracker 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 14:22:05 -0800

Hi Adam,

Thanks for taking a look! I'd like to avoid adding extra checks, such as looking for particular DNS records, to avoid deployment complexity and more opportunities for incomplete configuration. As such, I'd like to dig into this a bit further.

If the attacker in this case is a rogue actor on a local network sending out RAs on their local link, any given attacking host would be restricted in their scope to the devices they can reach. Of course, there could be coordination across many different local networks simultaneously, but that also requires more work on the side of the attacker.

The reason for the delay was to limit the impact on a relatively low-powered and unsophisticated local HTTPS server for serving PvD information, which may itself be on the router. I imagine that any large web server deployment would not have any issue with the load generated from a particular local network. Specifically, if we are limiting any given host to requesting only a few times within a 10 second window on the network at all, and the number of hosts on the network is bounded, the number of opportunities for the attacker to cause load on the servers is limited.

Another option, to avoid remaining concern about hitting wildcarded hosts, is to simply say that if the host keeps receiving PvD IDs with bogus (failing) servers, it disables all fetching of additional information for the duration of the network attachment. Networks that do implement better control over RAs (RA-guard, etc) presumably won't have this issue, and since the additional info is optional, it shouldn't cause any major connectivity issues.

If we require such a limit (you only get to fail to fetch 10 times total per attachment, say), does that mitigate things?

Thanks,
Tommy

> On Jan 22, 2020, at 1:51 PM, Adam Roach <[email-masked]> wrote:
> 
> Thanks! The new text is good, but I don't think it's sufficient. I have two remaining concerns in particular:
> The mitigation for wildcarded web hosts appears inadequate, especially given:
> The mechanism clearly anticipates a scale where it can generate *single* short torrential burst sufficient to knock an average server over (hence the random delay mechanism for fetching data over HTTP). Given that fact, simple rate-limiting will never be enough if a single tight burst of traffic can be orchestrated.
> The more I think about it, the more I believe the TXT-based opt-in solution I proposed in my earlier email is a reasonable approach to protect general-purpose web servers from PvD-client-based attacks.
> 
>> Thanks again for bringing this up! I've updated our text to include mitigations for this attack. It can be found here (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>), but here's an overview of the proposed text:
>> 
>> from servers. This can become a denial-of-service attack if not mitigated.
> 
> This doesn't really convey the amplification involved, which I think is highly relevant.
> 
> 
> 
>> To mitigate
>>> On Jan 21, 2020, at 9:26 PM, Adam Roach via Datatracker <[email-masked] <mailto:[email-masked]>> wrote:
>>>

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 16:58:16 -0600

Thanks for the explanation and the further proposed mitigation.

Allowing the RA to specify an arbitrarily small "Delay" parameter seems 
to still allow for a pretty big burst of traffic. If I read the proposed 
interpretation of the "Delay" bits correctly (2**(Delay * 2)), the 
current behavior is specified to allow a delay upper bound selected from 
one of the following (approximate) values:

  * 1 ms
  * 4 ms
  * 16 ms
  * 64 ms
  * 256 ms
  * 1 second
  * 4 seconds
  * 16 seconds
  * 1 minute
  * 4 minutes
  * 17 minutes
  * 70 minutes
  * 4 hours, 40 minutes
  * 18 hours 38 minutes
  * 3 days, 3 hours
  * 1 week, 5 days

That's a pretty breathtaking scope, and it's hard to imagine that the 
first six or so are strictly needed, while all six are in a range that 
might overload a DDoS target. The final several seem a bit questionable 
as well, given normal operational timelines for network attachment. If 
the formula were revised to, e.g., "2**(Delay + 12)" instead of the 
current formula, you would have an enforced lower bound of roughly four 
seconds (which should be enough to blunt most DDoS attacks), and an 
upper bound of roughly 37 hours (which still seems excessive, although 
not quite as much as the previous upper bound).

Assuming the additional mitigation you propose below (10 maximum 
failures per attachment) as well as some means of achieving a 
lower-bound for "Delay" on the order of multiple seconds, I think I'm 
good clearing when a new version comes out.

Thanks for your work in thinking through practical solutions to this issue.

/a

On 1/22/20 16:22, Tommy Pauly wrote:
> Hi Adam,
>
> Thanks for taking a look! I'd like to avoid adding extra checks, such 
> as looking for particular DNS records, to avoid deployment complexity 
> and more opportunities for incomplete configuration. As such, I'd like 
> to dig into this a bit further.
>
> If the attacker in this case is a rogue actor on a local network 
> sending out RAs on their local link, any given attacking host would be 
> restricted in their scope to the devices they can reach. Of course, 
> there could be coordination across many different local networks 
> simultaneously, but that also requires more work on the side of the 
> attacker.
>
> The reason for the delay was to limit the impact on a relatively 
> low-powered and unsophisticated local HTTPS server for serving PvD 
> information, which may itself be on the router. I imagine that any 
> large web server deployment would not have any issue with the load 
> generated from a particular local network. Specifically, if we are 
> limiting any given host to requesting only a few times within a 10 
> second window on the network at all, and the number of hosts on the 
> network is bounded, the number of opportunities for the attacker to 
> cause load on the servers is limited.
>
> Another option, to avoid remaining concern about hitting wildcarded 
> hosts, is to simply say that if the host keeps receiving PvD IDs with 
> bogus (failing) servers, it disables all fetching of additional 
> information for the duration of the network attachment. Networks that 
> do implement better control over RAs (RA-guard, etc) presumably won't 
> have this issue, and since the additional info is optional, it 
> shouldn't cause any major connectivity issues.
>
> If we require such a limit (you only get to fail to fetch 10 times 
> total per attachment, say), does that mitigate things?
>
> Thanks,
> Tommy
>
>> On Jan 22, 2020, at 1:51 PM, Adam Roach <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Thanks! The new text is good, but I don't think it's sufficient. I 
>> have two remaining concerns in particular:
>>
>>     server over (hence the random delay mechanism for fetching data
>>     over HTTP). Given that fact, simple rate-limiting will never be
>>     enough if a single tight burst of traffic can be orchestrated.
>>
>>> (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25), but here's 
>>> an overview of the proposed text:
>>>

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 15:51:54 -0800

Hi Adam,

Thanks for the feedback! The updated paragraph in the retrieval section, to indicate a maximum failure count per attachment, is:

If the request for PvD Additional Information fails due to a TLS error,
an HTTP error, or because the retrieved file does not contain valid PvD JSON,
hosts MUST close any connection used to fetch the PvD Additional Information,
and MUST NOT request the information for that PvD ID again for the duration
of the local network attachment. If a host detects 10 or more such failures
to fetch PvD Additional Information, the local network is assumed to be
misconfigured or under attack, and the host MUST NOT make any further
requests for PvD Additional Information, belonging to any PvD ID, for
the duration of the local network attachment. For more discussion, see {{security}}.

I've also expanded the security considerations DoS section as follows:

An attacker generating RAs on a local network can use the H-flag and the PvD ID
to cause hosts on the network to make requests for PvD Additional Information
from servers. This can become a denial-of-service attack, in which an attacker
can amplify its attack by triggering TLS connections to arbitrary servers in response
to sending UDP packets containing RA messages. To mitigate this attack, hosts
MUST:

- limit the rate at which they fetch a particular PvD's Additional Information;
- limit the rate at which they fetch any PvD Additional Information on a given local
network;
- stop making requests for a PvD ID that does not respond with valid JSON;
- stop making requests for all PvD IDs once a certain number of failures is reached
on a particular network.

Details are provided in {{retr}}. This attack can be targeted at generic web servers,
in which case the host behavior of stopping requesting for any server that doesn't
behave like a PvD Additional Information server is critical. Limiting requests for
a specific PvD ID might not be sufficient if the attacker changes the PvD ID values
quickly, so hosts also need to stop requesting if they detect consistent failure when
on a network that is under attack. For cases in which an attacker is pointing hosts at
a valid PvD Additional Information server (but one that is not actually associated
with the local network), the server SHOULD reject any requests that do not originate
from the expected IPv6 prefix as described in {{serverop}}.

For the delay calculation, you make a good point that the larger values get pretty unnecessarily large! I'm a bit concerned about making the minimum fetch range be ~4 seconds, as that could end up being user visible for some valid scenarios. How about making the formula "2**(10 + Delay)":

The target time for the delay is calculated
as a random time between zero and 2**(10 + Delay) milliseconds,
where 'Delay' corresponds to the 4-bit unsigned integer in
the last received PvD Option.

This limits it to 1 second as what the RA can request for fastest frequency bound. This isn't incredibly fast, and with the overall limits for how many requests can be made by a client (which provide the larger portion of the DoS prevention, I'd argue), I think this strikes a good balance between usability and precaution. Thoughts?

I've updated the GitHub text for anyone wanting to see the full flow: https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>

Thanks,
Tommy

> On Jan 22, 2020, at 2:58 PM, Adam Roach <[email-masked]> wrote:
> 
> Allowing the RA to specify an arbitrarily small "Delay" parameter seems to still allow for a pretty big burst of traffic. If I read the proposed interpretation of the "Delay" bits correctly (2**(Delay * 2)), the current behavior is specified to allow a delay upper bound selected from one of the following (approximate) values:
> 
> 1 ms
> 4 ms
> 16 ms
> 64 ms
> 256 ms
> 1 second
> 4 seconds
> 16 seconds
> 1 minute
> 4 minutes
> 17 minutes
> 70 minutes
> 4 hours, 40 minutes
> 18 hours 38 minutes
> 3 days, 3 hours
> 1 week, 5 days
> 
> That's a pretty breathtaking scope, and it's hard to imagine that the first six or so are strictly needed, while all six are in a range that might overload a DDoS target. The final several seem a bit questionable as well, given normal operational timelines for network attachment. If the formula were revised to, e.g., "2**(Delay + 12)" instead of the current formula, you would have an enforced lower bound of roughly four seconds (which should be enough to blunt most DDoS attacks), and an upper bound of roughly 37 hours (which still seems excessive, although not quite as much as the previous upper bound).
> 
> Assuming the additional mitigation you propose below (10 maximum failures per attachment) as well as some means of achieving a lower-bound for "Delay" on the order of multiple seconds, I think I'm good clearing when a new version comes out.
>

---

From: Adam Roach <[email-masked]>
Date: Wed, 22 Jan 2020 18:08:51 -0600

Thanks again for the quick turn-around on this.

Using your proposed 2**(Delay + 10) seems to strike an okay balance, if 
I'm understanding the situation correctly. Double-check my thinking 
here: the scope of RA reach from an attacker will be available only on a 
single local link, which deployments typically limit to on the order of 
500 clients or so. If all 500 are triggered at the same time and smooth 
out their requests over a one-second window, we're looking at a 500 TPS 
load on a web server. That's about 25% the capacity of a relatively 
low-end web server (e.g., Apache running on an Atom 1.66), which seems 
small enough to avoid major issues.

So, unless one of my assumptions above is wrong, I think your proposal 
below is a good solution to the issue. I'll clear my DISCUSS when a new 
version of the draft comes out (I would propose that you wait for 
instructions from your AD about when to do so).

/a

On 1/22/20 17:51, Tommy Pauly wrote:
> Hi Adam,
>
> Thanks for the feedback! The updated paragraph in the retrieval 
> section, to indicate a maximum failure count per attachment, is:
>
> For the delay calculation, you make a good point that the larger 
> values get pretty unnecessarily large! I'm a bit concerned about 
> making the minimum fetch range be ~4 seconds, as that could end up 
> being user visible for some valid scenarios. How about making the 
> formula "2**(10 + Delay)":
>
> This limits it to 1 second as what the RA can request for fastest 
> frequency bound. This isn't incredibly fast, and with the overall 
> limits for how many requests can be made by a client (which provide 
> the larger portion of the DoS prevention, I'd argue), I think this 
> strikes a good balance between usability and precaution. Thoughts?
>
> I've updated the GitHub text for anyone wanting to see the full flow: 
> https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25
>
> Thanks,
> Tommy
>
>> On Jan 22, 2020, at 2:58 PM, Adam Roach <[email-masked] 
>> Allowing the RA to specify an arbitrarily small "Delay" parameter 
>> seems to still allow for a pretty big burst of traffic. If I read the 
>> proposed interpretation of the "Delay" bits correctly (2**(Delay * 
>> 2)), the current behavior is specified to allow a delay upper bound 
>> selected from one of the following (approximate) values:
>>
>>   * 1 ms
>>   * 4 ms
>>   * 16 ms
>>   * 64 ms
>>   * 256 ms
>>   * 1 second
>>   * 4 seconds
>>   * 16 seconds
>>   * 1 minute
>>   * 4 minutes
>>   * 17 minutes
>>   * 70 minutes
>>   * 4 hours, 40 minutes
>>   * 18 hours 38 minutes
>>   * 3 days, 3 hours
>>   * 1 week, 5 days
>>
>>
>> first six or so are strictly needed, while all six are in a range 
>> that might overload a DDoS target. The final several seem a bit 
>> questionable as well, given normal operational timelines for network 
>> attachment. If the formula were revised to, e.g., "2**(Delay + 12)" 
>> instead of the current formula, you would have an enforced lower 
>> bound of roughly four seconds (which should be enough to blunt most 
>> DDoS attacks), and an upper bound of roughly 37 hours (which still 
>> seems excessive, although not quite as much as the previous upper bound).
>>

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 16:18:07 -0800

> On Jan 22, 2020, at 4:08 PM, Adam Roach <[email-masked]> wrote:
> 
> Using your proposed 2**(Delay + 10) seems to strike an okay balance, if I'm understanding the situation correctly. Double-check my thinking here: the scope of RA reach from an attacker will be available only on a single local link, which deployments typically limit to on the order of 500 clients or so. If all 500 are triggered at the same time and smooth out their requests over a one-second window, we're looking at a 500 TPS load on a web server. That's about 25% the capacity of a relatively low-end web server (e.g., Apache running on an Atom 1.66), which seems small enough to avoid major issues.

Yes, that sounds right to me. That limits the single burst size in response to a given RA being sent. Each of these hosts wouldn't request again for that PvD ID, even if RAs keep coming telling the host to update, for another ten seconds after that. And, the limit for the number of different hostnames (for a case in which a wildcard host exists, which presumably also has a higher capacity) is 5 different names in the ten second period, so that limits at 2500 TPS across all fetches to any server from a given network.
> 
> So, unless one of my assumptions above is wrong, I think your proposal below is a good solution to the issue. I'll clear my DISCUSS when a new version of the draft comes out (I would propose that you wait for instructions from your AD about when to do so).

Thanks! Yes, I'll wait for the go-ahead from Suresh. I appreciate your helping to work through these important details!

Best,
Tommy
>>> On Jan 22, 2020, at 2:58 PM, Adam Roach <[email-masked] <mailto:[email-masked]>> wrote:
>>>

---

From: Suresh Krishnan <[email-masked]>
Date: Thu, 23 Jan 2020 01:55:05 -0500

> On Jan 22, 2020, at 7:18 PM, Tommy Pauly <tpauly=[email-masked]> wrote:
> 
> 
> 
>> On Jan 22, 2020, at 4:08 PM, Adam Roach <[email-masked] <mailto:[email-masked]>> wrote:
>> 

Yes. I had asked Tommy to hold off on submitting a new version until after the telechat so that there is no confusion with different ADs reviewing different versions of the draft. Will work with Tommy to ensure that the agreed changes are reflected in the new submitted version.

Regards
Suresh

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Thu, 23 Jan 2020 13:53:01 +0000

Hello Adam,

Replying with my co-author's hat. 

First thank you for the review and the catch (I feel a little ashamed of not spotting it before!).

As noted by you and Tommy, sections 4.4 and 7 actually protect the real PvD additional information server based on the source IP address of the PvD-aware hosts but not 'normal web servers'.

In section 4.1, there is this text:
    " If the HTTP status of the answer is greater than or equal to 400 the
      host MUST abandon and consider that there is no additional PvD
      information."

This text could be augmented  to at least state 'not to be retried' for the same PvD sequence number. But, of course, the rogue RA[1] could rotate the sequence numbers... hence Tommy's proposal of rate limiting the outbound HTTPS connections is probably the best mitigation technique.

As Tommy wrote, adding a DNS TXT for an easy & quick check whether the PvD ID has actually a 'valid' additional information server is valid but add more work to do on the PvD hosts (that could be very lightweight). Even with this added complexity drawback, it sounds doable.

Good suggestion as well on the delay computation

Regards

-éric

[1] OTOH, I sincerely hope that networks are deployed nowadays with a feature such as RA-guard so that RA messages can reasonably be trusted esp in large networks.

On 22/01/2020, 06:26, "Adam Roach via Datatracker" <[email-masked]> wrote:

    

    
    
    §6:

    

    
    
    Nit: "...a URI..."

    

    

    

    
    
    Consider rephrasing.

    

    

    ...

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 22 Jan 2020 15:50:54 -0800

Hi Adam,

I've updated the GitHub text for anyone wanting to see the full flow: https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25

Thanks,
Tommy

>>> On Jan 22, 2020, at 1:51 PM, Adam Roach <[email-masked] <mailto:[email-masked]>> wrote:
>>>