Subject: [Ppm] PR #174 -- Direct Upload

From: Eric Rescorla <[email-masked]>
Date: Tue, 25 Jan 2022 11:54:52 -0800

Hi folks,

I've reviewed this PR and while I think thre are good ideas, it's
also not ready.

  https://github.com/abetterinternet/ppm-specification/pull/174/

The current design has clients always uploading to the leader. I agree
that it's limiting, but I think it's also fully worked out.

This design replaces that by having the clients always uploading to
the aggregators, except maybe they actually go through some relatively
loosely specified "intermediary". This seems not to be fully worked
out. For instance:

- What is the behavior of that intermediary?
  - Does it batch?
  - Does it do some kind of anti-fraud stuff and if so does this
    get propagated to the helper?

- What is the threat model vis-a-vis the intermediary?

- How do we deal with cases where the aggregators have inconsistent
  data, e.g., because an upload fails? Does this just fail? Do you
  retry? How do you know? Do we need some protocol to find out
  in advance so you don't get errors?

I think we need to answer this kind of design question prior to making
this change.

-Ekr

---

From: Christopher Patton <[email-masked]>
Date: Tue, 25 Jan 2022 15:58:43 -0800

Hi EKR,

At a high level, your questions seem to get at a more fundamental question
of the scope of the PPM spec over all: If the spec permits deployments with
an intermediary, does the spec need to fully specify how that intermediary
works? The consensus we reached in [3] seems to be that the answer is "no",
however we might have missed something important. See detailed responses to
your questions in-line.

> - What is the behavior of that intermediary?

Right now the PR presumes this behavior is out-of-scope. Can you think of a
reason why this behavior needs to be fleshed out in the core PPM spec?

>     get propagated to the helper?

By "anti-fraud" stuff I'm assuming you're referring to "Sybil attacks" as
described in [1]. Please correct me if I'm wrong.

Right now PPM does nothing about Sybil attacks, except that it provides an
extension point that a deployment could use for mitigating these. In this
regard, I don't think the situation has changed much.

   - Before #174, the leader controlled the set of reports that are
   aggregated, so it could mount a Sybil attack by simply stuffing the batch
   with reports it made up.
   - After #174, there may be an intermediary taht controls the set of
   reports that are aggregated, so it could mount a Sybil attack in exactly
   the same way.

Either way, an honest aggregator needs to do something to prevent Sybil
attacks. I don't think we've talked about this specifically in this WG yet,
but a good option might be the "Rate-Limited Tokens" spec being worked on
in the Privacy Pass WG [2]. A goal of ours should be to allow the
aggregators to deploy something like [2] *even when running with an
intermediary*. I believe this is possible: The clients still need to touch
the aggregator to get its HPKE public key, so it should be possible for an
aggregator to issue a token at that time.

Note that I have yet to take a close look at [2], so I might be completely
off base here :)

> - What is the threat model vis-a-vis the intermediary?

I don't believe that allowing an intermediary changes the threat model
beyond the fact that it gets to pick the set of reports that are
aggregated. As discussed above, the leader currently has the same power, so
this is already a problem. In fact, in the absence of a defense, any
aggregator can mount a Sybil attack by simply uploading report shares to
its peers. We don't want to have to trust any aggregator more than the
others, so a defense will be needed whether we take #174 or not.

>   in advance so you don't get errors?

This was discussed in detail in the previous thread, so I'll just leave a
link to Tim's summary: [3]. The gist is that many of the new failure modes
that arise are similar to failure modes that already exist, and we'll need
to deal with these in much the same way. Thus dealing with split uploads
doesn't appear to require too much additional plumbing. If we missed
something in that discussion, please let us know!

Best,
Chris P.

[1] Issue discussing Sybil attacks:
https://github.com/abetterinternet/ppm-specification/issues/89
[2] Internet-Draft: Rate-Limited Token Issuance Protocol:
https://tfpauly.github.io/privacy-proxy/draft-privacypass-rate-limit-tokens.html
[3] Tim's summary about split-upload failure modes:
https://mailarchive.ietf.org/arch/msg/ppm/OSfhcossIrEPqPhIEW5CkVcJhik/

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 25 Jan 2022 16:21:07 -0800

> Right now the PR presumes this behavior is out-of-scope. Can you
> think of a reason why this behavior needs to be fleshed out in the
> core PPM spec?

Yes, because it goes to the responsibilities of the various players
in the system. For instance, if the intermediary batches, it may
be necessary for the other parts of the system to know the batch
size in order to reliably avoid prematurely aggregating time
windows.

> By "anti-fraud" stuff I'm assuming you're referring to "Sybil attacks"
> as described in [1]. Please correct me if I'm wrong.

Well, it's sort of unclear what "Sybil attacks" means in the context
of unauthenticated clients, but consider the case where you have
an attacker who runs a bot farm and is interested in changing the
output of your computation.

> Right now PPM does nothing about Sybil attacks, except that it
> provides an extension point that a deployment could use for
> mitigating these. In this regard, I don't think the situation has
> changed much.

> Before #174, the leader controlled the set of reports that are
> aggregated, so it could mount a Sybil attack by simply stuffing the
> batch with reports it made up.
> After #174, there may be an intermediary taht controls the set of
> reports that are aggregated, so it could mount a Sybil attack in
> exactly the same way.

This is my point about the threat model. There are (at least) two
threats:

1. Privacy threats where an attacker is trying to determine
   individual report values.
2. Integrity threats where an attacker is trying to affect
   the result of the measurement.

It's true that we need some mechanism to prevent Sybil attacks of the
former category that doesnt depend on trusting the leader, but the
second category of attack does not have this problem because the
leader can already affect the result of the computation, and so in the
current model, the leader can already take care of this, e.g., by IP
reputation, but but we have to ask if the intermediary will do so and
so now it needs to be trusted to produce a correct computation.
And of course there are also attacks where the intermediary
*removes* shares, which can't be protected by PrivacyPass and hte like.

> Either way, an honest aggregator needs to do something to prevent
> Sybil attacks. I don't think we've talked about this specifically in
> this WG yet, but a good option might be the "Rate-Limited Tokens" spec
> being worked on in the Privacy Pass WG [2]. A goal of ours should be
> to allow the aggregators to deploy something like [2] *even when
> running with an intermediary*. I believe this is possible: The clients
> still need to touch the aggregator to get its HPKE public key, so it
> should be possible for an aggregator to issue a token at that time.
>
> Note that I have yet to take a close look at [2], so I might be
> completely off base here :)

I don't think that addresses my concern, for the reasons above.

> aggregated. As discussed above, the leader currently has the same
> power, so this is already a problem. In fact, in the absence of a
> defense, any aggregator can mount a Sybil attack by simply uploading
> report shares to its peers. We don't want to have to trust any
> aggregator more than the others, so a defense will be needed whether
> we take #174 or not.

See above. We already trust the leader to help produce a correct
result, but this design *also* trusts the intermediary to do so,
so now it's yet another trusted part of the system.

> This was discussed in detail in the previous thread, so I'll just
> leave a link to Tim's summary: [3]. The gist is that many of the new
> failure modes that arise are similar to failure modes that already
> exist, and we'll need to deal with these in much the same way. Thus
> dealing with split uploads doesn't appear to require too much
> additional plumbing. If we missed something in that discussion, please
> let us know!

Yeah, I don't think this conclusion is correct. It's correct that
all of these failure modes exist to some extent with the current
model, but scale matters. In particular, with some reasonable
design choices (for instance, having the same retention policy
on each helper, to use Tim's example), you can reduce the number
of inconsistencies to a fairly small number when you can just report
it as error. However, it's quite plausible that in the environment
contemplated by #174 where each client uploads separately, you
see regular inconsistencies at a reasonable fraction (single digit
percentages?) of the number of reports, in which case you can't
just write them off as measurement error but actually need a system
to recover from them. This design doesn't provide such a recoevery
mechanism, AFAICT.

-Ekr

---

From: Christopher Wood <[email-masked]>
Date: Tue, 25 Jan 2022 17:01:28 -0800

> On Jan 25, 2022, at 4:21 PM, Eric Rescorla <[email-masked]> wrote:
> 

I don’t think this needs to be detailed in the spec. At the end of the day, I see no technical reason why the aggregators — leader and helper — should distinguish one entity sending many reports from many entities sending few reports. And if they’re not distinguishing in this manner, reports can be batched and sent however makes sense for a given task configuration. Certainly the batcher needs to know about the batch parameters. So the answer here seems to be: don’t use a batcher unless that batcher knows what it’s doing.

> output of your computation.

I think anti-fraud stuff should remain out of scope for the core spec.

> *removes* shares, which can't be protected by PrivacyPass and hte like.

Hmm, I think this is a deployment specific problem, rather than a problem that the core spec needs to address. In particular, I don’t see a difference between a malicious ingester dropping select reports, and an adversary prohibiting specific clients from uploading shares. 

> so now it's yet another trusted part of the system.

I don’t really see this as a difference. The leader was already doing two functions: ingesting reports and aggregating. This proposal allows these functions to be split up without substantially altering the trust model.

> to recover from them. This design doesn't provide such a recoevery

Well, it _could_ bake in such recovery mechanisms, right? (I agree that more work is needed on the PR, but this doesn’t seem like a dealbreaker to me.)

Best,
Chris

> 
> -Ekr
> 
> 
> 
> 
> -- 
> Ppm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ppm

---

From: Christopher Patton <[email-masked]>
Date: Wed, 26 Jan 2022 08:01:35 -0800

I agree that, in deployments where clients upload report shares to
aggregators directly, data loss is more likely because the upload operation
is no longer atomic. One way we know how to mitigate this is to put an
ingestor in front of the aggregators, which would be functionally
equivalent to the current leader-upload architecture. I think all we need
to do is figure out how to spell this in the PR. In particular, it should
be possible for a deployment to basically work exactly like the
leader-upload architecture.

---

From: Tim Geoghegan <[email-masked]>
Date: Wed, 26 Jan 2022 12:36:28 -0800

I agree with Eric that we owe more text and diagrams illustrating how
deployments might use an ingestor, and to explain whether they affect the
threat model. My feeling/hope is that if we construct secure channels from
clients to the aggregators through any intermediaries, then the ingestor is
no more powerful than any network adversary who can drop messages.
Naturally my feelings aren't good enough for an IETF spec so we owe more
analysis and prescriptions for deployments.

However I'd like to know what we mean by "recovery mechanism". So the
scenario we are concerned about is one where each aggregator holds a
different subset of the reports generated by clients, such that the
intersection of the sets of reports held by leader and helper is (let's
say) only 80% of the generated reports. What does it mean to recover from
this? Does it mean that the system should still successfully aggregate over
the 80%, or that we should somehow augment the upload portion of the
protocol so that clients can retry and we get closer to 100%?

Tim

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jan 2022 05:58:24 -0800

On Wed, Jan 26, 2022 at 12:36 PM Tim Geoghegan <[email-masked]> wrote:

> no more powerful than any network adversary who can drop messages
>

It's probably not, but that's actually quite a bad outcome because that
adversary is extremely powerful for this kind of measurement and surveying
application.

Perhaps we haven't done as great a job defining the threat model as one
would like, but ISTM that for unauthenticated clients, we're going to be
relying for accuracy fairly heavily on topological type stuff (e.g., client
IP address) in order to prevent naive stuffing of the ingestion channel.
But those defenses will not work against the ingestor.

> . Naturally my feelings aren't good enough for an IETF spec so we owe more

Stepping back a level, a measurement protocol in which you only get 80% of
the data based on some unknown network conditions is likely to exhibit bias
in the measurements. So even with a conventional system you'd hope the
clients retry. But with a conventional system, if clients retry the data
eventually trickles in and your statistics correct themselves. But with
PPM, because we want to (1) avoid repeated aggregation of the same values
and (2) want to avoid aggregating small sets, it's a much bigger problem if
you just recompute the metric over the same value. So, that means that we
don't want to end up in a setting where (say) A and B have collectively
received 99% of the reports but actually have each only received 90% and
only have 90% overlap and then that 10% is lost to use forever.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jan 2022 06:15:23 -0800

On Tue, Jan 25, 2022 at 5:01 PM Christopher Wood <[email-masked]>
wrote:

> I don’t think this needs to be detailed in the spec. At the end of the
> day, I see no technical reason why the aggregators — leader and helper —
> should distinguish one entity sending many reports from many entities
> sending few reports. And if they’re not distinguishing in this manner,
> reports can be batched and sent however makes sense for a given task
> configuration. Certainly the batcher needs to know about the batch
> parameters. So the answer here seems to be: don’t use a batcher unless that
> batcher knows what it’s doing.
>

It's not a matter of knowing what you are doing but of coordination.

Suppose (for instance) that the ingestor batches on 10 day boundaries and
the minimum batch size for measurement is 7 days. How does the leader know
on day 16 that it can't aggregate period 7-14?

> Hmm, I think this is a deployment specific problem, rather than a problem
> that the core spec needs to address. In particular, I don’t see a
> difference between a malicious ingester dropping select reports, and an
> adversary prohibiting specific clients from uploading shares.
>

See my response to Tim. In general, this system is not going to provide
correct results if an attacker completely controls communications between
all clients and the servers. However, note that the intermediary actually
is strictly more powerful than such an attacker, because it gets to see the
HTTP traffic not just the TLS traffic. For instance, suppose it uses
User-Agent to drop every submission from a Mac?

> I don’t really see this as a difference. The leader was already doing two
> functions: ingesting reports and aggregating. This proposal allows these
> functions to be split up without substantially altering the trust model.
>

But this is precisely my point about the threat model: that the ingestor is
not some kind of untrusted service, it is an integral part of the system,
and has to be trusted, at least for the purpose of producing accurate
results. So, now we need an analysis of what the intermediary can or cannot
do.

> Well, it _could_ bake in such recovery mechanisms, right? (I agree that
> more work is needed on the PR, but this doesn’t seem like a dealbreaker to
> me.)
>

Those recovery mechanisms may well be quite complicated.  Given that the
nominal purpose of this change is to reduce certain kinds of operational
complexity, I'd like to understand what the resulting additional complexity
of recovery is before making it.

-Ekr

---

From: Christopher Wood <[email-masked]>
Date: Thu, 27 Jan 2022 14:00:10 -0800

> On Jan 27, 2022, at 6:15 AM, Eric Rescorla <[email-masked]> wrote:
> 
> 
> 
> On Tue, Jan 25, 2022 at 5:01 PM Christopher Wood <[email-masked]> wrote:
> 
> Suppose (for instance) that the ingestor batches on 10 day boundaries and the minimum batch size for measurement is 7 days. How does the leader know on day 16 that it can't aggregate period 7-14?

(replied below)

> See my response to Tim. In general, this system is not going to provide correct results if an attacker completely controls communications between all clients and the servers. However, note that the intermediary actually is strictly more powerful than such an attacker, because it gets to see the HTTP traffic not just the TLS traffic. For instance, suppose it uses User-Agent to drop every submission from a Mac?

(replied below)

> But this is precisely my point about the threat model: that the ingestor is not some kind of untrusted service, it is an integral part of the system, and has to be trusted, at least for the purpose of producing accurate results. So, now we need an analysis of what the intermediary can or cannot do.

Hmm, I think we may be talking past one another, or at least I may not be writing clearly, or I’m not understanding you. In any case, my claim is that the intermediary is just separating out the functionality that the Leader was already doing. And by your reasoning above, which I agree with, it would need to be trusted just like the Leader was trusted for ingestion. I don’t think the ingestion functionality can be considered an untrusted part of the system, for some of the reasons discussed above (batching considerations, report filtering, etc).

To try and restate my current position more clearly: I view use of an ingester as a deployment specific decision, and split upload should work such that it allows one to deploy an ingester if it’s useful to them.

> Those recovery mechanisms may well be quite complicated.  Given that the nominal purpose of this change is to reduce certain kinds of operational complexity, I'd like to understand what the resulting additional complexity of recovery is before making it.

Well, there’s at least two types of recovery here: recovery if the client fails to upload, and recovery between aggregators if they don’t agree on the shares. We already need to deal with the latter. What I was referring to was the former, which would be a new failure mode. As discussed in the other thread on this topic, the latter may be sufficient to address the former, or it may not be (I think your point about scale is important here). In any case, yeah, I think we can work through these mechanisms, especially the former.

Best,
Chris