Subject: Re: [Trans] Directory instead of .well-known for URL structure

From: Andrew Ayer <[email-masked]>
Date: Mon, 1 Jul 2019 12:37:01 -0700

I also think .well-known URLs are a bad idea, for the same reasons
as Jacob.

I think directories are even worse.  Although they've been used
successfully in ACME, CT has different needs which make directories
unsuitable.  ACME's usage pattern is to make a series of related
requests over a short time period to drive a certificate request to
issuance, with the client maintaining state between each request.
This makes it natural for a client to fetch the directory once at the
beginning of an issuance "session" and use it for the duration, since
the cost of fetching the directory is amortized over all the requests,
and the directory is unlikely to become invalid during the session.

However, CT's usage pattern is to make a lot of unrelated one-off
requests spread over a long period of time - e.g. submitting a
(pre)certificate, fetching an inclusion proof, fetching the latest STH
to send to auditors. Clients would have to either fetch a new directory
for every request (doubling the number of requests made to the log) or
cache directories in long-term state (which requires dealing with
cached directories going stale, and requires keeping long-term state
which might not otherwise be necessary).

ACME's use of directories is underspecified since it doesn't say how
long a directory remains valid.  It's not a big deal for ACME because
ACME servers are presumed to be sane, or people would switch to another
CA. However, CT is meant to be an adversarial protocol and has to
anticipate logs doing crazy things like constantly changing their
directory in an effort to stymie auditing and hide misbehavior.

Thus, CT's use of directories would need to be quite well specified.
It is not a change that should be rushed at the last minute without a
chance for people to carefully examine and poke holes in it.

CT clients already need to be configured with a number of parameters
for each log - MMD, hash algorithm, public key, log ID, and so on.
Adding a directory would bifurcate log metadata between the existing
parameter set and the new directory object.

I propose satisfying BCP 190 by simply specifying the URL for each
endpoint as a separate log parameter.  This is a very minimal change to
the protocol and avoids the problems above.

Regards,
Andrew

---

From: Tomas Gustavsson <[email-masked]>
Date: Tue, 2 Jul 2019 15:17:54 +0200

+1 for not liking .well-known URIs. From an implementor, .well-known
URIs have been hard, probably badly designed (i.e. RFC 7030), simply
making multi-tenancy, or multi-configuration hard by enforcing a single
end-point for each FQDN. We're actually bypassing .well-known, making it
possible to use normal URLs instead in some cases (in addition to the
specified and limited .well-known of course)

Agree with Andrew, with CT where you need to get and configure
parameters from the log anyhow, and getting the URL as part of that is
the most flexible way. This allows the server operator/implementor full
freedom in their architecture, load balancing, etc.

Regards,
Tomas

On 2019-07-01 21:37, Andrew Ayer wrote:
> _______________________________________________
> Trans mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/trans
>

---

From: Rob Stradling <[email-masked]>
Date: Tue, 2 Jul 2019 15:07:57 +0000

Andrew, thanks for your analysis.  Comments inline...

On 01/07/2019 20:37, Andrew Ayer wrote:
> CA.

It's not been a big deal for ACME...yet.  But why leave it to chance?

I just filed this erratum against RFC8555:
https://www.rfc-editor.org/errata/eid5771

> However, CT is meant to be an adversarial protocol and has to

Agreed.

> the protocol and avoids the problems above.

I agree that it would make sense to require each endpoint to be 
specified as a log parameter.  Your point about bifurcation is well made.

So then I think the question is: Should 6962-bis specify a "directory" 
endpoint that returns a JSON structure that contains all of the log's 
parameters (including each of the endpoint URLs)?  Or should 6962-bis 
not mandate any particular format by which log operators should 
distribute this information?

Currently, 6962-bis doesn't mandate a format, but it does point to an 
example format:
   "[JSON.Metadata] is an example of a metadata format which includes the
    above elements."

Sadly, this claim is not currently true, because [JSON.Metadata] 
(https://www.gstatic.com/ct/log_list/log_list_schema.json) does not 
currently "include the above elements".  We should fix this somehow. 
But how?

Also, how often is a CT client expected to refresh log metadata it has 
obtained from (for example) [JSON.Metadata] ?
Is it wise to underspecify this?

-- 
Rob Stradling
Senior Research & Development Scientist
Sectigo Limited

---

From: Rob Percival <[email-masked]>
Date: Tue, 2 Jul 2019 18:51:53 +0100

FYI: That JSON schema is planned for deprecation; a new version was
proposed last year and, following a period for feedback, parties have been
slowly migrating to the new schema:
https://groups.google.com/a/chromium.org/d/topic/ct-policy/26dmpWdaSHM/discussion

That thread would be a good place to propose schema changes.

---

From: Rob Stradling <[email-masked]>
Date: Wed, 3 Jul 2019 13:52:01 +0000

Thanks Rob.  Both the to-be-deprecated and new schemas are of course 
focused on CT v1 (RFC6962), so ISTM that neither of them are suitable 
candidates for 6962-bis to point at.

Maybe we should simply remove the "[JSON.Metadata] is an example of a 
metadata format which includes the above elements" sentence from 6962-bis.

On 02/07/2019 18:51, Rob Percival wrote:
> proposed last year and, following a period for feedback, parties have 
> been slowly migrating to the new schema: 
> On Tue, 2 Jul 2019, 16:08 Rob Stradling, <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     [email-masked] <mailto:[email-masked]>

Email: [email-masked]
Bradford, UK
Office: +441274024707
Sectigo Limited

This message and any files associated with it may contain legally 
privileged, confidential, or proprietary information. If you are not the 
intended recipient, you are not permitted to use, copy, or forward it, 
in whole or in part without the express consent of the sender. Please 
notify the sender by reply email, disregard the foregoing messages, and 
delete it immediately.

---

From: Eran Messeri <[email-masked]>
Date: Wed, 3 Jul 2019 17:04:04 +0100

I support Andrew Ayer and Rob Stradling's suggestion to " require each
endpoint to be specified as a log parameter."

The reasons outlined make a lot of sense from an implementation
perspective, our experience with RFC6962 suggest that absent caching
mechanisms, logs would face significant load if all clients were to audit
them. Adding a requirement to fetch a directory would increase the load for
(as far as I can tell) no good reason.

I think under-specifying it right now is the only option as we have no
specification of the log metadata. I don't think it's too big of a deal as
when people start implementing 6962-bis I expect the log metadata format
will evolve based on the existing schemas and, if necessary, could be
standardized.

---

From: Andrew Ayer <[email-masked]>
Date: Wed, 3 Jul 2019 09:29:38 -0700

On Wed, 3 Jul 2019 17:04:04 +0100
Eran Messeri <eranm=[email-masked]> wrote:

> specification of the log metadata. I don't think it's too big of a
> deal as when people start implementing 6962-bis I expect the log
> metadata format will evolve based on the existing schemas and, if
> necessary, could be standardized.

Agreed.

Regards,
Andrew

---

From: Rob Stradling <[email-masked]>
Date: Thu, 4 Jul 2019 14:53:34 +0000

SGTM.  How does this look?

https://github.com/google/certificate-transparency-rfcs/pull/311

---

From: Eran Messeri <[email-masked]>
Date: Thu, 4 Jul 2019 15:58:56 +0100

Looks good to me.

---

From: Rob Stradling <[email-masked]>
Date: Fri, 5 Jul 2019 11:44:38 +0000

James Manager commented on this PR [1]:

"The log parameters are not URLs, but URL templates.
The variables that can appear in the templates need to be defined as 
well. That is, 'first', 'second', 'hash', 'start, and 'end' for various 
templates.
Otherwise the spec is still forcing URL structure on servers (ie 
variables MUST be querystring fields with these given names)."

How do folks feel about this?

[1] 
https://github.com/google/certificate-transparency-rfcs/pull/311#pullrequestreview-258184865

On 04/07/2019 15:58, Eran Messeri wrote:
> Looks good to me.
> 
> On Thu, Jul 4, 2019 at 3:53 PM Rob Stradling <[email-masked] 
>      > Eran Messeri <eranm=[email-masked]
>     <mailto:[email-masked]>> wrote:
>      >

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Mon, 8 Jul 2019 15:34:33 -0700

> https://github.com/google/certificate-transparency-rfcs/pull/311#pullrequestreview-258184865

This is a good point. At this point we've examined a good number of 
alternatives (.well-known, directory, and log parameters), and found 
that they all introduce significant complexity and implementation problems.

On the other hand, RFC 6960, which specifies paths and parameters under 
a common URL root, has been working terrifically in production for many 
years. No implementer has complained about any sort of difficulty in 
implementing that particular path structure.

I think we should appeal to "rough consensus and running code," and say 
that BCP 190 does not add value here. Worse, it threatens to tie us into 
pretzels trying to work around it.

---

From: Andrew Ayer <[email-masked]>
Date: Mon, 8 Jul 2019 17:57:45 -0700

On Fri, 5 Jul 2019 11:44:38 +0000
Rob Stradling <[email-masked]> wrote:

> well. That is, 'first', 'second', 'hash', 'start, and 'end' for
> various templates.

I'm opposed to using URL templates, as it would increase client
complexity by forcing them to include template interpolation logic.

This WG has already reached consensus that BCP 190 compliance is
unnecessary for CT, as years of real-world experience with RFC6962
deployment has showed that no log operator needs the additional
flexibility that BCP 190 aims to provide.  I think we should go further
and say that BCP 190 compliance is detrimental to CT, as we've
explored several options to add compliance and they all make CT worse -
by *reducing* flexibility for log operators in a way that actually
causes them problems (well-known), increasing the load on logs
(directories), or by adding complexity to clients (URL templates).  I
doubt that the intent of BCP 190 was to make protocols worse.  I hope
we can move forward without BCP 190 compliance in CT.

Regards,
Andrew

---

From: Ryan Sleevi <[email-masked]>
Date: Tue, 9 Jul 2019 01:21:11 -0400

> we can move forward without BCP 190 compliance in CT.

As Andrew mentioned, many of the suggestions seem to concretely make the
ecosystem less reliable, by making it easier for Log Operators to exploit
the non-cryptographically-verifiable portions of the protocol. A
significant motivating factor in the development of Chrome's CT Policy, for
example, was to attempt to profile down the extant RFC 6962 flexibility,
such as with HTTP error codes and the ability to request payments or
authorization for "public" data, in order to ensure that the Logs it used
were themselves useful to clients. These, in turn, are themselves arguably
security considerations that impact the overall utility of CT.

I agree with the discussion about the reasons against .well-known and
directories; they highlight real tradeoffs that cause more harm than good.
.well-known should be tightly controlled and restricted; and thus, as a
practical matter, makes it difficult to impossible to host CT Logs on a
number of domains or server infrastructures. Similarly, as noted,
directories pass on costs not only to clients, but to the monitoring
ecosystem and to those that wish to ensure Logs adhere to a stated set of
operational policies and practices.

As it relates to URL templates, I'm less opposed, if only because I expect
that a natural consequence of this would be to use policy to profile the
URL templates into a predictable form in order to provide the necessary
security and reliability. As a consequence, I don't anticipate clients ever
needing to implement these, except those clients which might interact with
CT Logs that are limited in scope or application, and not widely used.
Functionally, this means that the flexibility intended by the use of URL
templates is unlikely to be realized in practice in running code. As a
consequence, it's a joint of flexibility that will quickly rust shut,
because keeping it well-oiled is counterintuitively hostile to the
ecosystem.

Hopefully that provides a compelling argument against adopting some of said
flexibility. However, if there is consensus to do so, despite the various
arguments against, I would prefer the URL templates so that they can be
most easily profiled away with limited impact to clients and with the least
'industry' profiling of the applicable standard.