Subject: Re: [Trans] [ct-policy] Re: Certificate Transparency Mirrors (experimental)

From: Pierre Phaneuf <[email-masked]>
Date: Fri, 4 Nov 2016 14:28:35 +0000

On Fri, Nov 4, 2016 at 1:42 PM, Florian M. <[email-masked]> wrote:

> if, for whatever reason, a get-entries response is corrupted, the root hash
> will be incorrect. If the user has no access to intermediate STH, there is
> no way to narrow down and track the corrupted get-entries result. The user
> can only start over.

That's not entirely true, there's a strategy you can use to find where
the corruption occurred.

If the root hash of the whole Merkle tree doesn't compute correctly,
you can do a binary search for the corrupted entry, by using the
MerkleTree::RootAtSnapshot method in the C++ implementation (for
example) to compute the root hash of your tree at half the size, and
do a get-sth-consistency between that tree size and the full one. If
the proof applies correctly, then you know the entries below are good,
and you can repeat the operation, until you isolate the tree size that
stops validating with the consistency proof. You could then drop all
the entries from there on and resume from there (simpler), or maybe
just re-fetch a chunk and recompute the Merkle tree (hoping the
corruption was localised).

You can also spot check single entries using inclusion proofs.

---

From: "Florian M." <[email-masked]>
Date: Fri, 4 Nov 2016 06:42:04 -0700 (PDT)

Hi list,

I recently developed a small tool to create offline mirrors of some of the 
logs.

I wrote it when I realized that bootstrapping a monitor to try and validate 
a whole log from scratch can be difficult. The difficulties come from:

   - a slow download procedure (some log get-entries calls can be 
   relatively slow (seconds and sometimes tens of seconds per query)
   - if, for whatever reason, a get-entries response is corrupted, the root 
   hash will be incorrect. If the user has no access to intermediate STH, 
   there is no way to narrow down and track the corrupted get-entries result. 
   The user can only start over.

I also think that Chrom(e|ium) does not currently fully implement the 
auditor role.

On Fri, Oct 28, 2016 at 1:16 AM, Ryan Sleevi wrote:
> Although CT is designed to prevent the damage any one of these
> organizations can do, it relies on a fully functioning ecosystem of
> gossip and accountability. I've repeatedly made clear that we're
> committed to moving towards that system, but I don't think it would be
> wise to create a false sense of urgency and suggest it be relaxed. I
> say this because relaxing, prior to that robustness, would
> particularly benefit organizations who may not be able to ensure their
> employees follow proper procedures, or which may not keep up to date
> with changes in the log policy, as that could allow for misissuance to
> happen without detection, or through coercion.
> 
> Again, we're very much committed to the long term and exploring ways
> to relax the policy, but at present, Chrome feels that while it's
> reasonable to (for the short-term), trust Google to be honest, it's
> not reasonable to trust all logs to do so. Which is why the policy
> exists :)

As such, I think that Google initiative to set up mirrors is a valid 
temporary approach, which helps mitigating split view attacks.

My tool assists the creation of offline mirrors, by:

   - downloading the log entries;
   - building partial tree hashes; 
   - generating Bittorrent metainfo files to allow these mirrors to be 
   downloaded quickly;
   - providing ways to verify partial downloads.

You may find some documentation and the torrents I generated at the 
following website:

https://www.x-cli.eu/ct

New torrents will be published every week or so, with the latest entries.

The source code of my tool is available at:

<goog_2082899890>
https://github.com/X-Cli/ATBTCT

Cheers,
Florian

---

From: Florian MAURY <[email-masked]>
Date: Sat, 5 Nov 2016 09:15:01 +0100

Hi lists,

I decided to discuss this in private with Pierre, so that we would not
spam the list. Here is a summary of our conversation. Pierre, please
feel free to comment that report, in case I missed something or if the
report is inaccurate.

Pierre statement in the previous email is accurate. One could request
a consistency proof between a subtree and the latest tree head to
check if this subtree contains corrupted data or not. This is an
efficient way to detect which entry or set of entries are invalid.
Current Google implementations (and some others) are willing to return
such proofs. It is however noteworthy that this behavior is not
required by RFC6962 and RFC69626-bis current draft. These documents
only require that logs be able to return consistency proofs between
STHs. This requirement is probably to allow precomputation to limit
abusive behavior impacts on log load. Since all subtree roots are not
necessarily STHs, the consistency checks suggested by Pierre are nice
to have but cannot be expected from all logs.

We both agreed that it would be great to have some way to find out
previous STH. I would suggest adding a tree_size optional parameter to
the v2 get-sth endpoint. If present, the returned STH would be the STH
with the greatest tree_size inferior to the tree_size parameter. I
have not given it much thoughts and this might not be a good idea.
Anyway, this is probably not the right place to discuss this.

Pierre's suggestion to use inclusion proofs to verify that a specific
entry is consistent is an excellent idea. However, if you don't know
for sure which entry or set of entries are corrupted, then we agreed
that this check method is less efficient than downloading everything
all over again.

Finally, Pierre suggested that computing the root hash from the
downloaded entries is not sufficient to ensure an entry consistency.
Indeed, downloaded data include an extra_data field that should be
checked too, by verifying the signature chain up to a trust anchor.
This is an excellent idea that I added on my TODO list. Thanks!

Cheers,
Florian

---

From: Paul Hadfield <[email-masked]>
Date: Mon, 7 Nov 2016 10:54:25 +0000

> Anyway, this is probably not the right place to discuss this.

Hi Florian,

there's been some recent discussion on [trans] about adding a new endpoint
for v2 that permits a client to retrieve historic STHs.  It sounds like
what you
are interested in is quite similar.

Rob Stradling has taken the proposal for this and created a pull request on
the RFC repo:
https://github.com/google/certificate-transparency-rfcs/pull/200

Perhaps you could comment there?

regards,
Paul

> --
> You received this message because you are subscribed to the Google Groups
> "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email-masked].
> To post to this group, send email to [email-masked].
> To view this discussion on the web visit https://groups.google.com/a/
> chromium.org/d/msgid/ct-policy/CAKMP%2BkdESqbM-DQ-9egZQzM7kgbzV%
> 3DaffU6uwYRHizmTJC_%3Dhg%40mail.gmail.com.
>

---

From: Rob Stradling <[email-masked]>
Date: Mon, 7 Nov 2016 11:15:46 +0000

On 07/11/16 10:54, Paul Hadfield wrote:
<snip>
> what you are interested in is quite similar.
> 

A few days ago Melinda wrote:
  "6962-bis has completed working group last call.  Minor editorial
   changes are fine, but let's try to avoid major changes that would
   require yet another WGLC."

I don't yet know if the Chairs consider a new endpoint to be a "major
change".

-- 
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online

---

From: Eran Messeri <[email-masked]>
Date: Wed, 16 Nov 2016 10:54:57 +0900

Regarding the ability to get a consistency proof between any two arbitrary
tree sizes, I agree it's useful for this scenario. Note it is not essential
for correct operation of the protocol - for correct operation of the
protocol only consistency proofs between STHs issued by the log are needed
(so, for simplicity of log implementation, the standard does not require
logs to provide consistency proofs between arbitrary tree sizes).

Regarding Rob Stradling's suggested get-sths, the consensus at the trans WG
meeting yesterday seemed to be that this belongs in the monitoring API
document rather than 6962-bis.

On Mon, Nov 7, 2016 at 8:15 PM, Rob Stradling <[email-masked]>
wrote:

> chromium.org/d/msgid/ct-policy/75df4d95-c923-00bd-
> 4eee-0027d7dbfd8b%40comodo.com.
>