Subject: Re: [Acme] Paging for authorizations and certificates lists

From: Ted Hardie <[email-masked]>
Date: Tue, 1 Mar 2016 10:21:54 -0800

On Mon, Feb 29, 2016 at 9:30 PM, Jacob Hoffman-Andrews <[email-masked]> wrote:

> I posted this pull request:
> https://github.com/ietf-wg-acme/acme/pull/86/files
>
> For some accounts, the list of authorizations or certificates will be
> too large to reasonably return in a single request. This adds a notion
> of paging: you get an initial list of resources, with a link rel=next
> pointing at a URL for more, if there are more.
>
>
So the current method is to return an array of strings within a JSON
object, where the strings are themselves URIs.  You anticipate that this
array will be large enough to require a separate response?  The overall
object limit is 128KB, right?  That's a lot of URIs.

Ted

> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme
>

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Tue, 1 Mar 2016 10:41:24 -0800

On 03/01/2016 10:21 AM, Ted Hardie wrote:
> object, where the strings are themselves URIs.  You anticipate that
> this array will be large enough to require a separate response?  The
> overall object limit is 128KB, right?  That's a lot of URIs.
Yep. The distribution of authorizations / certificates per account will
be highly skew: I expect some accounts to have millions of each,
especially since the certificates list will grow continuously.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 2 Mar 2016 13:20:33 +1100

I hope, for everyone's sake, that (assuming we accept this PR, which
is a good idea in general) implementations will be setting limits
quite low before triggering paging.  Otherwise, we will have a feature
that is surprising when developers first encounter it.  If you set the
limit to 1000 entries in the list, then you will find that very few
people encounter the problem, but those that do will find their tools
fail them.

You should recommend a limit that is a small number, like 10.  That
might appear often enough that developers get the message.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 2 Mar 2016 14:34:46 +0000

Are there common and well-specified semantics for merging "paged" JSON objects?

Is his necessary?  The serve can just stream it out and any client who is asking for this huge name must be willing to generate it, too.  Or is that wrong?

---

From: Martin Thomson <[email-masked]>
Date: Thu, 3 Mar 2016 04:59:35 +1100

> Are there common and well-specified semantics for merging "paged" JSON objects?

For arrays, yes.  It follows the pattern established with Atom, but
it's fairly widely used.  You put N things in the response, then
provide a rel=next link for the next part.

> Is his necessary?  The serve can just stream it out and any client who is asking for this huge name must be willing to generate it, too.  Or is that wrong?

Consuming a huge blob of JSON is pretty unwieldy.  JSON doesn't have a
SAX-equivalent API, so you are reduced to sucking the whole object up
into memory.  There is RFC 7464, which cuts it back to something
manageable, but it is also yet-another format.

https://tools.ietf.org/html/rfc7464

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 2 Mar 2016 18:03:10 +0000

> For arrays, yes.  It follows the pattern established with Atom, but it's fairly
> widely used.  You put N things in the response, then provide a rel=next link
> for the next part.

Thanks.