Subject: Re: [regext] Fwd: New Version Notification for draft-ietf-regext-rdap-sorting-and-paging-20.txt

From: Alexander Mayrhofer <[email-masked]>
Date: Tue, 1 Dec 2020 13:13:13 +0100

Mario, all,

i'm currently implementing a prototype RDAP server that uses sorting and
paging features. I've gotten so far to implement the basic parameters
(cursor in the limit/offset variant, and sort), and now i'm trying to
implement the "paging_metadata" elements. I have a few comments on that
(you know, some of theses things, you only notice once you start with an
implementation, so - sorry for being late to the game).

Ok, here we go:

   The "paging_metadata" element contains the following fields:

   o  "totalCount": "Numeric" (OPTIONAL) a numeric value representing
      the total number of objects found.  It MUST be provided if and
      only if the query string contains the "count" parameter;

I don't understand why a server is disallowed to include the
totalCount unless the client has requested it. I'd change that to
"MUST be provided if the query string contains the 'count' parameter,
but MAY be provided in other cases", unless there's good reasons for
disallowing it!

   o  "pageSize": "Numeric" (OPTIONAL) a numeric value representing the
      number of objects that should have been returned in the current
      page.  It MUST be provided if and only if the total number of
      objects exceeds the page size.  This property is redundant for
      RDAP clients because the page size can be derived from the length
      of the search results array but, it can be helpful if the end user
      interacts with the server through a web browser;

I find that one useless for the limit/offset case. I understand it
might be useful for the keyset variant, but it took me a long time
understanding that. For the limit/offset case, it can only be exactly
the given "limit" parameter anyway, as it must not be provided when
the total number of results is less than the page size. I won't
implement it in my case.

   o  "pageNumber": "Numeric" (OPTIONAL) a numeric value representing
      the number of the current page in the result set.  It MUST be
      provided if and only if the total number of objects found exceeds
      the page size;

I won't implement that one either, as there's no way calculating the number
of pages in the limit/offset case (since the client can always change the
"limit"). Also, it *would* be possible for the client to calculate that by
itself, *if* you'd allow the server to return the total number of items in
cases where the client didn't request it (see above).

   o  "links": "Link[]" (OPTIONAL) an array of links as described in
      [RFC8288 <https://tools.ietf.org/html/rfc8288>] containing the
reference to the next page.  In this
      specification, only forward pagination is described because it is
      all that is necessary to traverse the result set.

Why is only forward pagination described here? Is that because of the
"keyset" case? And does that mean that we MUST NOT encode a reference
to the previous page in those links? I'd like to see a more normative
language here, in the light of interopability.

I'd recommend to allow inclusion of at least "next" and "previous"
page (and, as i have just discovered, you are actually listing
"first", "last" and "prev" in section 2.4.1 as what a server MAY
implement..

Plus i'd like to see a more specific reference than just whole RFC
8288... It's quite a bit to read - i did the "RFC noob" thing of going
to the examples to understand it..

So far for the paging metadata.... I might find more as i work through the
rest of the specs.

best,
Alex

On Mon, Nov 30, 2020 at 9:45 AM Mario Loffredo <[email-masked]>
wrote:

> Hi all,
>
> Just to make the document more comprehensive, I added a final paragraph to
> Appendix A to reference the comparison between JSONPath operators and
> IETF JSONPath WG web site.
>
> Best,
>
> Mario
>
>
> -------- Messaggio Inoltrato --------
> Oggetto: New Version Notification for
> draft-ietf-regext-rdap-sorting-and-paging-20.txt
> Data: Mon, 30 Nov 2020 00:40:41 -0800
> Mittente: [email-masked]
> A: Maurizio Martinelli <[email-masked]>
> <[email-masked]>, Mario Loffredo
> <[email-masked]> <[email-masked]>, Scott Hollenbeck
> <[email-masked]> <[email-masked]>
>
>
> A new version of I-D, draft-ietf-regext-rdap-sorting-and-paging-20.txt
> has been successfully submitted by Mario Loffredo and posted to the
> IETF repository.
>
> Name: draft-ietf-regext-rdap-sorting-and-paging
> Revision: 20
> Title: Registration Data Access Protocol (RDAP) Query Parameters for
> Result Sorting and Paging
> Document date: 2020-11-30
> Group: regext
> Pages: 31
> URL:
> https://www.ietf.org/archive/id/draft-ietf-regext-rdap-sorting-and-paging-20.txt
> Status:
> https://datatracker.ietf.org/doc/draft-ietf-regext-rdap-sorting-and-paging/
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-ietf-regext-rdap-sorting-and-paging
> Htmlized:
> https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-20
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-regext-rdap-sorting-and-paging-20
>
> Abstract:
> The Registration Data Access Protocol (RDAP) does not include core
> functionality for clients to provide sorting and paging parameters
> for control of large result sets. This omission can lead to
> unpredictable server processing of queries and client processing of
> responses. This unpredictability can be greatly reduced if clients
> can provide servers with their preferences for managing large
> responses. This document describes RDAP query extensions that allow
> clients to specify their preferences for sorting and paging result
> sets.
>
>
>
> Please note that it may take a couple of minutes from the time of
> submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>
>
> _______________________________________________
> regext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/regext
>

---

From: Mario Loffredo <[email-masked]>
Date: Tue, 1 Dec 2020 19:32:41 +0100

Hi Alex,

thanks a lot for your feedback anyway.

Please find my comments inline.

Il 01/12/2020 13:13, Alexander Mayrhofer ha scritto:
> Mario, all,
>
> i'm currently implementing a prototype RDAP server that uses sorting 
> and paging features. I've gotten so far to implement the basic 
> parameters (cursor in the limit/offset variant, and sort), and now i'm 
> trying to implement the "paging_metadata" elements. I have a few 
> comments on that (you know, some of theses things, you only notice 
> once you start with an implementation, so - sorry for being late to 
> the game).
>
> Ok, here we go:
>

[ML] Firstly, I think we must define an unambiguous interoperability 
model between clients and servers so I don't agree about the phrase 
"provided in other cases". Since calculating the total number of results 
is something that affects the processing time, it is better to calculate 
and return it on demand rather than anyway.

Besides being time consuming, providing the totalCount anyway can be 
inefficient as well. For example, it never changes when scrolling a 
result set. In this case, a server willing to provide always the 
totalCount has two implementing options: either calculating it at each 
page (inefficient) or calculating it once at the first page and encode 
it in the cursor value so that it can be passed from one page to another 
(more efficient but adding complexity).

> the total number of results is less than the page size. I won't 

[ML] I don't understand your objection. It seems to me that you are 
explicitly implementing the limit/offset parameters rather than the 
cursor parameter based on an underlying limit/offset pagination. But 
this is not what is defined by this specification.

According to this specification, the client is not aware of what kind of 
pagination method the cursor implementation is based on, neither it can 
set the page size that is, on the contrary, set by the server. In 
addition, the page size is different from server to server. This 
information is redundant for clients (because it can be easily derived 
programmatically) but it is useful especially for a human interacting 
with a server through a web browser.

> I won't implement that one either, as there's no way calculating the 
> number of pages in the limit/offset case (since the client can always 
> change the "limit"). Also, it *would* be possible for the client to 
> calculate that by itself, *if* you'd allow the server to return the 
> total number of items in cases where the client didn't request it (see 
> above).

[ML] Here again, I agee with you that this information is useless if you 
explicitly implement pagination through limit/offset because every 
client might change the limit as it sees fit but this is not defined by 
this specification. In this specification, a client can't change the 
page size. Each RDAP provider implementing this capability decides which 
size is the more sustainable according to a number of factors (e.g. the 
number of potential users who are allowed to submit search queries, the 
number of search requests to process in a given interval, the average 
result set size). I imagine that the page size is something that a 
server should define carefully after a tuning process.

Finally, allowing a client to set a page size freely means, from one 
side, making the server to deal with possible unsustainable result set 
pages, on the other side,  making the server to check the allowed values 
for a page size and, consequently, to return an error.

>        [RFC8288  <https://tools.ietf.org/html/rfc8288>] containing the reference to the next page.  In this

[ML] Forward pagination is the natural way to scroll a result set 
regardless the pagination method. I mean, I have never met someone 
scrolling a result set starting from the end !?!. Especially when 
combined with sorting, it provides the user with a way to find the most 
relevant results at the beginning of the result set and to stop 
scrolling it when all the relevant results have been found. The other 
kinds of links make sense only if provided in addition to the "next" 
link. Anyway, they are not forbidden by this specification (as you have 
seen) but they are simply considered inessential.

Sorry, but I don't understand what should be the specific reference 
you'd like to see. The "rel" values presented in this specification are 
all the values that are commonly (more or less) used for the 
representing pagination links and all of them are defined in RFC8288. 
What other reference should I include?

Cheers,

Mario

>
> So far for the paging metadata.... I might find more as i work through 
> the rest of the specs.
>
> best,
> Alex
>

>
> On Mon, Nov 30, 2020 at 9:45 AM Mario Loffredo 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>     Just to make the document more comprehensive, I added a final
>     paragraph to Appendix A to reference thecomparison between
>     JSONPath operators and IETF JSONPath WG website.
>
>
>     Best,
>
>     Mario
>
>
>
>     Mittente: 	[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked]>, Mario Loffredo
>     <[email-masked]> <mailto:[email-masked]>,
>     Scott Hollenbeck <[email-masked]>
>     <mailto:[email-masked]>
>
>
>
>
>     Title: Registration Data Access Protocol (RDAP) Query Parameters
>     for Result Sorting and Paging
>     <https://www.ietf.org/archive/id/draft-ietf-regext-rdap-sorting-and-paging-20.txt>
>     Status:
>     <https://datatracker.ietf.org/doc/draft-ietf-regext-rdap-sorting-and-paging/>
>     Htmlized:
>     <https://datatracker.ietf.org/doc/html/draft-ietf-regext-rdap-sorting-and-paging>
>     Htmlized:
>     <https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-20>
>     Diff:
>     <https://www.ietf.org/rfcdiff?url2=draft-ietf-regext-rdap-sorting-and-paging-20>
>
>     Abstract:
>     until the htmlized version and diff are available at
>     tools.ietf.org <http://tools.ietf.org>.
>
>     The IETF Secretariat
>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/regext>
>
>

-- 
Dr. Mario Loffredo
Technological Unit “Digital Innovation”
Institute of Informatics and Telematics (IIT)
National Research Council (CNR)
via G. Moruzzi 1, I-56124 PISA, Italy
Phone: +39.0503153497
Web: http://www.iit.cnr.it/mario.loffredo