Subject: [weirds] rdap-query draft

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 21 May 2013 10:53:26 -0700

Colleagues,

Next up on our milestones list is the rdap-query draft, which is due to the
IESG next month.  I believe what's posted is current with our most recent
discussions, but I'd like to get some more reviews and commentary on the
record, especially from implementers.

We also don't appear to have any kind of consensus about search.  There are
two documents proposed, but also some claims (one based on a registrar
survey) that we don't really need it for a first version of WEIRDS.  My
thinking is that if there are going to be search capabilities in the query
protocol, they would at least need to be referenced from this document.

-MSK

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 21 May 2013 18:02:53 +0000

Murray, I read John Levine's search inventory message a little differently. He noted that a small number of registry operators (who happen to be administer multiple large TLDs) provide very limited search functionality. That tends to support my suggestion to document basic search functions in a "core" search document and defer the more complicated functions to a second document.

Scott

From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Tuesday, May 21, 2013 1:53 PM
To: [email-masked]
Subject: [weirds] rdap-query draft

Colleagues,

Next up on our milestones list is the rdap-query draft, which is due to the IESG next month.  I believe what's posted is current with our most recent discussions, but I'd like to get some more reviews and commentary on the record, especially from implementers.
We also don't appear to have any kind of consensus about search.  There are two documents proposed, but also some claims (one based on a registrar survey) that we don't really need it for a first version of WEIRDS.  My thinking is that if there are going to be search capabilities in the query protocol, they would at least need to be referenced from this document.
-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 21 May 2013 11:09:40 -0700

On Tue, May 21, 2013 at 11:02 AM, Hollenbeck, Scott <
[email-masked]> wrote:

>  Murray, I read John Levine’s search inventory message a little
> differently. He noted that a small number of registry operators (who happen
> to be administer multiple large TLDs) provide very limited search
> functionality. That tends to support my suggestion to document basic search
> functions in a “core” search document and defer the more complicated
> functions to a second document.****
>
> **
>

Hi Scott,

I'm fine with any interpretation that achieves consensus.  This thread is
my way of starting to figure out where that might fall.

An admittedly cursory review of the rdap-query document suggests that the
search features described there don't exactly match the ones found in the
survey.  Do you think implementers would rather move toward what you've
described in rdap-query, or should rdap-query move in the direction of what
the survey found?

Also, I believe Section 3 of the draft is begging for an ABNF description
of the syntax, but that's secondary to the main question above.

-MSK

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 22 May 2013 11:00:13 +0000

> Do you think implementers would rather move toward what you've described in rdap-query, or should rdap-query move in the direction of what the survey found?

rdap-search should move in the direction of what the survey found.

Scott

From: Murray S. Kucherawy [mailto:[email-masked]]
Sent: Tuesday, May 21, 2013 2:10 PM
To: Hollenbeck, Scott
Cc: [email-masked]
Subject: Re: [weirds] rdap-query draft

On Tue, May 21, 2013 at 11:02 AM, Hollenbeck, Scott <[email-masked]<mailto:[email-masked]>> wrote:
Murray, I read John Levine's search inventory message a little differently. He noted that a small number of registry operators (who happen to be administer multiple large TLDs) provide very limited search functionality. That tends to support my suggestion to document basic search functions in a "core" search document and defer the more complicated functions to a second document.

Hi Scott,

I'm fine with any interpretation that achieves consensus.  This thread is my way of starting to figure out where that might fall.
An admittedly cursory review of the rdap-query document suggests that the search features described there don't exactly match the ones found in the survey.  Do you think implementers would rather move toward what you've described in rdap-query, or should rdap-query move in the direction of what the survey found?
Also, I believe Section 3 of the draft is begging for an ABNF description of the syntax, but that's secondary to the main question above.

-MSK

---

From: "John Levine" <[email-masked]>
Date: 22 May 2013 14:33:04 -0000

>rdap-search should move in the direction of what the survey found.

We found that the usual gTLD suspects provide a simple prefix matching
search on various fields, including domain names.  The only ccTLDs
that provide any search I could find were ones that happened to use
one of the gTLD back ends.

I didn't see any more complex search, which isn't surprising since
prefix matching is easy to implement using typical database backends,
and anything else is vastly harder and/or slower.  While I wouldn't be
opposed on principle to fancy searching with regular expressions or
the like, I doubt anyone would ever provide it other than at toy
scale.

So my suggestion would be to spec a prefix match on the any of the
fields that TLDs currently allow to search, and return values of
here's all the matches, here's some of the matches but you exceeded a
match limit, nothing matched, and we don't have search on that field.
I don't see much use for a query about what searches are allowed; just
try it and see if it works.

---

From: Hugo Salgado <[email-masked]>
Date: Wed, 22 May 2013 10:44:48 -0400

> the like, I doubt anyone would ever provide it other than at toy

Hi John.
We at .CL provide search capabilities in prefix, suffix, and middle
of domain names. These capabilities are only in the web interface.
The whois/43 has only exact match.

We're aware that this capability is not popular outside .CL, but is
a feature we officially support and will be maintained in a future
restful web interface.

Regards,

Hugo

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 22 May 2013 09:09:18 -0700

> one of the gTLD back ends.

Just to be clear, by "prefix matching", you mean something like a search
for "foo*"?

(In my head, the "matching" happens when the wildcard is resolved, so you
might mean "*foo".)

-MSK

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2013 12:20:44 -0400

> Just to be clear, by "prefix matching", you mean something like a search

Yes, you give it a string, it finds the items that start with that string.

For databases using typical B-tree indices, this is one or two disk 
fetches, no scanning needed.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Andy Newton <[email-masked]>
Date: Thu, 23 May 2013 16:38:00 +0000

From: "Murray S. Kucherawy" <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, May 21, 2013 1:53 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [weirds] rdap-query draft

Murray,

Honing in on the last sentence, I do not think that is the case nor is it desirable. We have an extensible model and can add other URI paths in the future for search capabilities. Additionally, since lookups are 99% of Whois queries, I don't think it is necessary to hold up our current work for what amounts to a small subset of queries with requirements that are not well understood.

Forgive me if I've misunderstood your point though.

-andy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 23 May 2013 16:16:50 -0700

On Thu, May 23, 2013 at 9:38 AM, Andy Newton <[email-masked]> wrote:

>   From: "Murray S. Kucherawy" <[email-masked]>
> To: "[email-masked]" <[email-masked]>
>  We also don't appear to have any kind of consensus about search.  There
> are two documents proposed, but also some claims (one based on a registrar
>  Honing in on the last sentence, I do not think that is the case nor is
> it desirable. We have an extensible model and can add other URI paths in
> the future for search capabilities. Additionally, since lookups are 99% of
> Whois queries, I don't think it is necessary to hold up our current work
> for what amounts to a small subset of queries with requirements that are
> not well understood.
>

Hi Andy,

The idea I have is that rdap-query and json-response together would ideally
provide at least all the current capabilities of port 43 WHOIS, and if we
agree that rudimentary search is part of what we're replacing then it
should at least be referenced there.  But if we'd rather do two drafts that
are pure query-response, then your way is fine.

I'm happy to go with consensus on this one, as long as I know what it is.
:-)

-MSK, participatin'

---

From: Ernie Dainow <[email-masked]>
Date: Mon, 27 May 2013 13:19:05 -0400

On 5/23/2013 7:16 PM, Murray S. Kucherawy wrote:
> On Thu, May 23, 2013 at 9:38 AM, Andy Newton <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     To: "[email-masked] <mailto:[email-masked]>" <[email-masked]
>     <mailto:[email-masked]>>
>         We also don't appear to have any kind of consensus about
>         search.  There are two documents proposed, but also some
>         claims (one based on a registrar survey) that we don't really
>         need it for a first version of WEIRDS.  My thinking is that if
>         there are going to be search capabilities in the query
>     Honing in on the last sentence, I do not think that is the case
>     nor is it desirable. We have an extensible model and can add other
>     URI paths in the future for search capabilities. Additionally,
>     since lookups are 99% of Whois queries, I don't think it is
>     necessary to hold up our current work for what amounts to a small
>     subset of queries with requirements that are not well understood.
>
> The idea I have is that rdap-query and json-response together would 
> ideally provide at least all the current capabilities of port 43 
> WHOIS, and if we agree that rudimentary search is part of what we're 
> replacing then it should at least be referenced there.  But if we'd 
> rather do two drafts that are pure query-response, then your way is fine.
>
> I'm happy to go with consensus on this one, as long as I know what it 
> is.  :-)
>
> -MSK, participatin'
>

There are Whois search requirements in Sec 1.10 of ICANN's New gTLD 
Agreement that are beyond simple prefix matching. Since many operators 
will be launching new TLDs starting late 2013 or early 2014, the RDAP 
spec should address this requirement on first publication (current 
charter target Dec 2013).

Prefix matching search that is compatible with most legacy Whois 
services should be rdapConformance level 0 with the new gTLD search at 
level 1. Since both search requirements need to be published around the 
same time, it would be simplest to specify them both in one document 
(either search or query).

-Ernie

---

From: "John Levine" <[email-masked]>
Date: 27 May 2013 19:35:43 -0000

>Agreement that are beyond simple prefix matching.

There is no section 1.10 of the new gTLD agreement.  I expect you're
referring to Appendix A, Specification 4, section 1.10, which
describes WHOIS searching.

What I see there is basically prefix matching with and/or/not
combinations of matches on various fields.  Many of the applications
with Neustar as the back end say they're going to provide regular
expression matching, but I have to say that I don't understand how
they expect to do searches on large domains, since I am fairly sure
that the only way to implement general REs is to scan the entire data
set, with no way to speed them up using indexes like you can with
prefix matching.  (Is anyone from Neustar here who can enlighten us?)

Given the uncertainty about what new gTLDs will really implement, your
suggestion about prefix matching now, gTLD whatever later is probably
the best we can do.

---

From: Andy Newton <[email-masked]>
Date: Mon, 27 May 2013 22:50:38 +0000

>describes WHOIS searching.

It is my understanding that this is optional and a nice to have but not a
requirement.

Also, we must remember that ICANN gTLDs are not the only community being
served by this spec. I would rather not hold up the work of RDAP for the
RIR community on search. The RIR Whois services also have search
capabilities but the RIRs have purposefully set aside those needs for the
time being so that the IETF can publish a base spec that meets the needs
of 99% of Whois queries. Search is a larger topic than just the needs of
the ICANN gTLD requirements.

-andy

---

From: Don Blumenthal <[email-masked]>
Date: Mon, 27 May 2013 20:04:34 -0400

You are correct about search capabilities being optional.

Keep in mind also that current and pending documents relate to the
existing system. The agreements' WHOIS-related language will have to be
redrafted should RDAP become the law of ICANNland. The draft Registry
Agreements alludes to that fact at the beginning of Specification 4.1 on
Registration Data Directory Services: "Until ICANN requires a different
protocol, Registry Operator will operate a WHOIS service available via
port 43 in accordance with RFC 3912, and a web-based Directory Service at
<whois.nic.TLD>Š."

Don

On 5/27/13 6:50 PM, "Andy Newton" <[email-masked]> wrote:

>_______________________________________________
>weirds mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/weirds

---

From: "John R Levine" <[email-masked]>
Date: 27 May 2013 22:35:25 -0400

> It is my understanding that this is optional and a nice to have but not a

You're right, it's optional, but if a new gTLD offers search at all, it 
has to do what's in that section.

> RIR community on search.

Agreed, although this strikes me as one of the areas where the RIRs and 
domains will always be different, so we can just leave a placeholder for 
domain search without affecting RIRs.

Regards,

---

From: "John Levine" <[email-masked]>
Date: 28 May 2013 02:45:38 -0000

>redrafted should RDAP become the law of ICANNland. ...

But the current draft for new TLDs says:

 Registration Data Directory Services. Until ICANN requires a different
 protocol, Registry Operator will operate a WHOIS service available via
 port 43 in accordance with RFC 3912, and a web-based Directory Service
 at <whois.nic.TLD> providing free public query-based access to at
 least the following elements in the following format. ICANN reserves
 the right to specify alternative formats and protocols, and upon such
 specification, the Registry Operator will implement such alternative
 specification as soon as reasonably practicable.

 Registry Operator shall implement a new standard supporting access to
 domain name registration data (SAC 051) no later than 135 days after
 it is requested by ICANN if: 1) the IETF produces a standard (i.e., it
 is published, at least, as a Proposed Standard RFC as specified in RFC
 2026); and 2) its implementation is commercially reasonable in the
 context of the overall operation of the registry.

 [ followed by a description of the current thick whois ]

---

From: Don Blumenthal <[email-masked]>
Date: Mon, 27 May 2013 22:55:25 -0400

That's what I quoted from, the 04/01 version. The additional text,
especially the second paragraph, strike me as supporting my point that we
not get too wrapped around the axle on what's currently required.

---

From: Don Blumenthal <[email-masked]>
Date: Mon, 27 May 2013 22:57:26 -0400

Correction. Search isn't required.

On what's in specifications concerning WHOIS.

---

From: Don Blumenthal <[email-masked]>
Date: Mon, 27 May 2013 23:19:39 -0400

Correction deux. The current version is 04-29, not 04-01. For anybody who
REALLY has nothing else to do,
http://www.icann.org/en/news/public-comment/base-agreement-29apr13-en.htm.