Subject: Re: [weirds] I-D Action: draft-ietf-weirds-rdap-query-11.txt

From: Ernie Dainow <[email-masked]>
Date: Fri, 01 Aug 2014 12:09:24 -0400

On 2014-07-31 3:26 PM, John Levine wrote:
>> Tickets not addressed: 8, 9, 10, 11, 29
> Nos. 8, 9, 10 are from Andy.  Andy?
>
> No. 11 is about a flag for just saying whether a thing exists or not.
> I think we agreed that the ordinary http HEAD command would do what we
> wanted, no new mechanism needed.
>
>
I think there is a better way to provide this. In 
draft-ietf-weirds-json-response, it is implied that search results 
return the full object for each name found. This gets very large. Most 
of the time, the client is not interested in all that information. 
Search results should be a simple list of the names that were found. A 
search for exam*.com would return something like [ exam.com, 
examination.com, example.com, ... ]

Clients that want more detailed information can make a specific request 
for the name of interest. So to discover if a name exists or not, simply 
do a search for that name (no * in search) and you get a list of one 
name or a not found error.

-Ernie

---

From: "John R Levine" <[email-masked]>
Date: 1 Aug 2014 12:23:01 -0400

> return the full object for each name found.

Ticket #11 isn't about search.  As I read it, it's just "does foo.blah 
exist" or whatever other thing you're querying about. HEAD would be the 
way to do it, you get the response code with no contents.

Since it's http, the server has to support HEAD anyway so we get it for 
free.

R's,
John

---

From: Andy Newton <[email-masked]>
Date: Tue, 5 Aug 2014 17:52:34 +0000

On Jul 31, 2014, at 3:26 PM, John Levine <[email-masked]<mailto:[email-masked]>> wrote:

#8 and #9 were discussed on this list, and I thought we had agreement to include them.

#10 was a request to remove search by entity handles. However, after discussion on this mailing list we decided to keep it. This one can be closed.

-andy

---

From: Andy Newton <[email-masked]>
Date: Wed, 6 Aug 2014 20:18:53 +0000

On Aug 1, 2014, at 12:09 PM, Ernie Dainow <[email-masked]> wrote:

> I think there is a better way to provide this. In draft-ietf-weirds-json-response, it is implied that search results return the full object for each name found. This gets very large. Most of the time, the client is not interested in all that information. Search results should be a simple list of the names that were found. A search for exam*.com would return something like [ exam.com, examination.com, example.com, ... ]
> 
> Clients that want more detailed information can make a specific request for the name of interest. So to discover if a name exists or not, simply do a search for that name (no * in search) and you get a list of one name or a not found error.

I have not written a client specifically for domain searching, but my client will cache objects (including domains) from searches if they have a “self” reference. Therefore when the user asks specifically for a domain, the answer is pulled from the cache and not directly from the server.

Our experience with ARIN’s Whois-RWS is that client writers wanted all the information up front and that server load was lower by doing that than having the clients get a list of handles, names, or references and then having them requery for the full objects.

However, I can see the point you are making. A registry may want clients to requery for full objects for reasons I cannot imagine at this time. I would favor a flag in each object indicating that the object is incomplete so the clients know they must requery. Actually, an array of flags with an IANA registry would make this kind of thing very extensible in case we don’t exactly nail the semantics the first time.

-andy

---

From: "John R Levine" <[email-masked]>
Date: 6 Aug 2014 17:23:49 -0400

> Our experience with ARIN’s Whois-RWS is that client writers wanted all 
> the information up front and that server load was lower by doing that 
> than having the clients get a list of handles, names, or references and 
> then having them requery for the full objects. ...

I'm still not seeing what's wrong with HEAD.  It gives you the return code 
to say whether the object exists, and web servers all support it anyway.

What am I missing?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Andy Newton <[email-masked]>
Date: Wed, 6 Aug 2014 22:15:21 +0000

On Aug 6, 2014, at 5:23 PM, John R Levine <[email-masked]> wrote:

>> Our experience with ARIN’s Whois-RWS is that client writers wanted all the information up front and that server load was lower by doing that than having the clients get a list of handles, names, or references and then having them requery for the full objects. ...
> 
> I'm still not seeing what's wrong with HEAD.  It gives you the return code to say whether the object exists, and web servers all support it anyway.
> 
> What am I missing?

For a domain lookup, such as /domain/example.com, then HEAD would work.

But I thought Ernie was talking about searches, such as /domains?domain=exam*.com. In the second example a list of domains is returned.

-andy

---

From: "John R Levine" <[email-masked]>
Date: 6 Aug 2014 18:39:38 -0400

> But I thought Ernie was talking about searches, such as /domains?domain=exam*.com. In the second example a list of domains is returned.

Ticket 11 is just for individual lookups.

All of the existing WHOIS servers I know just give you a list of matching 
names or whatever when you do a search, so I don't see why RDAP would do 
any more than that.

R's,
John

---

From: Andy Newton <[email-masked]>
Date: Thu, 7 Aug 2014 13:25:05 +0000

> Ticket 11 is just for individual lookups.

Ok. Rereading #11 I see you are right. There was a whole thread on this list a year ago about the requirements for a domain existence protocol, and the upshot was that it was far more complicated that a simple binary answer. I agree with you that for our purposes HEAD works. If people want a domain existence protocol, then they need to go flesh out the full requirements.

> All of the existing WHOIS servers I know just give you a list of matching names or whatever when you do a search, so I don't see why RDAP would do any more than that.

Well, I listed my reasons. Quite simply, we are in a world where the clients can be smarter and better.

-andy

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Thu, 7 Aug 2014 13:39:43 +0000

> -----Original Message-----
> From: weirds [mailto:[email-masked]] On Behalf Of Andy Newton
> Sent: Thursday, August 07, 2014 9:25 AM
> To: John R Levine
> Cc: [email-masked]
> Subject: Re: [weirds] full objects in search results (was Re: I-D
> Action: draft-ietf-weirds-rdap-query-11.txt
> 
> 
> >> But I thought Ernie was talking about searches, such as
> /domains?domain=exam*.com. In the second example a list of domains is
> returned.
> >
> Ok. Rereading #11 I see you are right. There was a whole thread on this
> list a year ago about the requirements for a domain existence protocol,
> and the upshot was that it was far more complicated that a simple
> binary answer. I agree with you that for our purposes HEAD works. If
> people want a domain existence protocol, then they need to go flesh out
> the full requirements.

This suggests to me that there's text to be added to using-http, perhaps in Section 4.

Scott

---

From: "John R Levine" <[email-masked]>
Date: 7 Aug 2014 09:57:19 -0400

> Well, I listed my reasons. Quite simply, we are in a world where the clients can be smarter and better.

I'm wondering about server load.  For domain names, it is my impression 
that returning a list of 50 names is a lot cheaper than looking up all the 
glop for those 50 names.

Perhaps someone who runs an actual server can say whether it's an issue in 
practice.

Regards,

---

From: Ernie Dainow <[email-masked]>
Date: Thu, 07 Aug 2014 10:07:29 -0400

On 2014-08-06 6:39 PM, John R Levine wrote:
> All of the existing WHOIS servers I know just give you a list of 
> matching names or whatever when you do a search, so I don't see why 
> RDAP would do any more than that.
>

There are several issues here, all of which should be resolved.

The premise in Ticket 11 is about search and the volume of data it 
sends. It proposes a solution of providing an optional parameter to 
return a short list instead of the full object.

Comment #1 proposes another solution, to use HEAD. This presumably would 
make more sense to use with an rdap query as opposed to search.

The current wording in draft-ietf-weirds-json-response-07 implies that 
response to a search is the full object. This is not behavior of 
existing WHOIS servers.

I was proposing that search results be changed to a simple list, 
consistent with WHOIS.

Andy extended this by suggesting flags be added so clients know whether 
the object is complete or not. I think this provides the most 
flexibility by giving servers the option of returning either full 
objects or any partial subset they feel is appropriate without confusing 
the client.

-Ernie

---

From: Andy Newton <[email-masked]>
Date: Thu, 7 Aug 2014 14:22:50 +0000

On Aug 7, 2014, at 9:57 AM, John R Levine <[email-masked]> wrote:

> 
> I'm wondering about server load.  For domain names, it is my impression that returning a list of 50 names is a lot cheaper than looking up all the glop for those 50 names.
> 
> Perhaps someone who runs an actual server can say whether it's an issue in practice.

It all depends on server architecture. But with a traditional RDMS, a dip into the database is a dip into the database and returning 100 bytes is not more expensive than returning 10 bytes. But multiple hits to the database for the same data is more expensive than returning 100 bytes just once. And when you can’t guarantee that the client doing a requery will hit the same server, things like caches don’t help you.

-andy

---

From: "John R Levine" <[email-masked]>
Date: 7 Aug 2014 10:41:24 -0400

> It all depends on server architecture. But with a traditional RDMS, a 
> dip into the database is a dip into the database and returning 100 bytes 
> is not more expensive than returning 10 bytes. ...

Right, but given the way that domain names are used, I'm wondering whether 
they'd have a separate faster database just to check existence.  Domain 
snipers hammer on registries trying to snap up a domain the millisecond 
it's freed, so it's a problem they already have.

Again, reports from actual domain registries would be helpful.

Regards,

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Thu, 7 Aug 2014 14:52:19 +0000

> From: weirds [mailto:[email-masked]] On Behalf Of John R
> Levine
> Sent: Thursday, August 07, 2014 10:41 AM
> To: Andy Newton
> Cc: [email-masked]
> >> I'm wondering about server load.  For domain names, it is my
> impression that returning a list of 50 names is a lot cheaper than
> looking up all the glop for those 50 names.
> >>

In my experience registrars don't generally use EPP check commands when trying to snap up available names. They use repeated add commands.

Scott

---

From: "John R Levine" <[email-masked]>
Date: 7 Aug 2014 11:12:02 -0400

> In my experience registrars don't generally use EPP check commands when trying to snap up available names. They use repeated add commands.

Sure, but the add command has to check whether the domain exists.

In any event, got any insight into whether it's worth having an RDAP 
option that just returns names from a search, as opposed to returning 
everything about the names?

Regards,

---

From: Frederico A C Neves <[email-masked]>
Date: Thu, 7 Aug 2014 15:12:09 -0300

> objects or any partial subset they feel is appropriate without confusing 

I agree with this on the grounds of flexibility, operational cost and
the history of the previous protocol.

I would like to put another possible issue that I see as orthogonal to
the search reasult format. It is the situation a server encounter with
a too broad search term. It will simply deny the query signaling
though the transport, or it will return partial data, limited to their
local policy, signaling somehow through the response payload? Do we
need a standard way for better client experience?

> -Ernie

Fred

---

From: Andy Newton <[email-masked]>
Date: Thu, 7 Aug 2014 18:24:45 +0000

On Aug 7, 2014, at 2:12 PM, Frederico A C Neves <[email-masked]<mailto:[email-masked]>> wrote:

Thanks.

Do you mean something other than what is described in section 10 of json-response:
http://tools.ietf.org/html/draft-ietf-weirds-json-response-07#section-10

-andy

---

From: Frederico A C Neves <[email-masked]>
Date: Fri, 8 Aug 2014 09:54:31 -0300

On Thu, Aug 07, 2014 at 06:24:45PM +0000, Andy Newton wrote:
> 
> > On Aug 7, 2014, at 2:12 PM, Frederico A C Neves <[email-masked]> wrote:
...
> http://tools.ietf.org/html/draft-ietf-weirds-json-response-07#section-10

Actually not. Sorry for not paying attention to this (json-response
#10). With that said, this is pretty much a complete simple search
solution and I would like to encourage the wg to take this as the way
forward as it address the most common operational search scenarios
servers currently provide.

Fred

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 08 Aug 2014 18:49:51 +0200

On 08/07/2014 04:41 PM, John R Levine wrote:
>> dip into the database is a dip into the database and returning 100 
>> bytes is not more expensive than returning 10 bytes. ...
>
> Right, but given the way that domain names are used, I'm wondering 
> whether they'd have a separate faster database just to check 
> existence.  Domain snipers hammer on registries trying to snap up a 
> domain the millisecond it's freed, so it's a problem they already have.
>
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

Hi,

one DNR backend developer who would like to add his 2 cents here.

Starting from more general perspective - I mean no offense but suggest 
to be careful not get into premature optimization and optimization for 
unknown requirements especially. You are basically trying to guess 
implementation details of multiple complex DNR systems each of those 
having multiple layers of software (OSes, storage systems, several 
caching levels, libraries, ...) in order to optimize for those. This 
seems just impossible to me and would probably do more harm than good.

Regarding traditional RDMS - it is of course always more expensive to 
return more data (especially more data qualities - think e. g. of SQL 
JOIN statement) BUT in the whole context of DNR architecture this is 
almost probably irrelevant and negligible. So in short - the choice 
whether return 50 complete objects data or 50 string handles should be 
absolutely irrelevant for DNR operator.

My suggestion is to largely ignore DNR implementations when discussing 
RDAP protocol design and focus mainly on usability (use cases, real data 
statistics etc.). Should there be any regard to any implementation I 
would think more about client software as I guess there will be more 
code to be written from scratch and fewer hw resources.

Regards

Jan

-- 

Jan Korous

CZ.NIC
Americka 23
120 00 Prague
Czech Republic

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Mon, 11 Aug 2014 17:52:33 +0000

From: weirds [mailto:[email-masked]] On Behalf Of Andy Newton
Sent: Tuesday, August 05, 2014 1:53 PM
To: John Levine
Cc: [email-masked]
Subject: Re: [weirds] I-D Action: draft-ietf-weirds-rdap-query-11.txt

[SAH] Before I actually put writing time into these let me ask one more time if anyone has any concerns or statements of support. #8 is "Search of domain names by nameserver name.  It could look something like /domains?nsLdhName=XXXXX" #9 is "Search of domain names by nameserver IP address. It could look something like /domains?nsIp=YYYYY". Yes? No?

[SAH] Does anyone have any issues with closing this? Search by entity handle is currently described in Section 3.2.3.

-andy