Subject: datasets, referrals, and loop detection

From: Michael Mealling <[email-masked]>
Date: Wed, 13 Sep 2000 13:59:51 -0400

Hi all,
  After running through the documents one last time and putting some of
the finishing touches on our client, we have come across some issues that
need to be fixed with respect to datasets, referrals and loop detection.

  The issue comes up where in two places. The first is the fact that
a client can specify multiple dataseturis in a query. If the server
gives back results and no errors, should the client consider that
Service and _all_ of the dataseturis the client specified as 'visited'?
  This is part of a larger problem with the document not saying anything
about loop detection. IMHO, some of the changes that are need are listed
below:

1) Referential integrity: A server generates a referral that contains a
dataseturi. The client then connects to the referred to server and
submits the query with that dataseturi. If the server reports
that it either does not support datasets or doesn't know that dataseturi
then the client should consider that referral to be bad and should
drop the resutls on the floor. It should also consider this nod to be
visit.

2) a new status message:
3.1.3 Server does not support datasets

3) a new status message:
3.1.4 The first dataset in the list of datasets you gave in the query
      was not supported

4) A query contains properties and N of those properties contained a dataseturi.
   The client sends that query to a server and that server generates some
   number of referrals. The client MUST replace ALL of the dataseturi
   properties in the query with the one and only one dataseturi found
   in the referral that it is currently handling.

5) loop detection in general:
   Unless given a status message otherwise, a node in a loop detection list
   is considered to be a service/dataseturi pair.

Comments? If I don't see any I'll just incorporate this stuff into the
document. Think about this one kind of hard since loop detection isn't easy.

-MM

--
--------------------------------------------------------------------------------
Michael Mealling        |      Vote Libertarian!       | www.rwhois.net/michael
Sr. Research Engineer   |   www.ga.lp.org/gwinnett     | ICQ#:         14198821
Network Solutions       |          www.lp.org          |  [email-masked]

---

From: "Marshall L. Moseley" <[email-masked]>
Date: Wed, 13 Sep 2000 12:51:07 -0700

>>) Referential integrity: A server generates a referral that contains a
dataseturi. The client then connects to the referred to server and
submits the query with that dataseturi. >>

Is this an auto-referral?  Or is the client being operated manually by the
user?  We said in Montreal we were not going to do auto-referrrals.

--Marshall

---

From: Michael Mealling <[email-masked]>
Date: Wed, 13 Sep 2000 18:34:01 -0400

> user?  We said in Montreal we were not going to do auto-referrrals.

'auto-referrals' (server side chaining) is where the server handles all of
the referrals. This case is where the client that the user is using actually
handles the referrals....

-MM

---

From: "Marshall L. Moseley" <[email-masked]>
Date: Wed, 13 Sep 2000 15:57:39 -0700

Then it seems to me we're dictating client behavior when it's not necessary.
How a client chooses to handle referrals is a service differentiator.
Common sense says yes, the referral would treated as bad and not visited,
but I'm not sure that's an area in which we should be dictating anything.

--Marshall

---

From: Michael Mealling <[email-masked]>
Date: Wed, 13 Sep 2000 19:29:00 -0400

> but I'm not sure that's an area in which we should be dictating anything.

If its a feature specific to a particular server or service then yes. But
this speaks to interoperabilty between services and as such the client
implementors will be looking to this specification for what the basic
interoperabilty rules are for when they end up talking to multiple servers.
Without basic rules like this referrals are really all but useless...

-MM

---

From: Leslie Daigle <[email-masked]>
Date: Sat, 16 Sep 2000 16:33:51 -0400

Howdy,

I agree with your point of view that the basic semantics of referrals
and datasets needs to be agreed upon.  It is a bit delicate --
following on Marshall's comments, this is essentially saying that
some properties are handled more rigorously than others.  On
the other hand, perhaps the appropriate path to take (in keeping
with the "not every client/server will want to use datasets") is
to focus this as:  you can do whatever you want, but here is the
correct way to handle dataseturis in referrals if you want to
cover the area and avoid looping.  I.e., as a _client_ implementation
option.

At any rate, I'm having difficulty teasing apart what you are getting
at below; comments/questions in-line.

> Service and _all_ of the dataseturis the client specified as 'visited'?

In order to be as flexible as possible here, perhaps the short answer
should be:

        Yes, unless the server responds with a warning message
        to the effect that "Not all datasets recognized."

It doesn't need to specify which ones.

> that it either does not support datasets or doesn't know that dataseturi

Presumably the same warning/error message as above.  Also, I would
presume that the client could connect to the referred-to server
and make the query without the dataseturi property (just because it
can).

> drop the resutls on the floor. It should also consider this nod to be

This is true from the perspective of actively using the datasets
in the referral.  A client may nevertheless accept the results,
mark the node as visited, but not consider the dataset visited.  This
is particularly relevant if the _client_ doesn't want to work too
hard on the dataset idea.

> 3.1.3 Server does not support datasets

That, too.

> 3.1.4 The first dataset in the list of datasets you gave in the query

I don't understand why "the first" is singled out?

>    in the referral that it is currently handling.

IFF you want to be rigorous about pursuing referrals; otherwise, you
may not do this, and just consider the referrals as "hints".

Leslie.

--

-------------------------------------------------------------------
"Reality with a delicate splash of the imaginary...
    ... or was that the other way around?"
   -- ThinkingCat

Leslie Daigle
[email-masked]
-------------------------------------------------------------------

---

From: Michael Mealling <[email-masked]>
Date: Mon, 18 Sep 2000 13:32:39 -0400

> cover the area and avoid looping.  I.e., as a _client_ implementation

hehe.... as long as we're finessing words, that sounds fine...

> It doesn't need to specify which ones.

Right. That's message 3.1.3 from below...

> and make the query without the dataseturi property (just because it

Sure. But if you're writing a client (as we are) that wants to
enforce referential integretity then you'd need this message. If
your client doesn't then you can safely ignore it.

> hard on the dataset idea.

Sure.... it may get duplictae results because it doesn't do rigorous
loop detection but that's its problem...

> I don't understand why "the first" is singled out?

typo... it should be this:

3.1.4 This server does not support multiple datasets in a query but it
      does support datasets, therefore only the first was used and therefore
      you should only mark the first dataset as visited.

This is so that the server can generate referrals to datasets in the query
that it may not actually support itself. If the client were to send
multiple datasets to the server and the server itself couldn't support
all of them it could generate a referral to them. Without this message
the client would immediately mark them all as visited even if the
server wasn't capable of understanding the query completely.

> may not do this, and just consider the referrals as "hints".

Yep. If you treat referrals as hints and don't do loop detection you
may find infinite loops. If you do loop detection but don't follow these rules
then you can end up with black holes. If your application doesn't care
about either of these then it isn't an issue....

-MM

---

From: Leslie Daigle <[email-masked]>
Date: Mon, 18 Sep 2000 21:21:11 -0400

I'm sorry... I think I'm being amazingly obtuse, but:

> server wasn't capable of understanding the query completely.

Is this (from your earlier message):

"  The issue comes up where in two places. The first is the fact that
a client can specify multiple dataseturis in a query. If the server
gives back results and no errors, should the client consider that
Service and _all_ of the dataseturis the client specified as 'visited'?"

In which case, you're asking for a way to determine which
datasets can be marked as visited?  In which case, I don't see
what the proposed status message does to elucidate what's really
happened.

I think I need to see the whole proposed text, because agreeing to
things "piecemeal" sounds like it's creating something awfully
complex...

(But, maybe I'm just being incredibly obtuse...).

Leslie.

--

Leslie Daigle
[email-masked]

---

From: Michael Mealling <[email-masked]>
Date: Tue, 19 Sep 2000 15:56:28 -0400

> I'm sorry... I think I'm being amazingly obtuse, but:

Not really. This stuff ain't easy....

> what the proposed status message does to elucidate what's really

It covers the case where a server can only handle one dataset at
a time in a query. Unless we want to make the statement that, if a server
supports datasets, it has to be able to support mutliple datasets
in a query, OR remove the ability to specify multiple datasets in
a query.

> (But, maybe I'm just being incredibly obtuse...).

Nope. This wouldn't be so hard if we just didn't allow multiple datasets in
a query.  The only other alternative is to start saying that the contents
of particular messages have structure and that scares me...

-MM