Subject: [alto] Info Resource Dirs, OPTIONS, 300 Multiple Choices and all that jazz ....

From: Bill Roome <[email-masked]>
Date: Thu, 28 Jul 2011 10:35:24 -0400

Section 7.6.2, Encoding of Information Resource Directory:

   "Each entry MUST indicate a URI that either directly provides the
   indicated Information Resource, or responds to a HTTP OPTIONS request
   which provides an Information Resource Directory with entries of
   additional Information Resources.

   "If an ALTO Client makes a GET or POST request to a URI that does not
   directly provide an indicated Information Resource, the ALTO Server
   MUST either reply with an HTTP 300 status code ("Multiple Choices")
   and an Information Resource Directory in the HTTP response¹s entity
   body, or indicate an appropriate HTTP status code. Note that in
   general, it is preferred that ALTO Clients use HTTP OPTIONS requests
   to discover additional Information Resources."

Here's my philosophy about protocol designs: client-server protocols
should be designed to simplify the client. If that pushes work onto the
server, tough! The goal is to simplify the client.

So what's the problem with those paragraphs? Consider the first. That says
the server has two distinct choices. How can a client discover WHICH
choice the server made? "Try it and find out."

Please! That's an insult to client implementors. That's a clear case of
designing to simplify the server, at the expense of the client.

Furthermore, this means a client must use three techniques to get the
information from a info directory:
  * A GET request to the primary directory URI,
  * An OPTIONS request to a multiple-type URI, or
  * A "300 Multiple Choices" error response to a GET or POST.

So what's my suggestion?  Drop those two paragraphs completely. Instead,
allow the information resource directory to contain an entry for ANOTHER
resource directory, with additional services. The media-type would be
application/alto-directory+json, of course. Eg,

    }, {
       "uri" : "http://custom.alto.example.con/extra-dictionary",
       "media-types" : [ "application/alto-directory+json" ]
    }, {

Then if a client wants to explore those services, it can do a GET on that
URI.

And for those who think the original specification is necessary -- the
OPTIONS and 300 response and all that -- now that we've had a bake-off,
how many servers actually implemented those options? How many clients
supported them?

	- Bill Roome

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 28 Jul 2011 23:09:43 +0800

The idea that you propose - and your philosophy - violates one of the core tenets of RESTful design by increasing the coupling between client and server.

Stronger coupling is rarely good in the long run.  When you do that, you impose restrictions on both entities.  

Providing a robust resource discovery mechanism requires the sorts of mechanisms that are in the draft.

Avoiding trial and error is part of the reason that discovery exists.  However, that does not obviate the need for good error recovery. 

Your concern that a client is going to be forced to do more work than absolutely necessary is a little strange.  The directory provides enough information for a client to do what it needs to do.  

A basic client probably doesn't need to worry about 300 responses or OPTIONS for anything other than error recovery, providing that the server is sensibly structured and the directory is accurate.

The reason that we provide OPTIONS responses is that the directory cannot always provide _authoritative_ information about a particular resource, just a hint about it.  If the hint turns out to be wrong, then OPTIONS tells the client what the resource really provides.

That said...  The way the text is structured implies something more complicated than it needs to be.  That is, it implies that an entry MAY identify a resource in a way that is not sufficiently defined to actuate that resource.  That's bad.  It does the clients a disservice and ought not happen in anything other than an error condition.

Structuring the description in a different way could alleviate the problem.

1. a directory identifies resources as completely as possible
2. error handling requires good client feedback:
 (a) all resources provide responses to OPTIONS requests that identify their own capabilities
 (b) resources provide appropriate HTTP responses so that clients can correct mistakes (405, 406, etc...)

--Martin

On 2011-07-28 at 10:35:24, Bill Roome wrote:
> should be designed to simplify the client. If that pushes work onto 
> the server, tough! The goal is to simplify the client.
> 
> So what's the problem with those paragraphs? Consider the first. That 
> says the server has two distinct choices. How can a client discover 
> WHICH choice the server made? "Try it and find out."
> 
> Please! That's an insult to client implementors. That's a clear case 
> of designing to simplify the server, at the expense of the client.
> 
> So what's my suggestion?  Drop those two paragraphs completely.
> Instead, allow the information resource directory to contain an entry 
> for ANOTHER resource directory, with additional services. The media- 
> type would be application/alto-directory+json, of course. Eg,
> 
>     }, {
> Then if a client wants to explore those services, it can do a GET on 
> that URI.
> 
> OPTIONS and 300 response and all that -- now that we've had a 
> bake-off, how many servers actually implemented those options? How 
> many clients supported them?
> 
> 	- Bill Roome
> 
> 
> _______________________________________________
> alto mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/alto

---

From: Bill Roome <[email-masked]>
Date: Thu, 28 Jul 2011 12:33:10 -0400

I would appreciate it if you could explain how my proposal increases the
coupling between the client and the server.

	- Bill

On 07/28/2011 11:09, "Thomson, Martin" <[email-masked]>
wrote:

>The idea that you propose - and your philosophy - violates one of the
>core tenets of RESTful design by increasing the coupling between client
>and server.
>

---

From: Robert Varga <[email-masked]>
Date: Thu, 28 Jul 2011 10:17:25 -0700

>   (b) resources provide appropriate HTTP responses so that clients can correct mistakes (405, 406, etc...)

I think a flow diagram or description of how exactly the client/server 
interaction w.r.t. these three ways of discovering resources would be 
very useful.

The protocol, as well as discovery, assume the client's first request 
would be for the IRD. I do not exactly see why the IRD cannot be 
strictly authoritative for non-IRD resources. If it has 
non-authoritative knowledge, it should point to IRD of the authoritative 
entity -- simple and extensible.

Bye,
Robert

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 29 Jul 2011 01:19:15 +0800

On 2011-07-28 at 12:33:10, Bill Roome wrote:
> I would appreciate it if you could explain how my proposal increases 
> the coupling between the client and the server.

I apologise, your proposal did not increase coupling.  'twas the expression of philosophy that caused the allergic reaction.

Thinking on it more, the technical proposal - that a directory can identify other directories - is perfectly fine.  It's suboptimal, given that the first directory could just as easily include the second, but it's perfectly sound.

There's a problem with resources that don't support GET, which is one of the cases where you get 300 responses and where you need OPTIONS requests.  But the general guidance there is to avoid the creation of resources that don't do GET.

--Martin

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 29 Jul 2011 01:22:16 +0800

On 2011-07-28 at 13:17:25, Robert Varga wrote:
> I do not exactly see why the IRD cannot be 
> strictly authoritative for non-IRD resources. 

This is a general HTTP thing.  You can't get authoritative information about any resource from anywhere but that resource.

It is possible for the IRD to be hosted on a different server than the non-IRD resource.  In that case, the IRD resource might be based on outdated information.  You also have to consider an adversarial model for the two.

--Martin

---

From: Bill Roome <[email-masked]>
Date: Thu, 28 Jul 2011 14:07:12 -0400

I understand; I sometimes have that affect on people!

And yes, allowing a directory to identify other directories is
sub-optimal. But it's useful when the entity providing the resource
directory knows there's another server, but does not have the
authoritative answer for exactly what services that server provides.
Hence, "Sorry client, but ya better get the real information from this
guy."

Actually, here's my proposal in more detail. Each entry in the resource
directory would have one -- AND ONLY ONE! -- media type, and 0 or 1 accept
types. That's sufficient to define the service that entry provides and the
method it expects.

This does NOT mean that a URI is limited to one service, though, because a
uri can appear in several entries. For example, the following means that
"map-service" responds to a GET and returns the full map, or to a POST and
returns a filtered map:

  }, {
    "uri" : "http://foobar/map-service",
    "media-types" : [ "application/alto-networkmap+json" ]
  }, {
    "uri" : "http://foobar/map-service",
    "media-types" : [ "application/alto-networkmap+json" ]
    "accepts" : [ "application/alto-networkmapfilter+json" ]
  }, {

I'd also suggest that the resource directory MUST be authoritative (at
least within any Expires time). If the server cannot provide an
authoritative description of the detailed services, then it should have
give the uri of a directory server that can.

Given that, is OPTIONS still necessary?

	- Bill Room

On 07/28/2011 13:19, "Thomson, Martin" <[email-masked]>
wrote:

>I apologise, your proposal did not increase coupling.  'twas the
>expression of philosophy that caused the allergic reaction.
>
>Thinking on it more, the technical proposal - that a directory can
>identify other directories - is perfectly fine.  It's suboptimal, given
>that the first directory could just as easily include the second, but
>it's perfectly sound.
>
>There's a problem with resources that don't support GET, which is one of
>the cases where you get 300 responses and where you need OPTIONS
>requests.  But the general guidance there is to avoid the creation of
>resources that don't do GET.
>
>--Martin
>
>

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 29 Jul 2011 02:16:03 +0800

On 2011-07-28 at 14:07:12, Bill Roome wrote:
> Actually, here's my proposal in more detail. Each entry in the 
> resource directory would have one -- AND ONLY ONE! -- media type, and 
> 0 or 1 accept types. That's sufficient to define the service that 
> entry provides and the method it expects.

That sounds reasonable, and it does resolve the problem that you don't know what combinations of request and response content type are supported.  However, it doesn't reduce the complexity of the complete model.

However, if you have a statement that if a resource is indicated with multiple, then it must support all combinations of requests.

Given that, the affect on an implementation that doesn't want to deal with this can be minimized if you are careful.

> least within any Expires time). 

Sadly, that's not possible.  Even if the directory and the indicated resource are on the same server, there is no guarantee.

> Given that, is OPTIONS still necessary?

Sadly, yes.  However, you are always free to not request it.  Similarly, I expect that clients will have to survive if a resource doesn't support it.

--Martin

---

From: Richard Alimi <[email-masked]>
Date: Thu, 28 Jul 2011 11:25:23 -0700

>    "accepts" : [ "application/alto-networkmapfilter+json" ]

One quick note. We already say that this one must indeed be split out:

   If an entry has an empty list for "accepts", then the corresponding
   URI MUST support GET requests.  If an entry has a non-empty list for
   "accepts", then the corresponding URI MUST support POST requests.  If
   an ALTO Server wishes to support both GET and POST on a single URI,
   it MUST specify two entries in the Information Resource Directory.

Your point does apply in the general case though (e.g., networkmap and
costmap at the same URI).

Rich

---

From: Bill Roome <[email-masked]>
Date: Thu, 28 Jul 2011 14:56:19 -0400

So the directory server DOES have an authoritative list of uris -- because
there is no other URI discovery mechanism! -- but it doesn't really know
what services those URIs provide. That sounds truly bizarre to me. How
could that happen?

Okay, I know how that COULD happen -- idiotic configuration!

But more precisely, could you explain why a well-designed designed ALTO
server can give an authoritative list of URIs WITHOUT also knowing what
services they provide?

	- Bill Roome

On 07/28/2011 14:16, "Thomson, Martin" <[email-masked]>
wrote:

>Sadly, that's not possible.  Even if the directory and the indicated
>resource are on the same server, there is no guarantee.

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 29 Jul 2011 04:43:58 +0800

On 2011-07-28 at 14:56:19, Bill Roome wrote:
> So the directory server DOES have an authoritative list of uris -- 

The IRD is not authoritative.  Sorry if you got an impression otherwise.  The directory is much like a DNS delegation - a hint from the parent domain that the child domain might be present at a particular place.

Keep in mind that the IRD might be on a different server to the individual resources.

--Martin

---

From: Robert Varga <[email-masked]>
Date: Thu, 28 Jul 2011 13:57:09 -0700

> Keep in mind that the IRD might be on a different server to the individual resources.

Coming from HTTP, yes. Two questions arise:
- who has the authoritative information? OPTIONS request seems to be the 
last instance, but it does not have to exist...
- could we not be a limit this a bit in exchange for greatly simplifying 
clients?

I really would like to see this simplified, as all this complexity does 
not seem to be founded on documented ALTO deployment use cases...

Bye,
Robert

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 07:58:39 -0700

Replying to this thread, but not all in the same email...

On Thu, Jul 28, 2011 at 7:35 AM, Bill Roome <[email-masked]> wrote:
...
> how many servers actually implemented those options?

The server that I didn't get a chance to demo did not support either,
but due to the way I chose to lay out the resources, it didn't need
to.  Each entry provided one and only one entry, and the
implementation serving the directory knew the full URI to each
individual resource.  Since each URI supported only one resource,
OPTIONS wasn't required by the current text (Martin suggests it should
be, but I'll respond to that one elsewhere in this thread).
Similarly, a resource doesn't need 300 Multiple Choices in this case.

Admittedly, this is the simple scenario for clients.  So the rest of
this thread regarding flexibility is the one that reflects my actual
belief :)

> How many clients

My client discovered the appropriate URI to request by recursing
directories via OPTIONS requests (aside from the root directory being
requested via GET).  Supporting 300 Multiple Choices responses wasn't
done in time, but is simple to add and there's a marker in the code to
do it.

Rich

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 08:02:44 -0700

Hi Martin,

On Thu, Jul 28, 2011 at 8:09 AM, Thomson, Martin
<[email-masked]> wrote:
...
> That said...  The way the text is structured implies something more complicated than it needs to be.  That is, it implies that an entry MAY identify a resource in a way that is not sufficiently def…

I'll note that the original text was less perscriptive in that it did
not recommend that clients use OPTIONS before they actuate the
resource.  The new text was written in response to Ben's comment: " I
don't think POSTing to expect a 300 "multiple choices" response is
what folks should be doing"
(http://www.ietf.org/mail-archive/web/alto/current/msg01035.html).

Looking back, this was regarding a particular example of a POST. In
that case, I agree with him (and I suspect he was only responding to
this particular case and not in general; I'm sure he'll correct me if
I'm wrong).  However, for a GET, I would think that requesting via GET
and then handling a 300 if it comes back is the better way (removes a
round-trip in the common case).

>  (b) resources provide appropriate HTTP responses so that clients can correct mistakes (405, 406, etc...)

In general this is fine, but note that (2a), AFAIK, is giving up the
ability to deploy an ALTO Server by dumping some static files on a
standard HTTP Server (unless http server implementations that I don't
know about do support serving a static file in response to OPTIONS).
That was appealing in its simplicity and particularly if we would like
to reduce the barriers to adoption from the provider side.  However,
the discussion regarding making other services mandatory may end up
removing this anyways.

In (1), what does that actually mean to server implementers code-wise?
 Reduce the degrees of freedom (media-types, accepts, cost modes, cost
types) for each resource entry as much as possible?  Otherwise it
seems like some warm and fuzzy guidance that is hard to translate into
actual code.  Something more explicit along the lines of what Bill
described (exactly 1 media-type, 0 or 1 accepts) might be better here.

I'll note that aside from what I mentioned above regarding OPTIONS, I
would have thought that (2) should already be handled in the draft's
text (maybe aside from the "correct mistakes" phrase).  If not, can
you (or someone) suggest improved text?

Rich

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 08:03:25 -0700

Hi Bill,

>    "accepts" : [ "application/alto-networkmapfilter+json" ]

As I noted in a previous response, this is already done to distinguish
between GET and POST.  I'm okay with adding requirements as you note
above. It can reduce a small bit of logic at the client, but, as
Martin mentioned, I don't think it solves the problem you are trying
to solve.

Rich

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 08:05:06 -0700

On Thu, Jul 28, 2011 at 11:56 AM, Bill Roome <[email-masked]> wrote:
> what services those URIs provide.

I'm confused.  The current mechanism supposes that a client knows
which service it wants.  The directory and HTTP responses let it
discover the correct URI for that service.

I don't think its ever the case that a client or server don't know
which service(s) a URI provides.  The server (which produces the
directory), and by extension the client (because it consumes the
directory) knows the set of services that a URI provides.  The HTTP
server / ALTO Server managing that particular URI gets to say whether
that URI directly provides that service, or whether some other URI
that it manages actually provides it (and it returns an HTTP status
code indicating which is true).

Or is there a different scenario you have in mind?

Rich

> That sounds truly bizarre to me. How
> could that happen?
>

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 08:06:55 -0700

Hi Robert,

On Thu, Jul 28, 2011 at 1:57 PM, Robert Varga <[email-masked]> wrote:
>> The IRD is not authoritative.  Sorry if you got an impression otherwise.
>>  The directory is much like a DNS delegation - a hint from the parent domain
>> that the child domain might be present at a particular place.
>>

Well, whether it exists or not is under discussion.  Currently, if
OPTIONS doesn't exist, the one providing the directory is supposedly
authoritative.

To answer your question, the last server to give the client a
resource's URI is authoritative for that resource.  In a REST-ful
design (similar to DNS delegation as Martin pointed out), the root
(and if used, intermediate) directories only help you get there.

> - could we not be a limit this a bit in exchange for greatly simplifying

I'm not convinced this is difficult for clients.  For example, beyond
code that was already written for requesting and parsing requests, my
code for finding the URI for a resource is only 30 lines of code
(including logging), plus another 30 lines of code for determining if
a particular directory entry matches (which would be significantly
reduced if it weren't C++).  As Martin pointed out in another thread,
cachability is important on these responses, so once those are
appropriately specified by servers, the extra RTT's are typically
non-existent.

> I really would like to see this simplified, as all this complexity does not
> seem to be founded on documented ALTO deployment use cases...

Please don't let us return to -07....

Seriously though, REST-ful principles aren't as much about capturing
existing deployments (there are no deployments that speak this
protocol AFAIK).  They purposefully allow for very flexible deployment
scenarios, and give clients simple tools to allow for such flexible
deployments.

But, for what its worth, I could easily imagine a deployment scenario
in which endpoint cost and filtering are offloaded to a third party.
The existing scheme lets a server accomplish that painlessly and hides
the actual URLs used by the third party from the original ALTO Server
as implementation details.

Thanks,
Rich

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Sat, 30 Jul 2011 19:22:43 +0100

Robert,

On 28 Jul 2011, at 21:57, Robert Varga wrote:

> - who has the authoritative information? OPTIONS request seems to be the last instance, but it does not have to exist...

With HTTP the only thing that is truly authoritative for a given resource is the resource itself. That's just a fact of life when using HTTP.

> - could we not be a limit this a bit in exchange for greatly simplifying clients?
> 
> I really would like to see this simplified, as all this complexity does not seem to be founded on documented ALTO deployment use cases...

In general REST-ful design places some additional complexity on clients in order to reduce coupling between clients and servers and to provide server implementations with the flexibility to organise resources as they see fit.

As ALTO is using HTTP as a "transport" an ALTO server is within its rights to use any valid HTTP response as part of an ALTO response. I think what the draft is trying to do in some places is clarify some of the behaviour that is familiar to folks versed in HTTP but may not be immediately obvious to someone who isn't to try and avoid situations where an ALTO server returns a valid HTTP response but the ALTO client implementer/implementations reaction is "WTF did I get that response?"

Ben