Subject: Request that the WG reconsider section 3.4: Content Negotiation

From: [email-masked] (Henry S. Thompson)
Date: Mon, 04 Nov 2013 16:53:30 +0000

It's my impression that content negotiation hasn't turned out to play
the kind of significant role in Web Architecture in general, or in
HTTP use in particular, that was expected for it.

I think the section on conneg in p2-semantics [1] is so out-of-step
with actually deployment, usage and expectations that to publish it as
it stands would be a serious mistake.

In particular, the discussion of the relative disadvantages of the
newly (re-)named 'proactive' and 'reactive' variants are not only
out-of-date, but also this discussion appears to at least this reader
to amount to a recommendation for 'reactive' negotiation.  Yet as far
as I can tell no user agents _or_ servers actually support this
approach today, as it's described here.

I was sufficiently concerned about this question to undertake a
moderately extensive empirical investigation [2].  To summarise
perhaps too briefly, I found _no_ evidence of the use of reactive
conneg in over 75 million HTTP request/response exchanges.

I think 3.4 can and should be substantially simplified, with all the
evaluative/speculative prose removed, focussing simply on the
semantics of the Accept... headers and the 300 and 406 status codes,
perhaps also making clear that 'proactive' conneg is the only form of
conneg with any signficant degree of server-side support.

[2] discusses all this at more length -- I hope it will be helpful.  I
am of course aware that personal experience, backed up by one small
study, may well be misleading, but I did look moderately hard to find
any other relevant experimental results without success.  It is less
likely, but not impossible, that what I report in [2] about what is
implemented in IIS, Apache and the major web browsers is also
mistaken.  On either count, I would welcome concrete evidence of where
I've misunderstood or misrepresented the actual situation.

ht

[1] http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-24#section-3.4
[2] http://www.ltg.ed.ac.uk/~ht/reactive_conneg.html
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 04 Nov 2013 17:07:50 +0000

In message <[email-masked]>, Henry S. Thompson writes:

>I think 3.4 can and should be substantially simplified, 

Seconded.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 04 Nov 2013 18:50:18 +0100

>approach today, as it's described here.

It does not seem far off, it just so happens that for sub-resources a
substantial part of the process often happens on a level above HTTP.
As noted in the draft, it can be costly to contact the server first,
and then contact it again to request some alternative representation,
compared to encoding "we serve WOFF and SVG fonts, take your pick" in
HTML or CSS code. It might be good to note that shift in the document,
but I do not agree that keeping the text as is would be a "serious
mistake".

>conneg in over 75 million HTTP request/response exchanges.

So with `<a href='example.txt' rel='alternate'>Download plain text
version of this document</a>`, if the user downloads the plain text
version, would that be evidence? If not, is the problem that the
user agent did not generate a redundant selection menu, or perhaps
because people do not include the `rel='alternate'` attribute since
user agents tend to ignore that? Or because this doesn't use HTTP
headers (what if it was <link> elements instead)?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 04 Nov 2013 23:55:23 +0100

On 2013-11-04 17:53, Henry S. Thompson wrote:
> ...

Reactive conneg isn't just about 300s and 406s. Another example would be 
a representation returned with a 200 response that contains links to 
alternate versions of the content. That's what the

"If the user agent is not satisfied by the initial response 
representation, it can perform a GET request on one or more of the 
alternative resources, selected based on metadata included in the list, 
to obtain a different form of representation for that response. 
Selection of alternatives might be performed automatically by the user 
agent or manually by the user selecting from a generated (possibly 
hypertext) menu."

in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-24.html#rfc.section.3.4.2.p.1> 
is about.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 4 Nov 2013 15:12:34 -0800

What Julian says. Reactive negotiation is very common; formats are coming up with mechanisms for it all the time (e.g., the work on srcset, etc. in the W3C web perf WG).

Cheers,

On 4 Nov 2013, at 2:55 pm, Julian Reschke <[email-masked]> wrote:

> Reactive conneg isn't just about 300s and 406s. Another example would be a representation returned with a 200 response that contains links to alternate versions of the content. That's what the
> 
> "If the user agent is not satisfied by the initial response representation, it can perform a GET request on one or more of the alternative resources, selected based on metadata included in the list, to obtain a different form of representation for that response. Selection of alternatives might be performed automatically by the user agent or manually by the user selecting from a generated (possibly hypertext) menu."
> 
> in <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-24.html#rfc.section.3.4.2.p.1> is about.
> 
> Best regards, Julian

--
Mark Nottingham   http://www.mnot.net/

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 05 Nov 2013 12:10:11 +0000

> substantial part of the process often happens on a level above HTTP.

Then it doesn't belong in the HTTP spec., surely!

> but I do not agree that keeping the text as is would be a "serious

When it's not supported as part of HTTP, anything which suggests that
it is is surely just that, seriously misleading.

> version, would that be evidence?

Not as far as inclusion in HTTPbis, no.

> If not, is the problem that the user agent did not generate a
> redundant selection menu,

In response to what?  A 300 response?  Yes, that's the point.  No user
agent I'm aware of does _anything_ with a 300 response (except insofar
as they treat it as a 200).

> or perhaps because people do not include the `rel='alternate'`
> attribute

What does rel='alternate' have to do with conneg, or HTTP?  Its
semantics are defined, as you say above, at an entirely different
level.

Although intended as a rejoinder, your response in fact reads to me as
an endorsement.

ht

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 05 Nov 2013 12:18:23 +0000

Julian Reschke writes:

> Reactive conneg isn't just about 300s and 406s. Another example would
> be a representation returned with a 200 response that contains links
> to alternate versions of the content. That's what the

How is this in scope for discussion _in the HTTP spec._?  People (not
user agents, note) use 200 responses for a huge range of interesting,
powerful, innovative things.  We don't look in the HTTP spec. to find
a discussion of them.

> alternative resources, selected based on metadata included in the
> list, to obtain a different form of representation for that
> response. Selection of alternatives might be performed automatically
> by the user agent or manually by the user selecting from a generated
> (possibly hypertext) menu."

"based on metadata included in the list"!  That's a specific reference
to a 300 response.  There is no "list" in a 200 response, not that a
user agent can detect anyway.

And, as I said in the last message, I'm not aware of _any_ user agent
that does _anything_ specific to 300 responses.

ht

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 05 Nov 2013 13:50:55 +0100

>semantics are defined, as you say above, at an entirely different

"Reactive negotiation" is performed "after receiving an initial response
from the origin server that contains a list of resources for alternative
representations." and in HTML `rel='alternate'` is a way to encode such
information that allows selecting an alternate representation "manually
by the user". The text in question describes using HTTP header fields to
carry this information as an "if" and for status codes like 300 and 406
the draft specifically says that the payload should include such a list.

There is nothing wrong with discussing in the specification that as an
alternative to the "you say what you like and then the server chooses"
it is also possible to implement "the server says what it has and then
you choose" and including hyperlinks in the response body is a widely
used way of doing so and it is implemented e.g. by search engines like
Google, <https://support.google.com/webmasters/answer/189077>. When my
server responds with, using the example there,

  HTTP/1.1 200 OK
  Link: <http://es.example.com/>; rel="alternate"; hreflang="es"
  ...

and a search engine crawler solely interested in spanish content then
automatically chooses to follow the link and to ignore the non-spanish
site, how is that not "Reactive negotiation" as described in the draft?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 05 Nov 2013 14:05:34 +0000

Bjoern Hoehrmann writes:

> by the user".

See my response to Julian.  Either "contains a list" is a reference to
a 300 response, in which case I repeat my request to be pointed to
_any_ user agent which implements any kind of response to a 300 (other
than to treat it as a 200), or it refers to something in the message
body, in which case it has nothing to do with HTTP as a protocol.

Similarly, the semantics of rel='alternate' is entirely a matter for
the HTML specifications, it doesn't interact with HTTP-level conneg at
all.  I can't use rel="alternate" to, for example, force a particular
'Accept' header, and even if I could that _still_ wouldn't make this
reactive conneg, just good old simple proactive.

> The text in question describes using HTTP header fields to

Right, but until I see an example of _any_ user agent which actually
does _anything_ with a 300 header as such, I think describing 300 as
if it did anything is seriously misleading.

> used way of doing so

I see nothing in the spec. that suggests that hyperlinks in the
response body are in scope in section 3.4.  From the perspective of
HTTP, the response body is opaque.  It's not part of any HTTP client's
HTTP-licensed behaviour to do anything with response bodies other than
hand them over to applications, is it?

> and it is implemented e.g. by search engines like

> site, how is that not "Reactive negotiation" as described in the draft?

Let's follow this hypothetical example through with a bit of care.  

 1) Crawler does 'GET' of http://www.example.com/;
 2) Response comes back with
   HTTP/1.1 200 OK
   Link: <http://es.example.com/>; rel="alternate"; hreflang="es"
   . . .
   Content-Language: [something other than es]

   . . .
   [some body]
 3) Crawler doesn't index this page because it's not Spanish
 4) Crawler does a 'GET' of http://es.example.com/ and, presumably,
    does index it.

Frist, note of course this is not what Google itself does -- it
indexes the original page, it just doesn't show it to someone who asks
for Spanish-only results.

Second, whether the behaviour of your hypothetical crawler falls under
the description of reactive conneg or not is beside the point.  If it
does, then the definition of reactive conneg _in the HTTP spec_ is too
broad, because it doesn't tell us anything we need to know about how
conforming clients, proxies and origin servers have to behave wrt HTTP
requests and responses.  It doesn't have any semantics at the HTTP
level:  The Link: header in 5988 is a convenience for users that is,
effectively, layered on top of HTTP.  After all, you could have chosen
to encorporate it into HTTPbis, but you, correctly, didn't.

The fact of the matter is that at the HTTP protocol level, all that is
absolutely necessary in section 3.4 and the later sections is an
introduction of how the semantics of the Accept... request headers,
the 300 and 406 status codes and the Vary response header fit
together.

Going a bit beyond that to point out that use of Accept... headers is
common practise in modern browsers and means of controlling
Content-Type and Content-Language are sometimes made available to
users by origin servers, but that little if any support for 300
responses is found in user agents would obviously be helpful to
implementors and users.

Going much beyond _that_, as the spec. does now, is either out of
scope or misleading or both.

ht

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 05 Nov 2013 16:38:11 +0100

>body, in which case it has nothing to do with HTTP as a protocol.

The redirection response codes are for cases where further action needs
to be taken. The draft specifically mentions that a server might prefer
such action over offering one of the alternatives directly in which case
300 might be a good choice for a response code. Consider as an example

  Please choose your language to continue: English, Deutsch

This requires the user to react. A common alternative would be

  Herzlich Willkommen (also available in -> English)

Which allows the user to react but does not require it. That would be a
200 response. It does not make sense to me that a server would include a
list of available alternatives only if it forces the user to choose, so
your first interpretation strikes me as unreasonable.

Your second interpretation is mistaken, the draft specifically mentions
the option of including the list in header fields. It is appropriate to
discuss how users and clients and servers can select among alternative
representations including how to discover available alternatives. That
is a core concern for people developing and deploying implementations
of the protocol. If discovering alternatives often involves looking at
links in Hypertext documents, the Hypertext Transport Protocol specifi-
cation surely can mention that.

I think your proposal to remove a large portion of text without any
replacement would make the draft worse, not better. I do see much room
for improving the current text, but no particular need to do it.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 05 Nov 2013 16:06:28 +0000

> your first interpretation strikes me as unreasonable.

I'm not sure I follow the above, but at the very least if "contains a
list" is _not_ a reference to 300, that should be clarified.

> the option of including the list in header fields.

But says it's been removed!?  I presume you're refering here to 6.4.1 [1].  

Seems to me the first sentence of the first para there, namely

  The 300 (Multiple Choices) status code indicates that the target
  resource has more than one representation, each with its own more
  specific identifier, and information about the alternatives is being
  provided so that the user (or user agent) can select a preferred
  representation by redirecting its request to one or more of those
  identifiers.

But the next sentence

  In other words, the server desires that the user agent engage in
  reactive negotiation to select the most appropriate
  representation(s) for its needs (Section 3.4).

goes way too far.  All the actual uses of 300 responses we see on the
Web today invite, indeed require, the user, not the user agent, to
make a selection, and this really doesn't fit the definition of
reactive conneg from 3.4.

Note that _none_ of the 300 responses I explored included a Location:
header.

> It is appropriate to discuss how users and clients and servers can
> select among alternative representations including how to discover
> available alternatives. That is a core concern for people developing
> and deploying implementations of the protocol. If discovering
> alternatives often involves looking at links in Hypertext documents,
> the Hypertext Transport Protocol specifi- cation surely can mention
> that.

Only insofar as it makes clear that choices listed in the body, which
can only be interpreted relative to the Content-Type, are
application-specific, and not generic.  And that in fact no examples
are currently in place of user agents (as opposed to users) _making_
the choice.

Link: header elements enable HTTP-generic choice, but the spec. itself
is at least more honest about this case, by saying in the Note that
there's a chicken-and-egg problem wrt deployment.

> for improving the current text, but no particular need to do it.

3.4 describes reactive conneg in a way which is nowhere realised on
the Web today.  Leaving it as it is does your readers a disservice.

And, perhaps more important, by suggesting that reactive conneg is in
some way preferable, or more efficient, than proactive conneg, when
reactive conneg is at best rare, and the only examples you can offer
of it involve user interaction, and when proactive conneg _is_ widely
deployed, just reduces the credibility of the spec.

ht

[1] http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-24#section-6.4.1

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 5 Nov 2013 15:45:08 -0800

Hi Henry,

> powerful, innovative things.  We don't look in the HTTP spec. to find

You seem to be thinking of HTTP as a separate layer from the application; as section 3.4 of p2 says, it’s defining a pattern of use, and that is certainly in scope for this specification, given that it was also in scope for RF2616.

> user agent can detect anyway.

If there is more than one link in the response in *some* format that’s tied together in some fashion (e.g., with a link relation, as per RF5988), there certainly is a list available.

We talked about this issue at the WG meeting yesterday. Based upon that discussion, we decided to close this issue. Looking at <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#content.negotiation>, I can’t see any immediate clarifications that would help; if you can suggest some, please do bring them to our attention.

Regards,

---

From: Henri Sivonen <[email-masked]>
Date: Fri, 8 Nov 2013 11:58:10 +0200

On Mon, Nov 4, 2013 at 7:50 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> it just so happens that for sub-resources a

Right. Therefore, it would be appropriate to describe HTTP-level
content negotiation as a legacy thing.

See http://wiki.whatwg.org/wiki/Why_not_conneg and in particular the
codec thought experiment in the end.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/