Subject: Re: Atom Content Negotiation (follow-up)

From: Peter Krantz <[email-masked]>
Date: Sun, 1 May 2011 10:32:57 +0200

On Sat, Apr 30, 2011 at 21:20, Erik Wilde <[email-masked]> wrote:
>
> the problem is that this link relation needs a parameter (specifying what
> the media type of the linked feed/entry/content is) and maybe some title
> text.

Forgive me if I have misunderstood the use case scenarios here, but:

It looks like you will be duplicating the value of the link relation
in the link element type attribute for each entry in the target feed?
How would you handle discrepancies if the marked feed had entries with
a different value?

If there are multiple feeds for a given set of resources, how can I
trust I am getting the same updates in each of the feeds?

I see a lot of benefit in having one feed with multiple link elements.
If I'm only interested in the RDF entries I am still doing the same
number of requests (one for the feed and then one for each entry I am
interested in)?  And I can trust I get all updates at the same time.

Regards,

Peter Krantz

---

From: Erik Wilde <[email-masked]>
Date: Sun, 1 May 2011 10:21:40 -0700

hello peter.

thanks for your comment!

> How would you handle discrepancies if the marked feed had entries with

i think i wasn't clear enough. my proposal would only be for feed/link elements, not for feed/entry/link elements. the idea is to point to a variant of the whole feed, not to individual entries. it should be a link y can follow when you want to consume a different feed, one that has different "primary" content. and each "feed variant" would be a regular feed and thus be free to link to entry alternate versions. the label i am thinking of would indicate something you might call the feed's "primary content type".

> trust I am getting the same updates in each of the feeds?

i don't think you can trust the publisher a 100%, but it's a good thing to point out, and the link relation should probably say that "publishers SHOULD try to expose updates across feed variants in a consistent manner", or something along these lines.

> interested in)?  And I can trust I get all updates at the same time.

you're right that there is an issue with trust and consistency, which is probably very hard to formalize or standardize. on the other hand, if you follow a feed that embeds non-RDF (and limks to RDF alternates) and you got 10 new entries when reading the feed, you need a total of 11 GETs to retrieve the data you're interested in. if there was an feed embedding RDF, it's just one GET and you're done. for resource-constrained scenarios such as mobile or embedded systems, and in particular for push support, this is a very significant difference.

thanks and cheers,

dret.

---

From: James Holderness <[email-masked]>
Date: Mon, 2 May 2011 07:15:44 +0000

> i don't reject it, but it makes it necessary to issue additional GET 
> requests for each single entry. if you know that all you ever want are 
> the linked alternate variants, then it would be much more effective to 
> simply GET those in a feed (if there was one and you could find it).

With HTTP pipelining those multiple GETs shouldn't have that much of a negative impact, although I'll admit that in practice pipelining does have potential problems.

But even without pipelining, after the first feed retrieval, I would still expect those multiple GETs to be more effecient on subsequent updates. Two small GETs to retrieve one new feed entry would seem more efficient than one giant GET of a 20 entry feed when you only want the latest entry (unless you're also supporting RFC3229+feed?)

I guess this does depend to a some extent on the nature of your data, the frequency of updates, and the kind of clients you expect to have. Either way, though, I think you may be optimizing prematurely.

> i am interested in producing feeds that are useful to feed-consuming 
> clients. human-oriented feed clients prefer HTML, whereas other types of 
> clients might prefer XML or RDF or something else.

I still think you are going about it the wrong way. If the content-type for Atom doesn't sufficiently distinguish between the types of feed you want to serve for HTTP content negotiation to work, it seems to me you should be looking at ways to extend the content-type (like the type parameter that was proposed to distinguish between Atom entry documents and Atom feed documents). Inventing a new form of content negotiation that requires parsing links from the top of an Atom feed is just twisted.

But the real problem may just be an inappropriate use of Atom as a general purpose container format. If you have a client that prefers content as RDF, why not give them a true RDF feed? Why RDF embedded in Atom? That kind of misses the point of RDF. Even worse, if a client prefers JSON, do you really think it makes sense to serve them that JSON as base-64 encoded blobs inside an XML container? I can assure you that nobody is going to thank you for that option.

If you served RDF clients a real RDF feed, and JSON clients a real JSON format, then you could be using standard HTTP content negotiation - at least for most cases. If you wanted to provide an "HTML friendly" feed that was separate from the one with raw XML embedded, you'd still need a way to differentiate between the two, but at that point I would think it not worth the complication. I'd just make your feed content the raw XML, include a short summary, and then an alternate link to a more detailed HTML page if really necessary.

---

From: Erik Wilde <[email-masked]>
Date: Mon, 02 May 2011 09:09:19 -0700

hello james.

> With HTTP pipelining those multiple GETs shouldn't have that much of a
> negative impact, although I'll admit that in practice pipelining does
> have potential problems.
> But even without pipelining, after the first feed retrieval, I would
> still expect those multiple GETs to be more effecient on subsequent
> updates. Two small GETs to retrieve one new feed entry would seem more
> efficient than one giant GET of a 20 entry feed when you only want the
> latest entry (unless you're also supporting RFC3229+feed?)
> I guess this does depend to a some extent on the nature of your data,
> the frequency of updates, and the kind of clients you expect to have.

of course it does. and please not that everything i propose is entirely 
optional, of course. if you want to do it old-style, then that's fine. 
but in particular in the case of push with fat ping, there is a very 
substantial difference between being able to receive the data in the 
required format via push updates, or having to GET the alternate version 
of every single update.

> I still think you are going about it the wrong way. If the content-type
> for Atom doesn't sufficiently distinguish between the types of feed you
> want to serve for HTTP content negotiation to work, it seems to me you
> should be looking at ways to extend the content-type (like the type
> parameter that was proposed to distinguish between Atom entry documents
> and Atom feed documents). Inventing a new form of content negotiation
> that requires parsing links from the top of an Atom feed is just twisted.

i am not saying that i am proposing the perfect solution, and there are 
certainly different ways of approaching the problem i am looking at. it 
seems to me, however, that changing the way how this could be done in a 
more perfect way would require a lot of very substantial changes to the 
core of media types and HTTP content negotiation. it would be very nice 
to have HTTP content negotiation doing this, but unfortunately, atom 
sort of "hides" the "real content" (the /feed/entry/content) from 
visibility on the HTTP level.

> But the real problem may just be an inappropriate use of Atom as a
> general purpose container format. If you have a client that prefers
> content as RDF, why not give them a true RDF feed? Why RDF embedded in
> Atom? That kind of misses the point of RDF. Even worse, if a client
> prefers JSON, do you really think it makes sense to serve them that JSON
> as base-64 encoded blobs inside an XML container? I can assure you that
> nobody is going to thank you for that option.

saying that atom is inappropriate as a general purpose container format 
is your opinion. i happen to think that atom is exactly that. afaik, RDF 
does not have feeds, and if i as a service designer choose to model my 
service in a RESTful way based on atom abstractions, then atom probably 
is a very good way to represent that. serving RDF content as well would 
just be a convenience feature for those who would prefer to get RDF 
representations of what my service exposes.

here is something i wrote a while ago about atom as a general container:

http://dret.typepad.com/dretblog/2009/05/atoms-future-as-a-generalpurpose-format.html

you might not agree with what i am saying here, but there are more and 
more services where feeds are used as the RESTful abstraction of the 
service model, and this is the area i am looking at for the proposal i 
have made.

> If you served RDF clients a real RDF feed, and JSON clients a real JSON
> format, then you could be using standard HTTP content negotiation - at
> least for most cases. If you wanted to provide an "HTML friendly" feed
> that was separate from the one with raw XML embedded, you'd still need a
> way to differentiate between the two, but at that point I would think it
> not worth the complication. I'd just make your feed content the raw XML,
> include a short summary, and then an alternate link to a more detailed
> HTML page if really necessary.

again, you're saying "real RDF feed" and "real JSON feed" and these are 
not things i am aware of. it might be interesting to think of RDF and 
JSON serializations of the atom data model, but this would actually be 
very hard to do well because of atom's openness and support of XML 
namespaces. it's a very different discussion anyway, but we are 
currently in the process of encoding atom as an RDF ontology, and i can 
tell that it's suprisingly complicated.

so my assumption is to have atom XML as the feed container so that there 
is only one representation of the feed abstraction that can be handled 
by intermediaries, value-added components (such as push frameworks), and 
then make sure that various types on content can be transported in that 
framework in a well-defined and flexible way.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked]  -  tel:+1-510-6432253 |
            | UC Berkeley  -  School of Information (ISchool) |
            | http://dret.net/netdret http://twitter.com/dret |

---

From: Hadrien Gardeur <[email-masked]>
Date: Mon, 2 May 2011 18:18:49 +0200

>
> If the content-type for Atom doesn't sufficiently distinguish between the
> types of feed you want to serve for HTTP content negotiation to work, it
> seems to me you should be looking at ways to extend the content-type (like
> the type parameter that was proposed to distinguish between Atom entry
> documents and Atom feed documents). Inventing a new form of content
> negotiation that requires parsing links from the top of an Atom feed is just
> twisted.
>

+1. Media parameters are not sufficiently well defined yet, but I believe
that this is the right solution to this problem.

---

From: Erik Wilde <[email-masked]>
Date: Mon, 02 May 2011 10:29:56 -0700

hello hadrien.

On 2011-05-02 9:18, Hadrien Gardeur wrote:
> +1. Media parameters are not sufficiently well defined yet, but I
> believe that this is the right solution to this problem.

philosophically, i agree. the question i am asking myself is how likely 
it is that this is actually going to happen. it might require a rewrite 
of the media type specification, and a rewrite of how content 
negotiation in HTTP interacts with that updated media type model. is 
that something that is likely to happen, in the mid term or at all? i am 
having my doubts, which is why i was suggesting my admittedly suboptimal 
approach, which has the advantage of layering on top of the existing 
architecture without the need for updating some of the core standards. 
short version: do you think the right solution is going to happen? and 
if not, what then?

cheers,

dret.

---

From: Hadrien Gardeur <[email-masked]>
Date: Tue, 3 May 2011 16:05:55 +0200

>
> philosophically, i agree. the question i am asking myself is how likely it
> is that this is actually going to happen. it might require a rewrite of the
> media type specification, and a rewrite of how content negotiation in HTTP
> interacts with that updated media type model. is that something that is
> likely to happen, in the mid term or at all?

Sure, it's not likely at all to happen in the mid term, but with media
parameters you're not "breaking" anything either. If we want those
specifications to evolve, we need more examples and people interested in
those parameters. The AtomPub spec with its "type" parameter for Atom is a
good start but we need to go further.

---

From: Erik Wilde <[email-masked]>
Date: Mon, 09 May 2011 14:21:02 -0700

hello.

> good start but we need to go further.

like i said before, philosophically, i do agree that this would be 
cleaner and the better approach. but we would need to update the atom 
spec which, as i am reading it, does not allow for extensibility of 
media type parameters:

http://tools.ietf.org/html/rfc4287#section-7 says that the only allowed 
parameter is "charset". i think the way to go for all of this to be nice 
and clean would be to add a "content-type" parameter, which as a value 
would have a media type (the media type of the content embedded or 
linked to by the feed). this raises two questions:

- is that kind of parameter value even allowed in a media type? and if 
it is, how many media type parsers might possibly break because a media 
type would contain "two media types"?

- how realistic is it to update RFC 4287 in that way, which as i 
understand it would not be backwards compatible?

if that was a realistic way to go, even things such as 
http://www.w3.org/TR/html5/links.html#rel-alternate might need to be 
updated, so that the new features would be allowed/supported (and 
shouldn't HTML5 allow the optional charset parameter that is possible 
according to RFC 4287?).

any feedback on this would be greatly appreciated. kind regards,

dret.

---

From: Hadrien Gardeur <[email-masked]>
Date: Wed, 11 May 2011 14:51:13 +0200

>
> like i said before, philosophically, i do agree that this would be cleaner
> and the better approach. but we would need to update the atom spec which, as
> i am reading it, does not allow for extensibility of media type parameters:
>
> parameter is "charset". i think the way to go for all of this to be nice and
> clean would be to add a "content-type" parameter, which as a value would
> have a media type (the media type of the content embedded or linked to by
> the feed). this raises two questions:
>

There's also http://tools.ietf.org/html/rfc5023#section-12

> - is that kind of parameter value even allowed in a media type? and if it
> is, how many media type parsers might possibly break because a media type
> would contain "two media types"?
>

That's a valid point indeed.

> - how realistic is it to update RFC 4287 in that way, which as i understand
> it would not be backwards compatible?
>
> updated, so that the new features would be allowed/supported (and shouldn't
> HTML5 allow the optional charset parameter that is possible according to RFC
> 4287?).
>

I believe that what you're trying to deal with is actually a much more
generic problem than what you're describing.

If I'm distributing bitmap files, but for obvious reasons, I decide to use a
zip archive containing all these bitmap files together, how do I indicate in
Atom that this archive contains bitmap files ? We do not have an easy way to
express this kind of information in Atom, we simply point to the archive
URL, using the zip mimetype. Other containers (DRM files for example,
various audio & video containers) have a similar problem.

As far as I can tell, you're interested in using Atom as an
envelope/container, to distribute multiple representations of the same
information. In this case, what you're actually doing is not "content
negotiation", it is the exact same situation than with other containers.

Hadrien