Subject: Re: extension for update properties

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 6 Jul 2009 11:16:08 +0200

On Mon, Jun 29, 2009 at 01:51:05PM +0200,
 Erik Wilde <[email-masked]> wrote 
 a message of 25 lines which said:

> i am sure this has already been discussed at some point in time, but
> i cannot remember seeing it, so i am asking this here: it might be
> useful to give a feed publisher some simple methods of making
> statements about a feed's update properties, such as a timestamp, a
> frequency,

I don't know for Atom but RSS 1.0 has it, it may gives ideas:

http://web.resource.org/rss/1.0/modules/syndication/

Provides syndication hints to aggregators and others picking up this RDF Site Summary (RSS) feed regarding how often it is updated. For example, if you updated your file twice an hour, updatePeriod would be "hourly" and updateFrequency would be "2". The syndication module borrows from Ian Davis's Open Content Syndication (OCS) directory format. It supercedes the RSS 0.91 skipDay and skipHour elements.

updatePeriod
Describes the period over which the channel format is updated. Acceptable values are: hourly, daily, weekly, monthly, yearly. If omitted, daily is assumed.

updateFrequency
Used to describe the frequency of updates in relation to the update period. A positive integer indicates how many times in that period the channel is updated. For example, an updatePeriod of daily, and an updateFrequency of 2 indicates the channel format is updated twice daily. If omitted a value of 1 is assumed.

updateBase
Defines a base date to be used in concert with updatePeriod and updateFrequency to calculate the publishing schedule. The date format takes the form: yyyy-mm-ddThh:mm

---

From: James Abley <[email-masked]>
Date: Mon, 6 Jul 2009 12:41:32 +0100

>> statements about a feed's update properties, such as a timestamp, a

HTTP already provides Cache-Control: / Expires: to indicate how long a
client may like to wait before checking if the representation has been
updated. Would that meet your requirement?

Cheers,

James

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 6 Jul 2009 14:03:21 +0200

On Mon, Jul 06, 2009 at 12:41:32PM +0100,
 James Abley <[email-masked]> wrote 
 a message of 22 lines which said:

> HTTP already provides Cache-Control: / Expires: to indicate how long
> a client may like to wait before checking if the representation has
> been updated. Would that meet your requirement?

IMHO (I'm not the OP), no. Feed producers (who know the duration of
the feed) may not have a way to set HTTP headers. Also, HTTP headers
are not end-to-end, they are lost if, for instance, the feed is
transferred by non-HTTP means (such as rsync or scp). I would prefer a
solution inside the feed.

---

From: Erik Wilde <[email-masked]>
Date: Tue, 07 Jul 2009 15:30:19 -0700

> solution inside the feed.

+1. the update properties of a feed are a property of the feed, not of 
the protocol. these things are related, but keeping the properties as 
part of the feed format would make a feed document more self-contained, 
would allow feed publication in scenarios where HTTP headers cannot be 
easily set, and might also allow more sophisticated/specialized 
properties than HTTP's cache control.

thanks for the pointer to RSS 1.0's properties. i am not quite sure 
whether i like what has been done there, but now that i know that it 
exists in RSS 1.0, it would be really interesting to know how many feed 
providers and consumers actually use it, and why it is one of those 
properties that did exist in some RSS version, but did not make it into 
Atom...

cheers,

erik wilde   tel:+1-510-6432253 - fax:+1-510-6425814
        [email-masked]  -  http://dret.net/netdret
        UC Berkeley - School of Information (ISchool)

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 13 Jul 2009 13:02:21 +1000

The properties of an entity are the properties of an entity -- and  
this includes entity headers. Transferring the feed via rsync to  
another host means that the serving scenario is likely different, and  
therefore different metadata is appropriate. Meanwhile, duplicating  
the data in the feed introduces opportunities for misalignment and  
encourages people not to set the proper headers -- which means that  
caching intermediaries and clients (which have a much larger  
deployment footprint than any feed infrastructure) can't do the right  
thing.

Personally, I'm getting sick of the "can't set HTTP headers" scenario  
driving abuse of HTTP.

/me returns to cave

On 08/07/2009, at 8:30 AM, Erik Wilde wrote:

> +1. the update properties of a feed are a property of the feed, not  
> of the protocol. these things are related, but keeping the  
> properties as part of the feed format would make a feed document  
> more self-contained, would allow feed publication in scenarios where  
> HTTP headers cannot be easily set, and might also allow more  
> sophisticated/specialized properties than HTTP's cache control.
>
> exists in RSS 1.0, it would be really interesting to know how many  
> feed providers and consumers actually use it, and why it is one of  
> those properties that did exist in some RSS version, but did not  
> make it into Atom...
>
> cheers,
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Erik Wilde <[email-masked]>
Date: Sun, 12 Jul 2009 20:10:08 -0700

> Personally, I'm getting sick of the "can't set HTTP headers" scenario 

just for the record: i was not proposing to re-create HTTP headers as 
atom extension. i was suggesting that maybe some information about the 
update properties of a feed might be useful, and my guess is that these 
might go well beyond the properties expressible in HTTP. but i do agree 
that if all we want to do is what HTTP can do, then we should definitely 
stick with HTTP headers.

cheers,

dret.

---

From: James Abley <[email-masked]>
Date: Mon, 13 Jul 2009 08:58:18 +0100

2009/7/13 Erik Wilde <[email-masked]>

> just for the record: i was not proposing to re-create HTTP headers as atom
> extension. i was suggesting that maybe some information about the update
> properties of a feed might be useful, and my guess is that these might go
> well beyond the properties expressible in HTTP. but i do agree that if all
> we want to do is what HTTP can do, then we should definitely stick with HTTP
> headers.
>
> cheers,
>
> dret.

I was aware that there might be a different application protocol used for
transport; e.g. Jabber sending entries about, but I didn't want to be too
prescriptive when asking the OP for clarification. It might be worthwhile
discussing what information gets lost when HTTP is not involved.

The initial statement talked about clients polling a publisher, which lead
me to assume HTTP was being used. I'm with Mark in that people using the web
should *use* the web, and think about Last-Modified, If-Modified-Since and
Expires, or ETag, If-None-Match and Cache-Control. That's not
finger-pointing, but merely an indication of the types of services that I
would prefer to consume.

Cheers,

James

---

From: Martin Atkins <[email-masked]>
Date: Mon, 13 Jul 2009 09:49:18 -0700

> properties than HTTP's cache control.

If you rsync a feed from one host to another, does that guarantee that 
the new copy will be updated with the same frequency as the original? To 
pick another example, if I download a feed and put it on the hard drive 
of my laptop here, will it continue to update at the same frequency?

With these examples in mind, I'd argue it's actually wrong to put it in 
the feed itself, since the update frequency is a property of a 
particular published instance of a feed, not an "end-to-end" property 
that is preserved when you duplicate the feed elsewhere.

Of course, it may be useful to know where the "original" instance of a 
feed came from, which is why it makes sense to have link rel="self" in 
the feed rather than in the headers: that property doesn't change as 
copies are made by definition.

---

From: Keith Fahlgren <[email-masked]>
Date: Mon, 13 Jul 2009 09:49:00 -0700

Erik: I do believe that some feed consumers would like to understand the 
expected feed update properties as the feed publisher understands them.

On 7/13/09 12:58 AM, James Abley wrote:
> I'm with Mark in that people using the web should *use* the web, and
> think about Last-Modified, If-Modified-Since and Expires, or ETag,
> If-None-Match and Cache-Control.

While I expect many on this list will understand the relationship between these 
HTTP headers and the vocabulary for update properties Erik defined, it'd be 
helpful for someone to establish a mapping between these or point to a fairly 
non-technical introduction to these HTTP headers.

Erik: Are there update properties that a feed publisher might understand that 
are not covered by Expires/Cache-Control?

Thanks,
Keith

-- 
Keith Fahlgren
Publishing Technology Engineer
O'Reilly Media
[email-masked]

---

From: Erik Wilde <[email-masked]>
Date: Mon, 13 Jul 2009 10:18:43 -0700

Martin Atkins wrote:
>> part of the feed format would make a feed document more 
>> self-contained, would allow feed publication in scenarios where HTTP 
>> headers cannot be easily set, and might also allow more 

if you find it on your disk and the feed itself tells you what 
recommended update intervals are, you could set up a scheduled rsync 
that would be most appropriate for the feed.

> that is preserved when you duplicate the feed elsewhere.

actually, i should have been more precise in saying that update 
properties (at least the ones i want to be able to express) are a 
property of the collection (the abstract set of entries) underlying the 
feed. what i want to be able to say is that there are certain 
statistical properties about a collection's update properties, a little 
bit like arrival processes in queuing theory. you could get even more 
fancy in the context of feed queries, in which case a smart back-end 
could actually tell you that while the collection as a whole updates 
once every 3min on average, if you only get a filtered view of it, then 
this changes to once every 30min. in that case, the update property of 
the collection would change because of the "filtered view" of the 
collection, as expressed in a feed query.

cheers,

dret.