Subject: One reason we have duplicates entries is that we have duplicate feeds...

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 13:34:01 -0400

As many are aware, the Tim Bray[1] and others have recently been
railing on the subject of duplicate entries being delivered by aggregation
services and/or search engines like PubSub, Technorati, Feedster, etc. As
unexpected as this may sound, I am quite confident that when Atom V1.0 is
released and starts to be used, the problem of duplicates will get worse --
unless something (probably quite simple) is done. A major part of the
problem with duplicate entries is that people publish duplicate feeds. The
introduction of Atom V1.0 will result in even more duplicate feeds being
produced... This is not because of "bugs" in the producers of synthetic
feeds.

	A major cause of duplicates in at least some of the existing
services is the fact that bloggers insist on engaging in the apparently
illogical and wasteful practice of publish multiple versions of their feeds
and thus duplicates of their entries or items. 
	Often blogs will offer at least one flavor of RSS as well as Atom.
Those that don't offer Atom will often offer at least two flavors of RSS.
Some sites provide as many as six or seven distinct versions of their feeds
in addition to a sometimes large number of "category" feeds (each of which
is offered in multiple encodings) that contain nothing but duplicates of
entries/items found in some primary feed. Unfortunately, while RSS and Atom
both contain mechanisms to identify their "HTML" alternates, there isn't any
clear mechanism available to discover alternate feeds. In theory, if such
means existed, automated processors of large numbers of feeds could
selectively ignore duplicate feeds...
	The process of recognizing duplicates within a single feed is
relatively straight forward and is widely implemented not only by news
aggregator clients but also by the aggregating services. (I believe the most
common method is to generate an MD5 hash for each entry/item and doing the
obvious processing.) Problems arise, however, when one attempts to recognize
duplicates across feeds which may have different information-carrying
capacities (i.e. RSS, RDF and Atom are different). Recognizing duplicate
feeds is made particularly difficult if only because of the practice that
many sites have of producing "category" feeds that are organized to present
different selections of the posts for a single blog. The fact that an entry
has been published in one or another "category feed" is, in itself,
information that may be useful to some readers. In any case, there is no way
without using human, meat-machine based inspection to determine which of
many feeds is the "primary" feed...
	It should also be noted that if an aggregation service (such as
PubSub) allows users to subscribe to content based on the "source feed" of
the entry, the aggregation service is forced to read all the feeds that it
can find -- since any findable feed is something that some user might want
subscribe to... This issue becomes particularly troubling in contexts like
FeedMesh where aggregators are expected to share their knowledge of feed
updates with each other. Issues might arise if some aggregators are
selectively ignoring feeds or processing the feeds but not publishing their
results to the FeedMesh.
	Even if the aggregator could establish that two feeds carried
identical content, it would be difficult to stop processing one of the feeds
since someone might wish to subscribe to it... Admittedly, one can imagine
that an aggregator might install a rule that says: "Any subscription to feed
A should be read as a subscription to feed B." However, that would require
the aggregator to assume that simply because two feeds had substantially
identical content in the past, that they would have identical content in the
future. This is not an easy assumption to make without some explicit
indication of feed equivalence from the producer of the feed. We could go
further and require that the aggregator constantly "check" its assumptions
and if the two feeds diverge then uninstall the rule. This is getting
complex... Too complex.
	Tim suggests that aggregators should be able to rely simply on
atom:id to detect duplicates. However, as has often been pointed out,
applying this rule in an intermediary like PubSub would simply make PubSub a
marvelously efficient tool for denial of service attacks. I.e. if I didn't
like something you published, I would simply publish something in my blog
that had the same atom:id as something you had published. PubSub and other
synthetic feed producers would then flush your post from the system and
replace it with my post... Not good -- and not avoidable given the current
loose rules for defining instances of atom:id.
	This issue of duplicate detection has come up many times in this
forum and has never been dealt with seriously as far as I can tell. The
current Atom draft continues to fail to address the issue. Simply
complaining isn't getting us anywhere even as the introduction of Atom V1.0
makes the problem worse.
	One thing that we could do in the short term to reduce the number of
duplicate feeds is to define metadata that would allow feed publishers to
describe the relationships between feeds and describe the content of
different feeds. If, when reading a feed, I could be informed that this feed
was a duplicate of another "preferred" feed, I could switch to the preferred
feed and stop reading the duplicate. I could use this knowledge to map
subscriptions to the duplicate feeds to their equivalents. This would be
wonderful in the transition from Atom .03 to Atom V1.0... (i.e. if the old
format feed could point to the newer version, we'd all eventually stop
reading the old format feed.)
	For instance:

<link rel="preferred-feed" type="application/atom+xml"
href="http://example.org/feed.xml"/>
<link rel="subset-of" type="application/atom+xml"
href="http://example.org/fullfeed.xml"/>

	If these link rel types are acceptable to people, we should probably
define equivelant syntax to be inserted into old Atom feeds to allow the
transition to the new atom as well as define extensions to the various
flavors of RSS. If we can implement this minor extension, we may see many
fewer duplicate feeds being consumed and thus fewer duplicate entries.
	Comments?

	bob wyman

[1] http://www.tbray.org/ongoing/When/200x/2005/04/03/Atom-Now

---

From: Robert Sayre <[email-masked]>
Date: Thu, 07 Apr 2005 13:44:25 -0400

Bob Wyman wrote:
> Unfortunately, while RSS and Atom
> clear mechanism available to discover alternate feeds.

Without responding to the rest of the message, I'll note that this 
statement is somewhat inaccurate wrt Atom. You can point to an alternate 
feed like this

<link rel="alternate" type="some/feed" href="..." />

Of course, you can't have two alternates with the same media type...

Robert Sayre

---

From: David Czarnecki <[email-masked]>
Date: Thu, 7 Apr 2005 13:28:11 -0400

This seems similar to what exists in Really Simply Discoverability in the
individual <api.../> element [1]. 

-----
# <api> has 4 required attributes.

    * "preferred" is a boolean and takes either "true" or "false". The point is
to allow weblog software to list all the APIs supported, but choose which one
to "recommend" to the client software. Only one API should set as prefered.
-----

This couldn't be another/new attribute on the site feed autodiscovery links? As
opposed to the link rel types proposed below.

-David

[1] - http://archipelago.phrasewise.com/rsd#ODoxNjozMSBBTQdbdb

Quoting Bob Wyman <[email-masked]>:

<snip>

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 14:02:17 -0400

Robert Sayre wrote:
> You can point to an alternate feed like this
	Yes, you can point to an alternate. However, all you are doing at
that point is establishing equivalence between the two. The "alternate"
mechanism doesn't provide you with a way to say which feed is preferred. It
also does not allow you to establish that one feed is a sub-set of another
(as in the case of "category" feeds.).
	Establishing equivalence only addresses a part of the problem. 

	If you provide an old-format Atom feed today, will you continue to
do so after Atom V1.0 is released? If not, how will automated readers or
crawlers discover that it is no longer necessary to seek your old feed or
that you have a new feed that replaces it? If you had supported an RSS feed
in the past but now decided to switch to an Atom feed, how would the
aggregators discover that this was your intention without causing a break in
their ability to process your posts?

	bob wyman

-----Original Message-----
From: Robert Sayre [mailto:[email-masked]] 
Sent: Thursday, April 07, 2005 1:44 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: One reason we have duplicates entries is that we have duplicate
feeds...

Robert Sayre

---

From: Norman Walsh <[email-masked]>
Date: Thu, 07 Apr 2005 14:12:11 -0400

/ "Bob Wyman" <[email-masked]> was heard to say:
| 	A major cause of duplicates in at least some of the existing
| services is the fact that bloggers insist on engaging in the apparently
| illogical and wasteful practice of publish multiple versions of their feeds
| and thus duplicates of their entries or items. 

Illogical and wasteful? I've been doing it for a while as a benefit to
my readers. If you only care about DocBook, you can subscribe to

  http://norman.walsh.name/atom/docbook.xml

If you only care about geocaching, you can subscribe to

  http://norman.walsh.name/atom/geocaching.xml

If you want to see all the blather I write, you can subscribe to

  http://norman.walsh.name/atom/whatsnew.xml

Perhaps I shouldn't have done this, and I suppose I could stop, but I
have. I had hoped, not having considered the DOS potential, that using
the same atom:id in each entry would allow duplicates to be
suppressed.

I'm disappointed to learn that I've been niave.

| 	One thing that we could do in the short term to reduce the number of
| duplicate feeds is to define metadata that would allow feed publishers to
| describe the relationships between feeds and describe the content of
| different feeds. If, when reading a feed, I could be informed that this feed
| was a duplicate of another "preferred" feed, I could switch to the preferred
| feed and stop reading the duplicate. I could use this knowledge to map
| subscriptions to the duplicate feeds to their equivalents. This would be
| wonderful in the transition from Atom .03 to Atom V1.0... (i.e. if the old
| format feed could point to the newer version, we'd all eventually stop
| reading the old format feed.)

I'd be happy to generate this metadata.

| 	If these link rel types are acceptable to people, we should probably
| define equivelant syntax to be inserted into old Atom feeds to allow the
| transition to the new atom as well as define extensions to the various
| flavors of RSS. If we can implement this minor extension, we may see many
| fewer duplicate feeds being consumed and thus fewer duplicate entries.
| 	Comments?

+1

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <[email-masked]> | CNN is one of the participants in the
http://nwalsh.com/            | war. I have a fantasy where Ted Turner
                              | is elected president but refuses
                              | because he doesn't want to give up
                              | power.--Arthur C. Clark

---

From: Robert Sayre <[email-masked]>
Date: Thu, 07 Apr 2005 14:16:35 -0400

> 	Establishing equivalence only addresses a part of the problem. 

Fully agree. I just wanted to point out that a part of the problem is 
more solved than your post indicated.

> their ability to process your posts?

I've encountered this situation in the past. Some aggregators continue 
to poll both locations, though. Not sure why. If they're doing it on 
purpose, I'm not sure they'd pay attention to feed links either.

---------------------------------------------
GET /index.xml HTTP/1.1
Host: www.franklinmint.fm
...

HTTP/1.x 301 Moved Permanently
Date: Thu, 07 Apr 2005 18:10:30 GMT
Location: http://franklinmint.fm/atom.atom
...
---------------------------------------------

---------------------------------------------
GET /index.rdf HTTP/1.1
Host: www.franklinmint.fm
...

HTTP/1.x 301 Moved Permanently
Date: Thu, 07 Apr 2005 18:10:36 GMT
Location: http://franklinmint.fm/atom.atom
...
---------------------------------------------

Robert Sayre

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 14:29:28 -0400

> more solved than your post indicated.
	My apologies if I made the situation sound more dire than it is.
However, even with the "alternate" we've still got a pretty big problem.

> HTTP/1.x 301 Moved Permanently
	Another partial solution... Useful in some situations, but relying
on it prevents anticipating change and thus may result in some data loss.
Also, a change like this requires access to the web server configuration in
many installations and hosted users can't always get such access. 

> [When there are multiple feeds] Some aggregators continue to poll both
> locations, though. Not sure why.
> If they're doing it on purpose, I'm not sure they'd pay attention to
> feed links either.
	The automated aggregators are stuck polling all feeds in many cases
because:
	1. They can't determine the equivelance between feeds without
explicit statements from the publisher.
	2. Even if equivalence is known, we can't know which feed is
"authoritative" or if one feed will be deleted in favor of the other. Thus,
we've got to read both to ensure that we are reading which ever one is more
long-lived.
	3. We have to read all feeds since users like to be able to
subscribe to feeds using file names or masks on file names... If we don't
read all feeds, we will incorrectly fail to deliver to the user content that
we have available to us.

> I'm not sure they'd pay attention to feed links either
	I *guarantee* you that if we had an accepted way to address this in
feeds, PubSub *would* pay careful attention to it. 

		bob wyman

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 7 Apr 2005 20:52:07 +0200

* Bob Wyman <[email-masked]> [2005-04-07 19:45]:
> I.e. if I didn't like something you published, I would simply
> publish something in my blog that had the same atom:id as
> something you had published. PubSub and other synthetic feed
> producers would then flush your post from the system and
> replace it with my post...
> 
> [...]
> 
> If, when reading a feed, I could be informed that this feed was
> a duplicate of another "preferred" feed, I could switch to the
> preferred feed and stop reading the duplicate. I could use this
> knowledge to map subscriptions to the duplicate feeds to their
> equivalents.

Together, to me, these suggest that atom:entry/atom:id should be
conisdered unique only with respect to the originating feed, or,
per the proposal, with with respect to the originating feed and
all of the feeds it points to. Correct?

Under the assumption that my understanding is correct, then the
proposition is insufficient. It works for aggregating/
republishing services that consume feeds from original producers,
such as PubSub. But it breaks down for the aggregate feeds
published by third parties. Imagine someone using Bloglines
subscribes to PlanetFoo and PlanetBar, both of which republish
content from feed Baz. Now how does Bloglines assure that the
duplicate Baz entries it receives from PlanetFoo and PlanetBar,
neither of which are its original producers but (in absence of
malicious intent) reproduce content from the same source, do
indeed originate from the same source?

If look at more convoluted examples, it fast turns into web of
trust territory...

Regards,
-- 
Aristotle

---

From: Thomas Broyer <[email-masked]>
Date: Thu, 07 Apr 2005 20:51:56 +0200

Bob Wyman wrote:
> mechanism doesn't provide you with a way to say which feed is preferred.

Well, if you publish Atom 1.0, RSS 1.0 and RSS 2.0 feeds, they should 
all be equivalent. Which is "preferred" is on the end-user's hands.

Say you publish an interview as text/plain, text/html and 
application/msword. Each end-user will choose his/her preferred version. 
What you can do to indicate your preferred version (otherwise putting up 
some web page that links to each version and tell the end-user which one 
you prefer) is to put them all at the same location (URI) and use 
content negotiation, associating q-values on the server side; but client 
provided q-values and accepted media types will be used to serve the 
*client's* preferred version anyway.

 > It also does not allow you to establish that one feed is a sub-set of 
another
> (as in the case of "category" feeds.).

Well, actually yes: use atom:source in your entries.
Either you consider "category" feeds to be the "primary feeds" and use 
atom:source in the "whole site" feed (and you'll have to choose a 
"primary category" as well for entries belonging to two or more 
categories); or you treat the "whole site" feed as the primary feed and 
put atom:source elements in your "category" feeds.

However, I'm +1 on defining some new kind of link@rel. They don't have 
to be put in the spec, they could only be some "generally used values", 
defined and encouraged by the Atom WG without begin in the Atom 1.0 spec 
(and maybe be added to a future Atom version)

> their ability to process your posts?

If you keep the same URI, I think there won't be much problem, many 
readers/aggregators handling several syndication formats.
You might eventually continue to serve RSS as well and use content 
negotiation for a while, waiting for the Atom 1.0 format to be widely used.

If you change the URI (say you used to have 
http://example.net/myfeed.rss and now move to 
http://example.net/myfeed.atom), send an HTTP "301 Moved Permanently" 
status to redirect clients to the new URI.

Anyway, if you stop delivering some content in some media type, you 
can't expect not to "cause a break in their ability to process your 
[content]".

So in a few words: +1 for what you propose, not necessarily to be added 
to the Atom spec, but the problem you pointed out is not /that/ grave.

-- 
Thomas Broyer

---

From: Tim Bray <[email-masked]>
Date: Thu, 07 Apr 2005 11:50:38 -0700

> loose rules for defining instances of atom:id.

Yes.  Mea culpa; somehow I'd missed this.  Would the following work:

1. A new feed-level element <atom:alt-uri-prefix>, any number allowed.  
E.g.

<feed>
   <link>http://www.tbray.org/ongoing/</link>
   <alt-uri-prefix>http://www.tbray.org/</alt-uri-prefix>
   <alt-uri-prefix>http://tbray.org/</alt-uri-prefix>
   <alt-uri-prefix>http://www.textuality.com</alt-uri-prefix

It says: an atom:entry can't be a duplicate of an atom:entry in this 
feed unless it comes from a feed whose URI begins with one of these.  
In conjunction with atom:id and atom:updated, it would solve most of 
PubSub's problem, but note that it only solves the problem for one 
level of aggregation.  Which I suspect is a useful 80/20 point.

By the way... bear in mind that virtually all duplicates are coming via 
things like Technorati and PubSub, so if we can figure out a fix, the 
number of players who need to implement it is small, and the motivation 
for them to do it is high. -Tim

---

From: Graham <[email-masked]>
Date: Thu, 7 Apr 2005 19:57:42 +0100

I don't think this is within the scope of Atom or any other spec. If 
you want to differentiate your others, why not invest some time in a 
decent  duplicate detection algorithm like Google did? This is your 
problem.

Graham

---

From: Thomas Broyer <[email-masked]>
Date: Thu, 07 Apr 2005 21:15:45 +0200

>   <alt-uri-prefix>http://www.textuality.com</alt-uri-prefix

Although this doesn't solve the DOS potential, I'd rather go for a new 
atom:source child element indicating the source feed's Id or URI. When 
PlanetBar and PlanetFoo republish a feed from Baz, they should "move" 
the entry metadata into an atom:source element. They'll then also add 
that new element (atom:feedid ? doh, yet another two-word name). When 
PubSub republish the entry, it preserves the atom:source content as-is, 
preserving the source feed's Id at the same time.

-- 
Thomas Broyer

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 15:36:53 -0400

Thomas Broyer wrote:
> When PlanetBar and PlanetFoo republish a feed from Baz, they should
> "move" the entry metadata into an atom:source element.
	This is actually what we do at PubSub today. You may not be aware
that the atom:source element is modeled on the ps:source-feed element that
we have been publishing in our Atom feeds since last June. 
	Into every entry we publish at PubSub, we put in something like the
following:

<ps:source-feed>
  <title><![CDATA[Scoble's Link Blog]]></title>
  <link rel='alternate' type='text/html'
        href='http://scobleizer.com/linkblog'/>
  <link rel='alternate' type='text/xml'
       href='http://scobleizer.com/linkblog/feed/'/>
</ps:source-feed>

	The text/html alternate and the title are taken from the feed
element of the source feed. The text/xml link indicates the URI of the feed
we read (this isn't normally in the Feed metadata...). (Yes, I know it
probably should be "application/xml" and would ideally be specific as to
type since we know it is RSS or Atom, etc. But, that's not the point of this
discussion.).

		bob wyman

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 07 Apr 2005 20:43:35 +0100

Thomas Broyer wrote:
> 
> Tim Bray wrote:
> 
>> 1. A new feed-level element <atom:alt-uri-prefix>, any number 
>> allowed.  E.g.
>>
>> <feed>
> the entry metadata into an atom:source element. 

One step further: how does that work for PlanetPlanet?

cheers
Bill

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 16:17:37 -0400

Tim Bray wrote:
>Would the following work:
	I don't like this. Hopefully, I can explain why. The reason is a bit
subtle...
	Tim's proposal relies on a feed "empowering" one or more other
feeds.  He allows a feed to say: "Any of these other feeds are allowed to
override me." 
	The proposal I made relies on a feed making statements only about
itself. In my proposal, a feed can only say: "I contain copies of these
other feeds. I am a secondary feed." 
	Personally, I think we're safer if we only allow feeds to speak
about themselves. The moment we start down the road of granting rights,
we're heading towards quicksand.
	I should also say that I like the idea of a feed subordinating
itself to another in that we would inevitably get from that the idea of a
"primary" feed. i.e. The primary is that feed which defers to no other feed.
	A side benefit of having "primary feeds" would be that people would
be more likely to do things like include full "category" data in the entries
they publish rather than publishing entries into specialized feeds as the
way to indicate "category." 

		bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 16:32:24 -0400

A. Pagaltzis wrote:
> But it breaks down for the aggregate feeds published by third parties.
	You are correct -- with one caveat. If entries are signed, which
Atom supports, you have a mechanism that allows you to trust statements that
copies of entries make about their source feeds. This is precisely why we at
PubSub so strongly supported the introduction of digital signatures into
Atom and why we consider this to be one of the really significant advantages
of Atom over existing RSS definitions. (Yes, I realize that no one signs
entries. But, that is a problem to be dealt with later...)
	If an entry is not signed, or if you don't have the means to
validate the signature, then you are stuck simply trusting the third parties
from whom you receive copies of entries. Yes, we've entered into "web of
trust territory." This explains why I have in the past argued that synthetic
or aggregate feeds should be explicitly tagged as such or that we should
*require* that source data be inserted into all copies of entries. (I really
do not like the "May" and "Should" words concerning atom:source in the
current draft.)
	For a service like PubSub's, what we'll probably do is consider any
entry copies that we discover to be somewhat equivalent to "pings." i.e. if
we see something with a "foreign" source, we'll probably want to check that
source to validate the copied entry before we republish it. (Note: I'm not
sure what we'll do if the entry has been purged from the source-feed...
Perhaps, we could flag it as "questionable"???) I don't however, expect
personal aggregators to do this and I expect that those who take feeds from
us will learn that they can trust us to properly validate things. Thus, they
won't need to be as rigorous as we must be.
	In the absence of signatures, you have no choice but to trust your
intermediaries. There is no design that can change that. We're going to be
working hard to make sure you can trust PubSub.

		bob wyman

---

From: Thomas Broyer <[email-masked]>
Date: Thu, 07 Apr 2005 23:08:01 +0200

> One step further: how does that work for PlanetPlanet?

As I told in the phrase you stripped out?

>> They'll then also add that new element (atom:feedid ? doh, yet another
 >> two-word name) When PubSub republish the entry, it preserves the
 >> atom:source content as-is, preserving the source feed's Id at the
 >> same time.

-- 
Thomas Broyer

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 7 Apr 2005 23:20:37 +0200

* Bob Wyman <[email-masked]> [2005-04-07 22:40]:
> A. Pagaltzis wrote:
> > But it breaks down for the aggregate feeds published by third
> > parties. If look at more convoluted examples, it fast turns
> > into web of trust territory...
> You are correct -- with one caveat. If entries are signed,

Yes, of course.

> This explains why I have in the past argued that synthetic or
> aggregate feeds should be explicitly tagged as such or that we
> should *require* that source data be inserted into all copies
> of entries.

And it is a good argument to make.

> i.e. if we see something with a "foreign" source, we'll
> probably want to check that source to validate the copied entry
> before we republish it.

Yes, that’s the only sane approach I can think of, as well.

> (Note: I'm not sure what we'll do if the entry has been purged
> from the source-feed... Perhaps, we could flag it as
> "questionable"???)

Certainly questionable, unless there are indicators that the
foreign source is trustworthy. After all – assuming a reasonably
short timespan between each party polling the others –, if you
can’t find the updated entry in the source feed, how did the
foreign source get it?

Further, I agree with what you wrote in
<[email-masked]>:

> about themselves. The moment we start down the road of granting
> rights, we're heading towards quicksand.

What I don’t know though, is whether this problem can be solved
in entirety within the scope of the Atom format spec. It is
probably safer and wiser for the spec to provide ways for
republishers to make each others’ lives easier (cf Thomas’
atom:source suggestion), but not to dictate how they should do
their jobs.

Regards,
-- 
Aristotle

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 07 Apr 2005 23:12:49 +0100

Thomas Broyer wrote:
> Bill de hÓra wrote:
> 
>> Thomas Broyer wrote:
>>
>>> Although this doesn't solve the DOS potential, I'd rather go for a 
>>> new atom:source child element indicating the source feed's Id or URI. 

And will I be able to reverse engineer the order the atom:source 
annotations were added? I want to determine if this operation is robust 
and/or reversable beyond a single aggregation step.

cheers
Bill

---

From: Thomas Broyer <[email-masked]>
Date: Fri, 08 Apr 2005 00:24:18 +0200

Bill de hÓra wrote:
>>>> new atom:source child element indicating the source feed's Id or 
>>>> URI. When PlanetBar and PlanetFoo republish a feed from Baz, they 
>>>> should "move" the entry metadata into an atom:source element. 
>>>
>>>
>>>
>>>

Well, depends what you mean by "reverse engineer" and "reversable", I'm 
not sure to understand...

IMHO, only the first re-publisher changes the entry to move the metadata 
from atom:entry to atom:source (and adds the atom:feedId or whatever its 
name). If an entry already has an atom:source child, you just republish 
it as-is. If you move the atom:source children back into the atom:entry 
element (and strip the atom:feedId), you should (and it is a should 
since the use of atom:source is currently a should in the spec) have the 
same entry as in the "primary" feed. Additionally, the content of the 
atom:source/atom:feedId tells you the URI of that "primary" feed at the 
time the entry was republished.

Sorry if I didn't answer you question as I might have misinterpreted it...

-- 
Thomas Broyer

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 18:39:22 -0400

Thomas Broyer wrote in response to Bill de hÓra:
> If an entry already has an atom:source child, you just republish
> it as-is. [And, if it doesn't have an atom:source, you insert one.]
	This is what was agreed, I believe, when the issue was discussed on
the list. To do anything more would be addressing the "path" or "provenance"
issue (i.e. answering "How did this get here?"). 
	The source element is only intended to provide that data which would
have been available to a reader if the entry had been found in its source
feed. Any other data that might be interesting (and there is a great deal of
it) is the problem of some other element to handle.

		bob wyman

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 08 Apr 2005 00:48:26 +0100

> it) is the problem of some other element to handle.

Ok. The format spec ought to document what is expected to happen 
downstream in the event of repeated aggregations.

cheers
Bill

---

From: Eric Scheid <[email-masked]>
Date: Fri, 08 Apr 2005 10:00:13 +1000

> way to indicate "category."

publishers don't do that to indicate category, they do that to minimize
bandwidth (both bits on the wire, and bits on the brain).

e.

---

From: Eric Scheid <[email-masked]>
Date: Fri, 08 Apr 2005 10:03:18 +1000

> other feeds. I am a secondary feed."

How does this prevent DOS attacks? If I could insert entries with faked
<atom:id>, could I not also insert entries with faked <atom:source>, or any
other identification meta-data?

If my entry has a more recent atom:modified, then it would replace any
earlier entries you've seen with [whatever] metadata, right?

e.

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 22:12:35 -0400

Eric Scheid wrote:
>> itself. In my proposal, a feed can only say: "I contain copies of
>> these other feeds. I am a secondary feed."
> <atom:id>, could I not also insert entries with faked <atom:source>,
> or any other identification meta-data?
	Sure, you could create entries that had faked atom:id's but well
designed readers would only considered your entries to be duplicates if they
appeared in feeds that the "feed under attack" had explicitly declared
itself to be secondary to. Entries with duplicate ids that appeared anywhere
else in the blogosphere would not be considered duplicates. This severely
constrains the opportunity for DOS attacks, substitutions, etc. Readers must
assume that the world is filled with entries that all use the same atom:ids.
The task for the reader is to figure out, among all the entries that use the
same atom:id, which are actually duplicates or legitimate replacements of
each other.
	The essential thing to understand here is that while creators of
atom:ids should do what the spec says and create atom:ids that are globally
unique, readers of atom:ids simply cannot assume that atom:ids are, in fact,
globally unique. (No repetition of MUST's or SHOULD's in the spec will
change this truth.) Without any additional information, a reader can only
just barely assume that an atom:id is unique within a single feed and even
then, only in those entries of the feed that do not contain atom:source
elements indicating that the entries have been copied from a foreign feed.
If the community accepts this idea of "primary" and "secondary" or
"authoritative" and "non-authoritative" feeds and entries, it should be
obvious that an entry which is contained in a feed, yet has an atom:source
indicating a foreign feed, should *always* be treated as non-authoritative
(unless, of course, the feed was published by someone you trust or the entry
has a digital signature that can be associated with the entry's source feed
-- these mechanisms aren't defined yet).
	Please remember that what I'm talking about here concerns the
processing model for Atom feeds or the semantics of the system composed of
real-world atom feeds and real-world atom processors. To date, most of our
discussions have been simply about the syntax (atom-syntax...) for feeds and
entries. It is great that the atom-syntax requires uniqueness in ids. You
just can't expect processors to be terribly influenced by the fact that the
uniqueness requirement exists.

	bob wyman

---

From: "Thomas Broyer" <[email-masked]>
Date: Fri, 8 Apr 2005 15:14:35 +0200 (CEST)

Doh! Seems I was very tired yesterday night...

Thomas Broyer wrote:
> [...] atom:source child element indicating the source feed's Id or URI
[...]

This already exists with atom:id and atom:link rel="self".

However, I'd go for a MUST or at least a SHOULD on the atom:source
presence in republished entries, along with a MUST in the metadata from
the source feed that would be copied into the atom:source element: if you
use atom:source, then you MUST copy all the Atom-defined feed metadata
into the atom:source element, and you SHOULD (or MAY ?) copy extension
metadata as well.

> [...] "move" the entry metadata into an atom:source element. [ ...]

...move the *feed* metadata into an atom:source element...

-- 
Thomas Broyer