Subject: Spaces supports slash:comments. Result = Duplicates Galore!

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 14:48:07 -0400

Spaces.msn.com recently announced support for "slash:comments," an
element which shows how many comments an RSS item has associated with it.
	As Dare Obasanjo explains[1]:

    "Another cool RSS enhancement is that the number of comments on
    each post is now provided using the slash:comments elements. Now
    users of aggregators like RSS Bandit can track the comment counts on
    various posts on a space. I've been wanting that since last year."

	Of course, the side effect of this change is that any aggregator
that uses an MD5-like approach to detect changes will now think that an
entry has been updated every time a new comment is made. This may or may not
be what is desired by consumers of feeds... In any case, there are now
millions of blogs whose entries are changed every time anyone comments on
them. Should aggregators ignore changes that are limited to the
"slash:comments" element? If so, are there other elements that should be
ignored?
	Now, Spaces only publishes RSS feeds... However, if similar atom
extensions were to be defined, the problem would appear with Atom feeds as
well.

	bob wyman

[1]
http://spaces.msn.com/members/carnage4life/Blog/cns%211piiOwAp2SJRIfUfD95CnR
Lw%21430.entry

---

From: Walter Underwood <[email-masked]>
Date: Thu, 07 Apr 2005 12:10:18 -0700

One way to look at this is to define what parts are local content
as opposed to caches of remote, and base the Etag or other hash on
that.

I still think we should address caching in Atom 1.0. This would
have been part of that. Scaling is an essential thing for syndication,
and caching is the best known way to scale.

wunder

--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: Graham <[email-masked]>
Date: Thu, 7 Apr 2005 20:06:05 +0100

> "slash:comments" element? If so, are there other elements that should 

Not that I have any data, but I don't seriously believe any aggregator 
that uses the content hash approach would survive very long in the 
market place without being buried under user complaints. Most (the 
one's I know of) either use identifiers or failing that some subset of 
the elements. Shrook uses an adaptive subset that means that if one 
element is unreliable it uses the others.

Graham

---

From: Robert Sayre <[email-masked]>
Date: Thu, 07 Apr 2005 15:11:43 -0400

Graham wrote:
> market place without being buried under user complaints. Most (the one's 
> I know of) either use identifiers or failing that some subset of the 
> elements. Shrook uses an adaptive subset that means that if one element 
> is unreliable it uses the others.

Mozilla Thunderbird's approach will be similar when the relevant bugs 
are closed.

Robert Sayre

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 7 Apr 2005 15:23:10 -0400

Graham wrote:
> I don't seriously believe any aggregator that uses the content hash
> approach would survive very long in the market place without being
> buried under user complaints. Most (the one's I know of) either use
> identifiers or failing that some subset of the elements.
	The "identifier-based" approach works wonderfully for a single user
aggregator which has a "feed" focus as most personal aggregators do today.
However, as pointed out elsewhere, the identifier-based approaches don't
work in cross-feed duplicate detection because of the ease with which denial
of service attacks can be launched.
	If a subset of elements is to be used in duplicate detection, what
is that subset? Can/Should this subset be commonly known? It seems to me
that it is important enough to the atom-ecosphere that it might even make
sense to have it in the spec as an important interoperability note. i.e.
"Entries will be considered duplicates if...."

> Shrook uses an adaptive subset that means that if one element is
> unreliable it uses the others.
	Can you describe your algorithm? What do you consider "unreliable"
to mean?

		bob wyman

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 7 Apr 2005 22:49:13 +0200

* Bob Wyman <[email-masked]> [2005-04-07 21:30]:
> Can/Should this subset be commonly known? It seems to me that
> it is important enough to the atom-ecosphere that it might even
> make sense to have it in the spec as an important
> interoperability note. i.e. "Entries will be considered
> duplicates if...."

Yes, absolutely, implementors should be made aware of such
issues. I don’t know if it is in the scope of the Atom format to
specify normative rules for when to consider entries duplicates –
and more importantly, when not to –, though.

F.ex, entries maliciously published with someone else’s entry ID
will not actually constitute a DOS attack for consumers whose
aggregator maintains a history of previously seen versions of an
entry.

Other approaches may also form. Generally, practice will probably
lead implementors to pick among various approaches the one best
suited to their respective needs. I don’t believe we can forsee
all of these, so let’s not try to. The Atom format should specify
hooks (such as the atom:source subelement Thomas suggested,
possibly?) which would allow trust models or other coping
mechanisms to be implemented, but leave their precise nature to
the discreetion of implementors after bringing the potential
problems to attention.

Regards,
-- 
Aristotle

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 7 Apr 2005 23:12:17 +0200

* A. Pagaltzis <[email-masked]> [2005-04-07 22:55]:
> F.ex, entries maliciously published with someone else’s entry
> ID will not actually constitute a DOS attack for consumers
> whose aggregator maintains a history of previously seen
> versions of an entry.

Sorry, wrong thread. I have been hesitating to post (after my
first posts to the list missed the topic as well) because I’m
constantly tired lately and my concentration is shot. Ugh.

As concerns the thread I was addressing, I still stand by my
argument about the scope of the Atom format, though.

Embarrassed,
-- 
Aristotle

---

From: David Powell <[email-masked]>
Date: Thu, 7 Apr 2005 22:20:55 +0100

> and caching is the best known way to scale.

Me too.

I'd like to see: atom:etag or atom:instanceId; or alternatively
atom:modified.

Having 1 publisher say whether an entry has changed seems more
reliable than having thousands of different clients guess.

-- 
Dave

---

From: Eric Scheid <[email-masked]>
Date: Fri, 08 Apr 2005 10:07:18 +1000

On 8/4/05 6:49 AM, "A. Pagaltzis" <[email-masked]> wrote:

> I don¹t believe we can forsee
> all of these, so let¹s not try to. The Atom format should specify

+1

I don't think we can boil all the spammers out of the syndication ocean in
the 1.0 spec.

e.