Subject: PubSub CAN NOT support Atom with existing "no duplicate id" constraint

From: "Bob Wyman" <[email-masked]>
Date: Wed, 27 Apr 2005 11:48:00 -0400

I know that nobody seems to like this issue. However, I have explained on
numerous occasions that the existing prohibition against duplicate ids in a
feed simply cannot be supported by PubSub or any other feed aggregator. The
problem is, once again, that prohibiting duplicate ids provides an easy to
use attack vector for those wishing to effectively "erase" entries written
by another author. (i.e. by publishing an entry with an id identical to one
published earlier, one can force the earlier entry to be flushed from Atom
feeds.) 

If synthetic feed generators such as PubSub implement the current
prohibition against duplicate ids, they will simply become open channels for
attack. We'll soon see anti-abortionists publishing entries with ids
identical to those of entries whose content they disapprove of. We'll see
ex-boyfriends attack critical entries written by girls whom they displeased
on Saturday night. We'll see skinheads "flush" from the system any entries
that might have been written by jews and blacks. This is not good and I will
not allow the system I build to be used in this manner.

Graham Park has proposed that we loosen the existing language to permit
duplicate ids in the case where the entries have atom:source elements which
identify different URI's in "self" links. I support this compromise and
believe it should be supported by the WG and incorporated into the Atom
Draft.

I don't believe the issue here is one of taste and I don't see any
responsible option other then modifying the existing constraint. No one has
been able to provide any argument which explains why the attack vector I
describe is *NOT* a problem. If the WG refuses to modify this constraint,
then it should at least feel compelled to include text in the security
considerations section that clearly describes the attack and lets people
know that no atom entry is "safe" from trivial attacks as a result of this
constraint. If the WG refuses to modify this constraint, then they should
expect that responsible feed aggregators and generators of synthetic feeds
will simply be forced to deny support for Atom as specified.

If anyone has a clear argument explaining why the attack is not possible or
not a problem, they should make it. Alternatively, if you have a better
method then that proposed by Graham for defending against the attack, please
describe it. Otherwise, Graham's compromise should be accepted and the
specification revised.

bob wyman

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 27 Apr 2005 17:05:53 +0100

Bob Wyman wrote:
> I know that nobody seems to like this issue… However, I have explained 
> on numerous occasions that the existing prohibition against duplicate 
> ids in a feed simply cannot be supported by PubSub or any other feed 
> aggregator. The problem is, once again, that prohibiting duplicate ids 
> provides an easy to use attack vector for those wishing to effectively 
> “erase” entries written by another author. (i.e. by publishing an entry 
> with an id identical to one published earlier, one can force the earlier 
> entry to be flushed from Atom feeds.)
> 
> [...] 
> duplicate ids in the case where the entries have atom:source elements 
> which identify different URI’s in “self” links. I support this 
> compromise and believe it should be supported by the WG and incorporated 
> into the Atom Draft.

Bob,

What will prevent people overwriting the atom:source[@rel='self'] links 
as well as the id?

cheers
Bill

---

From: Sam Ruby <[email-masked]>
Date: Wed, 27 Apr 2005 12:05:11 -0400

Bob Wyman wrote:
> prohibition against duplicate ids, they will simply become open channels 
> for attack. We’ll soon see anti-abortionists publishing entries with ids 
> identical to those of entries whose content they disapprove of. We’ll 
> see ex-boyfriends attack critical entries written by girls whom they 
> displeased on Saturday night. We’ll see skinheads “flush” from the 
> system any entries that might have been written by jews and blacks. This 
> is not good and I will not allow the system I build to be used in this 
> manner.
> 
> I don’t believe the issue here is one of taste and I don’t see any 
> responsible option other then modifying the existing constraint. No one 
> has been able to provide any argument which explains why the attack 
> vector I describe is **NOT** a problem. If the WG refuses to modify this 
> constraint, then it should at least feel compelled to include text in 
> the security considerations section that clearly describes the attack 
> and lets people know that no atom entry is “safe” from trivial attacks 
> as a result of this constraint. If the WG refuses to modify this 
> constraint, then they should expect that responsible feed aggregators 
> and generators of synthetic feeds will simply be forced to deny support 
> for Atom as specified.
> 
> If anyone has a clear argument explaining why the attack is not possible 
> or not a problem, they should make it. Alternatively, if you have a 
> better method then that proposed by Graham for defending against the 
> attack, please describe it. Otherwise, Graham’s compromise should be 
> accepted and the specification revised.

It would help if a Pace could be written.

IIRC, the objections to PaceRepeatIdInDocument centered around the 
request to support multiple versions of a resource - a concern that does 
not apply to Graham's proposed compromise.

- Sam Ruby

---

From: Robert Sayre <[email-masked]>
Date: Wed, 27 Apr 2005 12:11:05 -0400

> not apply to Graham's proposed compromise.

Yes, Graham's proposal seems like a good idea to me as well.

Robert Sayre

---

From: Henry Story <[email-masked]>
Date: Wed, 27 Apr 2005 18:09:11 +0200

I support this.

I think we should vote on the two solutions:

	A- the compromise solution you propose
     B- the simple solution which simply allows entries with duplicate 
ids
     C- No change

My preference is clearly B then A, and certainly not C.

As far as I am concerned there is no interop purpose for C (the current 
limitation). Therefore
it should be dropped.

Henry

On 27 Apr 2005, at 17:48, Bob Wyman wrote:

> “erase” entries written by another author. (i.e. by publishing an 
> entry with an id identical to one published earlier, one can force the 
> earlier entry to be flushed from Atom feeds.)
> prohibition against duplicate ids, they will simply become open 
> channels for attack. We’ll soon see anti-abortionists publishing 
> entries with ids identical to those of entries whose content they 
> disapprove of. We’ll see ex-boyfriends attack critical entries written 
> by girls whom they displeased on Saturday night. We’ll see skinheads 
> “flush” from the system any entries that might have been written by 
> jews and blacks. This is not good and I will not allow the system I 
> build to be used in this manner.
> Graham Park has proposed that we loosen the existing language to 
> permit duplicate ids in the case where the entries have atom:source 
> elements which identify different URI’s in “self” links. I support 
> this compromise and believe it should be supported by the WG and 
> incorporated into the Atom Draft.
> responsible option other then modifying the existing constraint. No 
> one has been able to provide any argument which explains why the 
> attack vector I describe is *NOT* a problem. If the WG refuses to 
> modify this constraint, then it should at least feel compelled to 
> include text in the security considerations section that clearly 
> describes the attack and lets people know that no atom entry is “safe” 
> from trivial attacks as a result of this constraint. If the WG refuses 
> to modify this constraint, then they should expect that responsible 
> feed aggregators and generators of synthetic feeds will simply be 
> forced to deny support for Atom as specified.
> If anyone has a clear argument explaining why the attack is not 
> possible or not a problem, they should make it. Alternatively, if you 
> have a better method then that proposed by Graham for defending 
> against the attack, please describe it. Otherwise, Graham’s compromise 
> should be accepted and the specification revised.
>  
> bob wyman
>

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 27 Apr 2005 09:36:46 -0700

At 11:48 AM -0400 4/27/05, Bob Wyman wrote:
>I know that nobody seems to like this issueŠ However, I have 
>explained on numerous occasions that the existing prohibition 
>against duplicate ids in a feed simply cannot be supported by PubSub 
>or any other feed aggregator. The problem is, once again, that 
>prohibiting duplicate ids provides an easy to use attack vector for 
>those wishing to effectively "erase" entries written by another 
>author. (i.e. by publishing an entry with an id identical to one 
>published earlier, one can force the earlier entry to be flushed 
>from Atom feeds.)

Question (not a disagreement): Why wouldn't the later entry be 
dropped instead of the first one being flushed?

At 5:05 PM +0100 4/27/05, Bill de hÓra wrote:
>What will prevent people overwriting the atom:source[@rel='self'] 
>links as well as the id?

Seems like a good question. If someone is trying every avenue to 
erase an old entry, why wouldn't they try this as well?

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 27 Apr 2005 12:39:31 -0400

> What will prevent people overwriting the atom:source[@rel='self'] links
	Now you see why Graham Park's proposal is a "compromise." It
improves things but doesn't provide a definitive solution to all cases. It
means a great deal more processing for us, but at least allows us to defend
against this form of attack.
	Certainly, people could rewrite the links and certainly they will if
we modify the existing constraint to require it. However, I expect that at
PubSub, we'll insist on verifying the existence the entry in the reputed
source before we publish any entry with an atom:source[@rel='self'] link
that we haven't generated ourselves. (We will insert atom:source into
*every* entry we publish to ensure that attribution is properly provided.)
Alternatively, we'll insist on the entry being signed with a key which is
known to be associated with the claimed source feed. (We're still working on
the mechanisms by which keys would be associated with blogs.) I haven't yet
decided what to do in the case where we can't confirm that the entry
actually comes from the claimed source or someone who is authorized to
publish in the claimed source. We may publish the suspect entry with a new
id or we may simply discard the suspect entry. What do you suggest?
	Any ideas you may have for handling these cases would be
appreciated.

	bob wyman

---

From: Eric Scheid <[email-masked]>
Date: Thu, 28 Apr 2005 02:40:04 +1000

>    B- the simple solution which simply allows entries with duplicate

+1

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 27 Apr 2005 13:00:13 -0400

> dropped instead of the first one being flushed?
	The order of entries is not significant for the subject attack. The
reason is that it is possible for an attacker to mount an "anticipatory"
attack by discovering and predicting the atom:ids that will be generated by
a publisher under attack. In the simplest case, the author under attack
might use an id generating method that distinguished between ids by simply
incrementing a counter. Thus, if you knew the id of the last item published,
you could generate a "blocking" entry that used what was probably going to
be the next id generated. The fake id would block "real" posts if the rule
was to always drop the second or later instance of an id. This permits the
attacker to impose a prior-restraint or anticipatory block against a
publisher. 
	Also, strict enforcement of this rule would make it virtually
impossible for a legitimate "update" to an entry to ever get propagated
through the system. In the case of legitimate updates, I'm sure people would
argue that the older entry should be dropped, not the newer one.

> erase an old entry, why wouldn't they try this as well?
	This is answered in full in my previous email to the list.
Basically, the aggregator would have to verify the truth of the claim made
in the atom:source[@rel='self'] before republishing the entry. This is
cumbersome and expensive, but will often be possible.

		bob wyman

---

From: Antone Roundy <[email-masked]>
Date: Wed, 27 Apr 2005 11:56:20 -0600

On Wednesday, April 27, 2005, at 10:39  AM, Bob Wyman wrote:
> that we haven't generated ourselves.

If PubSub is subscribed to the feed pointed to by 
atom:source[@rel='self'], then can't you simply drop any entries 
claiming to be from that feed but found at a different URI?  You'll be 
getting those entries from the source if they're legitimate, so is 
there any reason to accept them from elsewhere?  Or are you keeping 
them in case they contain additional information added by someone else?

BTW, I'm in favor of the compromise solution.

---

From: Graham <[email-masked]>
Date: Wed, 27 Apr 2005 19:09:57 +0100

> attacker to impose a prior-restraint or anticipatory block against a

After years of dealing with seriously crappy RSS feeds, aggregator 
authors are smart enough to know everything in a feed is merely a hint 
and should not be not be trusted. This attack is not worth worrying 
about, though for completeness may be worth mentioning in the security 
section.

Graham Parks

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 27 Apr 2005 14:28:08 -0400

> about, though for completeness may be worth mentioning in the security 
	You may be happy to have your aggregator susceptible to such
attacks; however, I can't accept having my aggregator similarly compromised.
My users expect and demand a higher level of service from PubSub than they
get from other feed providers and aggregators...
	However strictly we may interpret the contents of feeds, the feed
format itself should be defined in such a manner so that an aggregator that
strictly interprets the specification will work properly when presented with
conforming feeds.
	In the absence of evidence that the proposed change will result in
damage or interoperability problems, I think the spec must change. A
security note is not sufficient.

		bob wyman

---

From: Robert Sayre <[email-masked]>
Date: Wed, 27 Apr 2005 14:44:08 -0400

> security note is not sufficient.

OK, we have two errors to balance here. One of them is the text we
currently have, which creates problems for republishers. The other is
detailed in Sam's "Clone Wars" post:

http://www.intertwingly.net/blog/2005/04/09/Clone-Wars

We could go with Graham's suggestion. We could also rework the
inheritance/considered-to-apply/attribution text more generally, to
include atom:source more explicitly. Is there anything else we can do 
that would license the validator to flag the Yahoo feed in Sam's post,
but give  PubSub et al. a little more room to breath?

Robert Sayre

---

From: Thomas Broyer <[email-masked]>
Date: Wed, 27 Apr 2005 20:44:24 +0200

Henry Story wrote:
>     B- the simple solution which simply allows entries with duplicate ids
>     C- No change
> 

Same for me.

> limitation). Therefore it should be dropped.

<troll>
should or must?
</troll>

-- 
Thomas Broyer

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 27 Apr 2005 14:47:18 -0400

> them in case they contain additional information added by someone else?
	Certainly, the content of an entry might be different depending on
its source. PubSub, for instance, now adds what we consider to be important
information to entries and we'll be doing more of that in the future. But,
that's only one of the reasons for concern here.

	The mere fact that an entry has been published in a secondary feed
is, in some cases, important information and in some cases, we should be
delivering up content which in other situations would be filtered out as
duplicative.
	Consider something like Scoble's well-known "LinkBlog" which
contains copies of things that he has found to be interesting when reading
through blogs. If we followed the policy of discarding everything that was
sourced somewhere outside the LinkBlog itself, we would never deliver any
content to anyone who subscribed explicitly to receive data from Scoble's
LinkBlog... (i.e. A PubSub weblog subscription such as:
"SOURCE:www.scobleizer.com/linkblog" )
	What we are considering doing to handle things like link blogs and
other cases where copied entries are found is flagging items as "duplicates"
when we discover them and then attaching a default "no-duplicates" predicate
on all normal subscriptions. However, subscriptions that included "SOURCE:"
predicates would have an "allow duplicates" predicate attached to them.
(Yes, users could override this.) This means that you wouldn't get
duplicates with a general subscription, however, you *would* receive content
that we would normally filter out -- if you subscribed to a specific feed.
Internally, this means that we would process all new or changed entries that
we find in any feed, but whether or not a duplicate is delivered to the user
depends on the users' subscription.
	Does this sound complicated? Well, it is... But, providing an
intermediary like we do just isn't quite as simple as building a desktop
aggregator.
	There are a number of other similar situations that are problematic.

		bob wyman

---

From: Tim Bray <[email-masked]>
Date: Thu, 28 Apr 2005 00:03:01 -0700

On Apr 27, 2005, at 9:05 AM, Sam Ruby wrote (about the proposal to 

> It would help if a Pace could be written.

Funny you should say that, Sam.

We have another week or so to go on our IETF Last Call, but the volume 
of input is low enough that Paul & I think we can safely start doing 
our own internal-issues final cleanup.  We've been following the WG 
discourse as best I can, but the volume is high.  Thus, we think that 
to be fair and transparent, we require a last round of Paces for 
any/all changes to the format-08 draft.

Sam, perhaps you could have a scan at the current set of Paces and see 
if any other than PaceOptionalSummary are actually up-to-date and 
suitable for this last round? (I'd be surprised, but you never know).

I think there are quite a few little things for which I've seen 
evidence of consensus on the mailing list, in fact I think Rob/Mark 
have pounced on a couple and they're in -08; but if you're behind one 
and it isn't, please Pace-i-fy it and let's get with the cleanup.  LAST 
CHANCE, we mean it.  And of course we're going to have to fish some 
sort of consensus out of this horrid summary-required mess.  -Tim

---

From: Robert Sayre <[email-masked]>
Date: Thu, 28 Apr 2005 04:10:00 -0400

On 4/28/05, Tim Bray <[email-masked]> wrote:

> And of course we're going to have to fish some
> sort of consensus out of this horrid summary-required mess.  

That shouldn't be hard. I don't think the WG has ever been more
decisive. If PaceOptionalSummary received 10 negative opinions, I'm
quite sure the chairs would characterize it as "drowning in -1s". Are
we "drowning in +1s" yet? Every bit of syndication code written since
my.netscape.com in 1999 can deal with title-only feeds. There's your
running code.

The IETF process is designed get work done when a vocal minority of WG
members makes a lot of noise about a given issue. In this case, the WG
has let the minority present a series of arguments, and some of the
arguments contradict previous ones. I think perhaps we've hit bottom.
I'm pretty sure Graham just wrote that I couldn't use the 7-line
PaceOptionalSummary, which made summaries and content optional by
striking one line from the spec, to assert that the WG wanted to make
summaries optional.

This is so over, I can't believe it. I cannot believe we are
entertaining this tripe.

Robert Sayre

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 28 Apr 2005 10:48:02 +0100

Tim Bray wrote:
> 
> And of course we're going to have to fish some sort of 
> consensus out of this horrid summary-required mess.  -Tim

I can't agree with that observation. Although there are a few strenuous 
objections against title only feeds, on the balance consensus is for 
them. Is there something you're seeing that should make us think the 
current consensus level is inadequate?

cheers
Bill

---

From: Graham <[email-masked]>
Date: Thu, 28 Apr 2005 11:14:21 +0100

On 28 Apr 2005, at 10:48 am, Bill de hÓra wrote:

>
> Tim Bray wrote:
>> And of course we're going to have to fish some sort of consensus out 
>> of this horrid summary-required mess.  -Tim
>
> I can't agree with that observation. Although there are a few 
> strenuous objections against title only feeds, on the balance 
> consensus is for them. Is there something you're seeing that should 
> make us think the current consensus level is inadequate?

Exactly because the two sides don't understand each other. I haven't 
seen any objections to "title only feeds" which you state is my and 
Sam's and other's position (we object to feeds that could have a 
summary included but don't). Similarly, it's not clear whether people 
on your side would equally support an alternate proposal to the current 
Pace.

Graham

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 28 Apr 2005 11:34:08 +0100

Graham wrote:
> Exactly because the two sides don't understand each other. 

I believe I understand the positions quite well. But maybe you can 
convince me otherwise.

> I haven't 
> Sam's and other's position (we object to feeds that could have a summary 
> included but don't). 

That last objection in parens sounds like some of the positions held 
around dates - that providers ought to do the right thing for some 
definition of the right thing. Given that legacy, I'll claim it's clear 
we're not here to police what people ought do with feeds that could have 
a summary. The single argument along those lines that was sensible was 
to accessibility, which was eventually dispensed with.

And if Robert's assertion elsewhere ("Every bit of syndication code 
written since my.netscape.com in 1999 can deal with title-only feed") is 
remotely true , ie there's not large number of codebases that will 
break, then you can add innovation through standards to my list of 
objections to any position that makes summaries non-optional.

> Similarly, it's not clear whether people on your 
> side would equally support an alternate proposal to the current Pace.

Any number of such counter-factual arguments can be made (after all 
that's largely the point of introducing them). They won't convince me, 
nor are they a basis for getting our work done.

cheers
Bill

---

From: Graham <[email-masked]>
Date: Thu, 28 Apr 2005 12:04:30 +0100

On 28 Apr 2005, at 11:34 am, Bill de hÓra wrote:

>> I haven't seen any objections to "title only feeds" which you state 
>> is my and Sam's and other's position (we object to feeds that could 
>> have a summary included but don't).
>
> definition of the right thing. Given that legacy, I'll claim it's 
> clear we're not here to police what people ought do with feeds that 
> could have a summary.

Then what are we here to do? Would you support removing all of the 
other requirements from the spec?

> written since my.netscape.com in 1999 can deal with title-only feed") 
> is remotely true , ie there's not large number of codebases that will 

Every code base can also cope without dates or ids. We still require 
them.

Graham Parks

---

From: Sam Ruby <[email-masked]>
Date: Thu, 28 Apr 2005 07:30:30 -0400

Graham wrote:
> Then what are we here to do? Would you support removing all of the other 
> requirements from the spec?
> 

First, note that this is mistating Graham's position.  He is not arguing 
that summaries be made non-optional.

> Every code base can also cope without dates or ids. We still require them.

This is not a theoretical discussion.  Quoting from RSS 0.92[1]:

  * All sub-elements of <item> are optional
  * any 0.92 source is also a valid 2.0 source

Is this really where we want to go?

- Sam Ruby

[1] http://backend.userland.com/rss092

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 28 Apr 2005 13:00:39 +0100

> Is this really where we want to go?

No, but please see my other mail replying to Graham on why I think 
PaceOptionalSummary does not go there.

cheers
Bill

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 28 Apr 2005 13:04:44 +0100

>> Bill:
> Graham:

 > Sam:
 > First, note that this is misstating Graham's position.
 > He is not arguing that summaries be made non-optional.

I support discussing requirements that can be framed operationally and 
which induce a benefit. This is why I keep coming back to title-only 
feeds rather than concerning myself with enforcing some sort of policy 
on what people think other people ought to do with summaries. Rationales 
along the lines of "we object to feeds that could have a summary 
included but don't", don't work for me, and I'll try to explain why 
here. I also (and this is possibly unfair to Graham) think that when 
counter-factual arguments are being presented against, then the argument 
against the pace are running out of steam.

> Every code base can also cope without dates or ids. We still require them.

Yes, and there we agreed not to require enforcing publishers to update 
on every change and have 3 date items to support that policy. Largely 
because it can't be enforced and downstream clients will end up baking 
bad assumptions into their tools.

My point here is probably worth elaborating on. I feel the same way 
about the must-be including summaries because you should be including 
summaries position, as I did about the must be updating dates because 
you should be updating dates position, that they are usefully 
enforceable. I also feel the same way about insisting on empty summary 
elements if that is used to obtain a compromise. I'm well aware of 
SHOULDs and optionals causing interop issues, however this must have 
summary approach, like must update timestamps, is not the kind of 
constraint that induces useful behaviour.

I've read and re-read the -1 positions to PaceOptionalSummary and I can 
neither see the harm in optional summaries nor the benefit in always 
having a summary. That is, operationally, I can't see what harm is 
caused by allowing title-only feeds or what the benefit of disallowing 
them is. But I can see the pointlessness, if not quite harm, of making 
people populate a field they either don't need or don't want to 
populate. I suspect that might induce junk data unneccesarily.

I believe I understand the -1 position, but I just haven't be swayed by 
the arguments for it; that leaves me at +1.

cheers
Bill

---

From: Antone Roundy <[email-masked]>
Date: Thu, 28 Apr 2005 08:36:28 -0600

The following Paces have not had their fate officially decided:

PaceOptionalSummary (+1)

PaceXmlContentWrapper (+1 -- there was plenty of opposition during the 
discussion of this one, but it seemed to be directed at both the 
current spec and the Pace, so I don't think it's clear whether people 
prefer the current spec or the Pace)

PaceCoConstraintsAreBad (written to draft 07 but I don't think the 
section to be changed changed since then)
PaceFeedIdOrAlternate (withdrawn by the author, so perhaps it shouldn't 
be listed here--written to draft 07 but the line in question hasn't 
changed since at least 06)
PaceFeedIdOrSelf (+1 -- just updated to draft 08 by changing a "6" to 
an "8")

Would a Pace be required to change the name of atom:image to atom:logo 
or could that be considered editorial?  "Logo" is much more descriptive 
of what it's for, and would reduce confusion if somebody creates an 
extension for non-logo images.

I assume a Pace is coming for atom:id being repeatable within a 
document as long as the entries in question have different source 
feeds? (I'd give it a +1)