Subject: Tombstones Update

From: James Snell <[email-masked]>
Date: Tue, 18 May 2010 23:10:33 -0700

Another backwards compatible update... explicitly allowing for an
optional atom:source element as a child of at:deleted-entry.

http://www.ietf.org/internet-drafts/draft-snell-atompub-tombstones-08.txt

- James

---

From: Bob Wyman <[email-masked]>
Date: Wed, 19 May 2010 02:29:07 -0400

Can an at:delete-entry be signed in the same way that an Atom entry can be?

Should I be concerned about an unsigned at:delete-entry which tells me to
delete a signed entry?

bob wyman

---

From: James Snell <[email-masked]>
Date: Tue, 18 May 2010 23:39:49 -0700

Very good question. Short answer is obviously yes it can be, but
calling that out explicitly in the spec is a good idea. There are
likely several concerns with this that would need to be considered...
such as what if both the entry and deleted-entry include valid
signatures but have been signed with different keys...

- James

-- 
- James Snell
  http://www.snellspace.com
  [email-masked]

---

From: Bob Wyman <[email-masked]>
Date: Wed, 19 May 2010 02:56:08 -0400

Your definition of the atom:source element is looser than that found in
4.2.11 of RFC4287 in that it doesn't specify which elements of a feed's
metadata "SHOULD" be preserved and it also doesn't prohibit modification of
an atom:source element which exists within a deleted-entry that is being
copied. (i.e. in Atom you only "MAY" insert an atom:source if there isn't
already one present in the entry.) Do you really intend to loosen the rules
for atom:source? If not, then why not just incorporate 4.2.11 by reference
in the same way you do for Atom specified elements like atom:link?

bob wyman

---

From: Bob Wyman <[email-masked]>
Date: Wed, 19 May 2010 22:58:45 -0400

In Atom, both Entry Documents and Feed Documents are "top-level" objects. An
entry may be contained within a Feed Document, but it need not be. This is
an important characteristic of Atom and one that distinguishes it from
previous, legacy syndication formats. This attribute of Atom makes it
particularly well suited for use in applications that rely on combining
entries from multiple sources as well as applications that provide real-time
delivery of data.

However, the deleted-entry that you define in the tombstone draft is defined
as only existing within Feed Documents. This represents a significant and
unfortunate departure from the base Atom RFC.

I would strongly encourage you to ensure that deleted-entry is symmetrical
with atom:entry and that it is available in all contexts (i.e. both within
and without a Feed Document) that an Atom entry is. Just as we can have
Entry Documents, we should be able to have Deleted-entry Documents.

bob wyman

---

From: James Snell <[email-masked]>
Date: Wed, 19 May 2010 20:17:17 -0700

Heh, Bob... you and I seriously need to get into the same room with a
whiteboard sometime.. lol..

Ok, first about your previous comment about atom:source... yes, the
intention is for it to be identical to the use within atom:entry so I
will tighten that up in the spec...

As for standalone deleted entry documents... I definitely can think of
a few generally interesting potential use cases... the key question is
whether they are compelling.

- James

---

From: Bob Wyman <[email-masked]>
Date: Wed, 19 May 2010 23:36:12 -0400

James Snell <[email-masked]> wrote:
> As for standalone deleted entry documents...
> the key question is

An example... "Atom over
XMPP<http://xmpp.org/internet-drafts/draft-saintandre-atompub-notify-07.html>"
calls for publishing streams of Atom Entry Documents over XMPP. Because
there was no mechanism defined for deleting in Atom itself, "Atom over XMPP"
ended up defining a "retract" message for that purpose. It would be more
natural to use a tombstone.

But, the general question is: Atom format permits my application to publish
only Entry Documents. However, if I do that, and if a delete-entry can only
exist as an element of a Feed Document, how do I tell anyone about a
deleted-entry? Would I need to modify my protocol to support Feed Documents
simply as a means to transport deleted-entries? Doesn't seem right.

bob wyman

---

From: Ed Summers <[email-masked]>
Date: Thu, 20 May 2010 02:54:06 -0400

On Wed, May 19, 2010 at 10:58 PM, Bob Wyman <[email-masked]> wrote:.
> Entry Documents, we should be able to have Deleted-entry Documents.

+1

I could well imagine wanting to return a Tombstone as the message body
for a 403 Gone response for a resource that I know has been deleted. I
guess I had been reading the following pretty liberally:

"""
The at:deleted-entry element MAY appear as a child of atom:feed to
   represent an Atom Entry that has been removed from a feed.

     deletedEntry = element at:deleted-entry {
       atomCommonAttributes,
       attribute ref { atomUri },
       attribute when { atomDateConstruct },
       ( element at:by { atomPersonConstruct}?,
       & element at:comment {atomTextConstruct}?,
       & element atom:link*,
       & element atom:source?,
       & extensionElement* )
     }
"""

Does the MAY prevent at:deleted-entry being used outside the context
of a atom:feed?

//Ed

[1] http://tools.ietf.org/html/rfc2616#section-10.4.11

---

From: Colm Divilly <[email-masked]>
Date: Thu, 20 May 2010 09:10:36 +0100

+1

---

From: Niklas Lindström <[email-masked]>
Date: Thu, 20 May 2010 12:21:37 +0200

On Thu, May 20, 2010 at 8:54 AM, Ed Summers <[email-masked]> wrote:
> for a 403 Gone response for a resource that I know has been deleted.

+1 from me as well.

Apart from the use cases already mentioned (XMPP, 403
representations), I see potential for using deleted-entry documents in
our system (the swedish legal information system). I'm currently using
entry documents internally in our "depot", both as manifests and to
represent the collected source entries ("via" entries). Having
deleted-entry documents there as well would be very beneficial.

(In fact, this was a primary motive for my wish for allowing
atom:source in at:deleted-entry elements. Which I'm very happy to see
in the latest draft -- thanks James!)

Two points come to mind if this is done:

* What would the mime-type for deleted-entry documents be? A new
"application/atom+xml;type=deleted-entry"?
* For good measure, making sure that this doesn't lead to a pattern of
PUT:ing deleted-entry docs instead of using DELETE... ;)

Best regards,
Niklas

---

From: James Snell <[email-masked]>
Date: Thu, 20 May 2010 08:03:50 -0700

>> for a 403 Gone response for a resource that I know has been deleted.

Ok, looks like we've got quite a bit good support for standalone
docs.. I'll add that in.

> in the latest draft -- thanks James!)

No prob. Thanks for the great feedback and input.

> "application/atom+xml;type=deleted-entry"?

Putting these under the application/atom+xml media type would likely
cause badness. I'm thinking something like
application/atomdeleted+xml.

> PUT:ing deleted-entry docs instead of using DELETE... ;)

Yeah, this is a concern of mine also. Need to make sure people don't go there.

---

From: James Snell <[email-masked]>
Date: Fri, 21 May 2010 07:28:31 -0700

http://www.ietf.org/internet-drafts/draft-snell-atompub-tombstones-09.txt

Quite a few important updates.

On May 20, 2010 8:03 AM, "James Snell" <[email-masked]> wrote:

2010/5/20 Niklas Lindström <[email-masked]>:

>> On Wed, May 19, 2010 at 1...

> representations), I see potential for usi...

> * What would the mime-type for deleted-entry documen...

> PUT:ing deleted-entry do...
Yeah, this is a concern of mine also. Need to make sure people don't go
there.

--

- James Snell

---

From: Bob Wyman <[email-masked]>
Date: Mon, 24 May 2010 10:28:52 -0400

The Tombstone draft is coming along nicely, however, I can't help
wondering... Since it appears that a deleted-entry is so much like a normal
entry, why isn't it just an extended atom:entry with some additional element
or attribute flagging it as deleted?

There are a number of "deletion" use cases, some of which already have
support in the base Atom Format specification. For instance, since Atom
defines "replacement," (i.e. publish an entry with the same atom:id as one
previously published but with a later atom:updated date), there is already a
crude method for asking that content be removed from aggregators. However,
this tends to encouraging people to publish entries that have titles of
"Deleted," content of "Deleted." Etc. Not particularly elegant -- but it
works, sort of.

I'm afraid that if deleted-entry is introduced into the system, what we'll
see is that during some arbitrarily long transition period while we wait for
people to modify their software to support deleted-entry, people who really
want to purge the network of some late night blog post will tend to publish
both a "Deleted" replacement entry as well as a deleted-entry... They would
do this since it is potentially going to be a long time (perhaps infinite)
before deleted-entry is widely implemented and just publishing a normal
entry has much the same effect (but not completely) of publishing a
deleted-entry.

If, on the other hand, the definition of entry was extended to permit a
"deleted" flag or attribute, then we'd find that applications would have
implicit support for entry deletion without needing to write any more code.
Certainly, applications that understood what a deletion actually meant and
that understood the extra fields like when, by and comment, would be able to
do interesting things based on that understanding.

I'm also concerned that the utility of the deleted-entry mechanism is
limited to only those systems that rely on "topic-based" filtering -- i.e.
following a specific feed and consuming all of its content. We would need
some entirely different mechanism to support content-based filtering -- i.e.
extracting entries based on their content (presence of words, phrases,
etc.). Deleted-entry can't help a content-based system since unless such
systems maintain cumbersome logs of all atom:ids that are published and who
they were published to. The problem is that since a deleted-entry contains
virtually none of the data contained in the entry deleted, any topic-based
filtering can't figure out who might have previously received a copy of the
now-deleted entry.

Of course, the easiest way to delete things in a content-based system would
be to republish the original entry (with all its content) and mark the entry
as "deleted." This will work in, but not all applications, since in some
applications, one does not wish to republish the data that is being deleted.
However, in at least some applications, where one can trust that consumers
are paying attention to the deleted flag, it is acceptable to republish the
now out-of-date data.

If a deleted-entry is so much like an entry, why can't it just be one? (with
some extra metadata).

bob wyman

2010/5/21 James Snell <[email-masked]>

---

From: Erik Wilde <[email-masked]>
Date: Mon, 24 May 2010 11:28:57 -0700

hello.

Bob Wyman wrote:
> wondering... Since it appears that a deleted-entry is so much like a 
> normal entry, why isn't it just an extended atom:entry with some 
> additional element or attribute flagging it as deleted?

i think this is a great approach of looking at what "deletion" means, in 
a way it's just another "update". however, there probably are two major 
problems with this approach:

- it is not backwards-compatible with prior versions of the draft, and 
it seems that the draft already has seen some adoption. if the goal is 
to not break these early implementations, then moving from deleted-entry 
to entry is not an option, i am afraid.

- atom disallows extensions to change the semantics of any standard atom 
elements. whether additional metadata changing the semantics of an 
"updated" entry to a "deleted" entry is such a change in semantics is a 
question of perspective. one could say that "deleted" is different from 
"updated", or one could say that it's just a special case. 
http://tools.ietf.org/html/rfc4287#section-4.2.15 just says that 'The 
"atom:updated" element is a Date construct indicating the most recent 
instant in time when an entry or feed was modified in a way the 
publisher considers significant', which i think could be seen as 
covering the case of deletion of an entry as well.

personally, i think that having an entry/@deleted would be quite a bit 
more elegant and consistent than having a deleted-entry (there is no 
updated-entry, after all...), but that still does not solve the problem 
of breaking early adopters' code. but for me, the most important thing 
is to have something in feeds that covers the CRUD's D, so i am very 
glad to see the tombstone draft moving along again, whatever it will end 
up defining.

cheers,

erik wilde   tel:+1-510-6432253 - fax:+1-510-6425814
        [email-masked]  -  http://dret.net/netdret
        UC Berkeley - School of Information (ISchool)

---

From: Bob Wyman <[email-masked]>
Date: Mon, 24 May 2010 15:08:53 -0400

Erik,
I don't think we would have to go as far as changing the semantics of any
standard atom elements. Rather, what I'm suggesting is that "delete" is
really just a bit of an extension to what the existing replace means. In
other words, if you understand the "delete" flag, however, it is encoded,
you would do a delete, if not, then you would simply do a replace as per the
existing spec. This would be properly backwards-compatible.

I would, of course, regret breaking any early-adopter code. However, I
suggest that there is a much larger population of long-ago-adopters who
wrote Atom code potentially years ago and would, if deleted-entry became
accepted, have to consider rewriting or at least modifying their code. One
of the risks of being an early adopter is that things may change, however,
people who wrote code to the published spec should be granted the right to
expect that things will be more stable.

The only downside I can see in extending the existing entry is that the
resulting tombstones would be a bit less bit-efficient than desirable. (i.e.
they would, in most cases, probably end up having empty required elements --
such as atom:title.)

bob wyman

On Mon, May 24, 2010 at 2:28 PM, Erik Wilde <[email-masked]> wrote:

> i think this is a great approach of looking at what "deletion" means, in a
> way it's just another "update". however, there probably are two major
> - it is not backwards-compatible with prior versions of the draft, and it
> seems that the draft already has seen some adoption. if the goal is to not
> break these early implementations, then moving from deleted-entry to entry
> is not an option, i am afraid.
>
> elements. whether additional metadata changing the semantics of an "updated"
> entry to a "deleted" entry is such a change in semantics is a question of
> perspective. one could say that "deleted" is different from "updated", or
> one could say that it's just a special case.
> instant in time when an entry or feed was modified in a way the publisher
> considers significant', which i think could be seen as covering the case of
> deletion of an entry as well.
>
> personally, i think that having an entry/@deleted would be quite a bit more
> elegant and consistent than having a deleted-entry (there is no
> updated-entry, after all...), but that still does not solve the problem of
> breaking early adopters' code. but for me, the most important thing is to
> have something in feeds that covers the CRUD's D, so i am very glad to see
> the tombstone draft moving along again, whatever it will end up defining.
>
> cheers,
>

---

From: Peter Keane <[email-masked]>
Date: Mon, 24 May 2010 14:28:44 -0500

Hi All-

For what it is worth, there was a fairly in-depth conversation about
tombstones back in Dec 2007.  James Snell (and others) put forth
strong arguments against it just being another atom:entry (I was among
those, at the time, advocating for atom:entry, but I have come around
to the lighter option mainly for practical (not aesthetic) reasons).

The thread begins here, and may be worth perusing (follow links to follow-ups):

http://www.imc.org/atom-syntax/mail-archive/msg20050.html

--Peter

---

From: Peter Keane <[email-masked]>
Date: Mon, 24 May 2010 14:32:07 -0500

A bit farther into that thread:

http://www.imc.org/atom-syntax/mail-archive/msg20111.html

--peter

---

From: James Snell <[email-masked]>
Date: Mon, 24 May 2010 12:55:07 -0700

The fundamental argument for at:deleted-entry is that Atom processors
that do not understand Tombstones would see anything along the lines
of <entry at:deleted="true">...</entry> or whatever as any other
entry. Further, many systems that currently do not support the notion
of a soft-delete typically do not keep around enough metadata about
the deleted entry to meet the minimum content requirements of an
atom:entry. The title, the content, etc would likely either have to be
blanked out or set to something like "DELETED", etc. The challenge
with this is that any given feed could contain any number of
tombstones and entries mixed together... users of non-tombstone aware
clients could potentially see a bunch of useless "DELETED" entries
that serve no purpose whatsoever.  With the at:deleted-entry approach,
unaware clients will have a better user experience and aware clients
can still do the right thing. With the entry/@deleted approach,
unaware clients get suboptimal results. With either approach,
processors that want to be aware of tombstones have to do the same
amount of work to implement support.

---

From: Bob Wyman <[email-masked]>
Date: Mon, 24 May 2010 16:16:20 -0400

It might be good to include the reason for *not* reusing atom:entry in
non-normative text. I imagine that many folk will ask this question.

I realize that achieving perfection is sometimes not possible...
Nonetheless, I'd like to point out again that if deleted-entry doesn't
support all the fields that entry does, then it becomes impossible to
support a variety of applications that rely on content-based rather than
topic-based routing.

bob wyman

On Mon, May 24, 2010 at 3:55 PM, James Snell <[email-masked]> wrote:

> normal
> it
> "updated"
> publisher

---

From: James Snell <[email-masked]>
Date: Mon, 24 May 2010 13:36:41 -0700

Well, there is a definite reason why I allow at:deleted-entry to
contain extension elements.. one could easily do something like:

 <at:deleted-entry ...>
   ...
   <atom:title>the entry title</atom:title>
   <atom:content type="text">the entry content</atom:content>
   ...
 </at:deleted-entry>

Because of the extension model, at:deleted-entry can contain all of
the atom:entry children, the exact semantics, however, is undefined by
the base tombstone spec. A content-based application could define the
necessary semantics and things would just work.

---

From: John Panzer <[email-masked]>
Date: Mon, 24 May 2010 15:43:18 -0700

Note that Salmon (http://salmon-protocol.org) also needs standalone (signed)
deleted entry documents.  Its security model allows for changing of keysm --
e.g., if the original was compromised -- as long as both keys are bound to
the same author/uri.

On Wed, May 19, 2010 at 8:36 PM, Bob Wyman <[email-masked]> wrote:

> An example... "Atom over XMPP<http://xmpp.org/internet-drafts/draft-saintandre-atompub-notify-07.html>"

---

From: James Snell <[email-masked]>
Date: Mon, 24 May 2010 15:51:13 -0700

Yes, I wanted to be careful about dictating any kind of behavior if
the keys ended up being different because there are legitimate reasons
why they would be (e.g. if an editor deletes an entry created be a
different author). Applications just need to be aware of the potential
risks involved.  Ironically, the greater risk is deletion with a
compromised key.

On Mon, May 24, 2010 at 3:43 PM, John Panzer <[email-masked]> wrote:
>> An example... "Atom over XMPP" calls for publishing streams of Atom Entry
>> Documents over XMPP. Because there was no mechanism defined for deleting in
>> Atom itself, "Atom over XMPP" ended up defining a "retract" message for that
>> purpose. It would be more natural to use a tombstone.
>> But, the general question is: Atom format permits my application to
>> publish only Entry Documents. However, if I do that, and if a delete-entry
>> can only exist as an element of a Feed Document, how do I tell anyone about
>> a deleted-entry? Would I need to modify my protocol to support Feed
>> Documents simply as a means to transport deleted-entries? Doesn't seem
>> right.
>> bob wyman

---

From: James Snell <[email-masked]>
Date: Mon, 24 May 2010 16:12:41 -0700

One additional comment, because of the extension model allowed by the
tombstone, it would also be possible for a publisher to include the
complete atom:entry as a child of the at:deleted-entry to address the
scenarios that Bob has brought up... e.g.

<at:deleted-entry>
  ...
  <atom:entry>
    ...
  </atom:entry>
  ...
</at:deleted-entry>

- James

---

From: Sam Johnston <[email-masked]>
Date: Tue, 25 May 2010 01:51:48 +0200

James et al,

I have to admit that this all sounds rather complicated. I'm already having
stiff resistance to adoption of Atom in other standards because of
[unfounded] perceived complexity and would rather not give the critics
justification.

>From what I understand of the requirements I tend to agree with Bob in that
atom:entry should contain a "deleted" attribute or entry, possibly
specifying the deletion time. As tombstones are almost always the exception
rather than the rule I don't see "deleted" placeholder text being a huge
problem and it allows publishers some flexibility in leaving some/all of the
content in place in an interoperable fashion (e.g. avoiding the need to
extend deleted-entry with atom:entry or its children). In any case I'm not
the first to suggest<http://www.imc.org/atom-syntax/mail-archive/msg20057.html>that
you can always exclude the tombstones by default and enabling them
with
a query parameter.

I do think this draft runs the risk of being widely implemented, and in its
current form will add non-negligible complexity to most client and server
code bases; in contrast ignoring entries with a "deleted" attribute/element
is a trivial change. If it's well justified and there's a strong consensus
then fine, but implementors of draft specifications know what they're
getting themselves into so I don't see this as a strong case to keep the
status quo.

Sam
Google

On Tue, May 25, 2010 at 1:12 AM, James Snell <[email-masked]> wrote:

> >>>> > entry may be contained within a Feed Document, but it need not be.
> This
> >>>> > is
> >>>> > as only existing within Feed Documents. This represents a
> significant
> >>>> > and

---

From: James Snell <[email-masked]>
Date: Mon, 24 May 2010 17:53:46 -0700

For the overwhelming majority of Atom clients, there's nothing easier
than ignoring an extension element they don't understand and not
having to worry about phantom atom:entry elements that represent
deleted content.... it doesn't take any effort at all to ignore an
atom:entry that doesn't exist ;-) ...

1. Simple example A
  <feed>
    <at:deleted-entry ref="abc" when="..." />
    <entry>
      <id>xyz</id>
      ...
    </entry>
  </feed>

2. Simple example B
  <feed>
    <entry at:deleted="true">
      <id>abc</id>
    </entry>
    <entry>
      <id>xyz</id>
    </entry>
  </feed>

Given these two choices, ordinary clients that do not know anything
about tombstones will, correctly, only see one entry in Example A...
but will see two entries in Example B.. one that doesn't really exist
and whose title, author, updated, alternate link and summary/content
won't necessarily actually exist or provide any useful information to
the client whatsoever.

For clients that support tombstones, looking for at:deleted-entry vs
atom:entry/@deleted is actually fairly trivial... Using
atom:entry/@deleted however still leads to an unnatural mapping of
atom:entry for the most common tombstone cases because the values for
atom:title, alternate links, atom:summary or atom:content just might
not exist. An implementation only needs to worry about extension
elements in at:deleted-entry when they have specific use cases that
call for them.  Not really all that complicated :-D

On Mon, May 24, 2010 at 4:51 PM, Sam Johnston <[email-masked]> wrote:
> the first to suggest that you can always exclude the tombstones by default
> and enabling them with a query parameter.
>> >>>> > However, the deleted-entry that you define in the tombstone draft
>> >>>> > is
>> >>>> > defined