Subject: On organization and abstraction

From: Tim Bray <[email-masked]>
Date: Thu, 03 Feb 2005 18:37:37 -0800

On reflection, I am growing very negative on almost all of the 
"Organization" Paces, including FeedRecursive, PaceEntriesElement, 
PaceCollection.  Here's why: they represent to increase the level of 
abstraction in Atom, and in my experience, when the goal is 
interoperability across the network, abstraction hurts.  I would like 
it if the markup found in Atom documents was as close as possible to a 
typical human mental model.  The word "feed" has entered the 
vocabulary, even the non-geek vocabulary, and the notion that there are 
"things" (entries, stories, items, whatever) in "feeds" likewise.  Any 
attempt to abstract away and generalize along the lines of "well, a 
feed is really an entry" or "well, an entry is really a feed" or 
"really, they're all just web resources and collections" is dangerously 
close to verging on Architecture Astronautics.

Put another way, Adam Bosworth, likely one of the best computer 
programmers in the world, said "every time I add abstraction to an 
interface I lose 90% of the audience."  I would like Atom to be more 
like Visual Basic and less like Lisp.

Put another way, starting in 1986 there was a system called SGML, an 
ISO standard for marking up documents, which was awesome in its 
generality; the notion of what a delimiter was, what a character was, 
what an element was, all these things were entirely abstract.   So, it 
was almost impossible to implement and never really caught on.

On the other hand, the notion that sometimes you have collections of 
feeds is easy to understand, easy to verbalize, and widely evidenced in 
practice (cf PubSub & friends), if not perhaps widely seen outside of 
geekland.  So I think I'm +1 on PaceAggregationDocument.  (And I think 
if we adopted that we could certainly lose PaceHeadInEntry, right Bob?)

  -Tim

---

From: Robert Sayre <[email-masked]>
Date: Thu, 03 Feb 2005 21:52:21 -0500

Tim Bray wrote:
> interoperability across the network, abstraction hurts. 

There's a diff of your feed in PaceEntriesElement. It's pushing it to 
claim one is more interoperable than the other. You just asked for 
examples 15 minutes ago, why don't you wait for them?

> The word "feed" has entered the vocabulary, 
> even the non-geek vocabulary, and the notion that there are "things" 
> (entries, stories, items, whatever) in "feeds" likewise.  

None of the "feed formats" out there use the term "feed" in their 
vocabularies.

> Any attempt to 
> abstract away and generalize along the lines of "well, a feed is really 
> an entry" or "well, an entry is really a feed" or "really, they're all 
> just web resources and collections" is dangerously close to verging on 
> Architecture Astronautics.

OK, define "Feed" and "Entry". We haven't done it yet.

I think calling other designs "astronautics" is unjustified. HeadInEntry 
is astronautics in the worst way--just nest them. But hey that's just my 
opinion. Also, atom:content is architecture astronautics. Base64? 
Please. HTML in atom:copyright? architecture astronautics.

> like Visual Basic and less like Lisp.

This is not a meaningful comparison to me.

> if we adopted that we could certainly lose PaceHeadInEntry, right Bob?)

I think you should wait for examples.

Robert Sayre

---

From: Graham <[email-masked]>
Date: Fri, 4 Feb 2005 03:06:24 +0000

On 4 Feb 2005, at 2:37 am, Tim Bray wrote:

> feeds is easy to understand, easy to verbalize, and widely evidenced 
> in practice (cf PubSub & friends), if not perhaps widely seen outside 
> of geekland.  So I think I'm +1 on PaceAggregationDocument.  (And I 
> think if we adopted that we could certainly lose PaceHeadInEntry, 
> right Bob?)

If you removed the ability to have entries within the feeds in 
aggregation documents, I'm in. PaceHeadInEntry covers a fundamentally 
different task.

A collection of feeds would be something like a list of blogs about a 
certain topic? You get all the information you need in the feed-head, 
and supplying entries is fairly redundant. Essentially supplying recent 
entries from those feeds would be extra metadata about the feed.

Whereas something like Feedster search results are a collection of 
entries that come from various blogs. It's the entries the user is 
interested in, and knowing which feed they came from is essentially 
extra metadata about the entry.

Graham

---

From: Robert Sayre <[email-masked]>
Date: Thu, 03 Feb 2005 22:19:32 -0500

> aggregation documents, I'm in. PaceHeadInEntry covers a fundamentally 

Why in the world should we define some other document format? OPML will 
do that just fine.

Robert Sayre

---

From: Eric Scheid <[email-masked]>
Date: Fri, 04 Feb 2005 14:26:08 +1100

> entries from those feeds would be extra metadata about the feed.

I think the purpose of PaceAggregationDocument is to provide a means to
subscribe to an intermediary aggregator like pubsub, and thus receive
entries.

A collection of feeds for the purpose of a blog roll (ie. no entries) can be
modelled in atom as it currently stands - each entry is descriptive about
some resource, being a feed, and you'd just have a <link rel=alternate /> to
the feed itself.

e.

---

From: Tim Bray <[email-masked]>
Date: Thu, 03 Feb 2005 19:25:33 -0800

> aggregation documents, I'm in. PaceHeadInEntry covers a fundamentally 

I'm claiming they do the same thing.  Instead of

<feed>
  <head><id>f1</id></head>
  <entry>
   <head><id>f1</id></head>
   <id>e1</id>
   </entry>
  <entry>
   <head><id>f1</id></head>
   <id>e2</id>
   </entry>
  <entry>
   <head><id>f2</id></head>
   <id>e3</id>
   </entry>
  </feed>

you'd have

<aggregation>
  <head><id>f1</id></head>
  <feed><id>f1</id>
   <entry><id>e1</id></entry>
   <entry><id>e2</id></entry>
   </feed>
  <feed><id>f2></id>
   <entry><id>e3</id></entry>
   </feed>
  </aggregation>

Which on the face of it seems like an improvement. -Tim

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 3 Feb 2005 23:08:43 -0500

Tim Bray wrote:
> So I think I'm +1 on PaceAggregationDocument.  (And I think 
	-1...
	PaceAggregationDocument does not seem to me to add much benefit for
all the costs that it entails. I'm particularly concerned that adding a new
type of root document "aggregation" is likely to add enough to the
complexity of Atom that only a subset of developers will actually get around
to supporting this third type of document -- a type which doesn't exist in
the simple RSS aggregators that exist today.
	I see two non-compelling benefits to PaceAggregationDocument over
PaceHeadInEntry:
	1. In the case where a feed will contain more than one entry from a
"foreign" feed, you only have to include the atom:head data once. Thus,
there would be some increase in efficiency of encoding... However, as I've
pointed out on numerous occasions, this is a rare case. Typically, at
PubSub, we DO NOT see many cases where atom:head information is repeated in
a single instance of a feed.
	2. It is easier to see how one would sign an aggregated entry since
you don't muck with the contents of entry by inserting the HeadInEntry. This
problem, however, would be solved by canonicalization rules that said that
you should remove HeadInEntry before processing signatures, or, rules that
said that you had to insert HeadInEntry before signing anything, or filter
rules that allowed one to flag HeadInEntry as not part of the signed
content. (i.e. alternative solutions exist...)

	But, there are tremendous problems with the proposal:
	1. It looks like I can't intersperse "native" entries with
aggregated entries. This is a nuisance. I would like to see people insert
entries with "HeadInEntry" when they copy something from another feed into
their own feed. PaceAggregationDocument says that your feed must have either
"native" entries or "foreign" entries, but it can't have both. This seems
like an unnecessary constraint.
	2. As mentioned before, the introduction of a new level of
abstraction (i.e. aggregation) is likely to scare off a good proportion of
the aggregator developers. Current aggregators don't have the concept of
"aggregation" in them. Thus, new design and new architecture will be
required to address this Pace. On the other hand, staying with HeadInEntry
is much more gentle on the existing systems and thus much more likely to be
useful in the field.
	3. Since there will be at least some subset of aggregators that
won't understand the "aggregation" thing, it is likely that we'll have a
chicken and egg problem. No one will produce "aggregation" documents since
not everyone supports them. Thus, no one will support them since no one
generates them. On the other hand, HeadInEntry will slide past minimally
updated aggregators with no trouble -- i.e. folk who don't want to do the
work of handling the stuff properly will simply ignore the Head element...
(i.e. it would be suicide for producers like PubSub to support
PaceAggregationDocument.)
	4. Massive changes need to be made to the specification when we
don't have a great deal of time left before we're supposed to be doing a
"Last Call." This is dangerous.
	
	-1. I really don't see any compelling benefit in exchange for the
tremendous risk and cost of accepting PaceAggregationDocument and dropping
HeadInEntry. Let's avoid adding to the levels of abstraction here and keep
it simple. We should have feeds and entries -- nothing else.

> I would like Atom to be more like Visual Basic and less like Lisp.
	As an ex-Visual Basic Product Manager, I think this would be a good
idea! Let's keep it simple and NOT accept PaceAggregationDocument. (Note to
reader: "Visual Basic .NET" is .NOT "Visual Basic"...

		bob wyman

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Tim Bray
Sent: Thursday, February 03, 2005 9:38 PM
To: Atom Syntax
Subject: On organization and abstraction

  -Tim

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 3 Feb 2005 23:35:14 -0500

Eric Scheid wrote:
> I think the purpose of PaceAggregationDocument is to provide a 
> means to subscribe to an intermediary aggregator like pubsub, and
> thus receive entries.
	If that is the motivation, then I think you really should find some
other motivation. 
	Frankly, it is exceptionally unlikely that we at PubSub could take
the risk of relying on "aggregation" documents since it is probable that it
will be quite some time before substantially all aggregators are updated to
support them. It is much more likely that a large proportion of existing and
new aggregators will rapidly upgrade their Atom 0.3 support to support the
Atom 1.0 feed and entry documents -- but not aggregation documents. 
	My experience with past protocol upgrading events leads me to
believe that a large percentage of developers will tire once they have the
obvious and familiar bits supported and will then release what they claim to
be "compliant" code but with "aggregation" support "delayed until next
release..." At PubSub, we'll end up with customers complaining that the Atom
we're generating is not supported by their aggregators... Folk will accuse
us of having "buggy code". Competitors will attack us for screwing our users
in some Quixotic quest for "Atom Purity" when we should be addressing user
needs by supporting RSS V2.0 and "core Atom"... No thank you. I really don't
want to go there. Find some other reason to push aggregation documents. But,
whatever you do, PLEASE don't drop HeadInEntry. If you do drop HeadInEntry,
we'll just have to keep using the "ps:" namespace to put it back in.
	For us to support aggregation documents might be politically correct
-- if this Pace is accepted, however, it would be suicide from a business
point of view. Please don't ask us to do that which we can not do...

		bob wyman

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 3 Feb 2005 21:15:23 -0800

+1 - someone else made a comment about OPML which really hit the spot; 
if you try to make a format do all things, it does most of them 
badly...

On Feb 3, 2005, at 6:37 PM, Tim Bray wrote:

> vocabulary, even the non-geek vocabulary, and the notion that there 
> are "things" (entries, stories, items, whatever) in "feeds" likewise.

--
Mark Nottingham     http://www.mnot.net/

---

From: James Snell <[email-masked]>
Date: Thu, 3 Feb 2005 21:37:20 -0800

> the simple RSS aggregators that exist today.

+1 to this -1.  The aggregation element is not a necessary part of the
core and adds complexity without adding *significant* benefit to the
core.

> is much more gentle on the existing systems and thus much more likely to be

This is precisely why I don't think this belongs in the core.  For
those who need this type of functionality, the opportunity exists for
them to create a separate Internet-Draft that describes how to
aggregate feeds -- without adding complexity to the core syntax by
introducing elements that the vast majority of users will *likely*
never use.  (I obviously base this assumption on the reasoning that
most Atom users will use it as a drop in replacement for RSS).

> generates them. 

Well, I wouldn't go this far.  There is obviously a requirement for it
for some folks and those folks are likely to make good use of it. 
What I would rather see the WG do is take a minimalist approach with
this.  Right now, I see aggregated feeds falling into that limited 20%
of use cases that could be done, but aren't necessarily critical to be
done.  Aggregation could be defined as an extension to Atom, and
later, based on an analysis of its implementation over time, can be
merged into the Atom core if deemed appropriate.  Merging it into the
core now does not guarantee that its adoption and use will be any
greater than if it were done as an extension.

> "Last Call." This is dangerous.

+1. Big +1.  

> it simple. We should have feeds and entries -- nothing else.

One point, HeadInEntry solves the problem of having a standalone Atom
Entry document be able to reference a feed of which it is a part. 
Unless I'm misreading something, if PaceAggregationDocument gets rid
of the ability to put Head elements in the entry, don't we lose this
capability?  (this is something that is important, for instance, in my
Atom Notification Protocol proposal)

> reader: "Visual Basic .NET" is .NOT "Visual Basic"...

Aargh! VB... it burns!  (sorry, temporary flashback to past hell. 
Sorry Bob, VB was an excellent product, I just was forced to be
witness to some extremely bad uses of it.  Nightmares, I tell you,
nightmares!)

-- 
- James Snell
  http://www.snellspace.com
  [email-masked]

---

From: Robert Sayre <[email-masked]>
Date: Fri, 04 Feb 2005 00:56:07 -0500

>>"Last Call." This is dangerous.

I really regret writing down the religion that format-05 seems to 
embody. Take a look at PaceEntriesElement. It does not propose a 
"massive change". It's just *shorter*.

Some quotes from format-05:

"Atom is an XML-based document format which describes lists of related 
information known as "feeds". Feeds are  composed of a number of items, 
known as "entries", each  with an extensible set of attached metadata. 
For example, each entry has a title."

"The "atom:feed" element is the document (i.e., top-level)  element of 
an Atom Feed Document, acting as a container for  metadata and data 
associated with the feed"

"The "atom:entry" element represents an individual entry."

I really must find out more about this theology that allows one to 
divine the precise nature of lists, data, metadata, items, containers, 
and representations.

Robert Sayre

---

From: Antone Roundy <[email-masked]>
Date: Fri, 4 Feb 2005 00:03:06 -0700

> a single instance of a feed.
In the past few days, I've been reading people talking about different 
ways emerging for people to combine or divide the content of their own 
feeds based on what's going to end up in our <category> element and by 
other methods.  (See 
http://www.mezzoblue.com/archives/2005/02/03/information_/index.php).  
This kind of thinking could very well point the way to the emergence of 
more feeds being aggregated from a small number of sources.  Time will 
tell.

> content. (i.e. alternative solutions exist...)
When you remove HeadInEntry, do you remove from the start of the <head> 
tag to the end of the </head> tag, or do you also take out the newline 
after the </head> tag...?  We'd have to be certain we defined the 
process precisely and that people followed it precisely.  The benefit 
of avoiding that complexity may be a little greater than it appears at 
first blush.

> like an unnecessary constraint.
You could always have a feed in the aggregation for the "native" 
entries.  But you do have a point--the thing that would be more 
difficult would be to occasionally import external content into a feed 
that is not always an aggregation--you'd either have to temporarily 
become an aggregation, refer to it rather than importing it, or forget 
it.  Out of curiosity though, how common is it to import entries 
wholesale from external sources as opposed to referring to them?

> is much more gentle on the existing systems and thus much more likely 
Current aggregators don't have the concept of HeadInEntry either.  With 
HeadInEntry, it's easier because you don't have to deal with one more 
level of hierarchy--if you don't know about HeadInEntry and just ignore 
it, you can still function.  However, that leads to erroneously 
attributing an entry to the wrong feed.  To get it right, aggregators 
are going to have to understand SOMETHING new.  The containment in an 
aggregation document would be more intuitively obvious in meaning than 
seeing a head in an entry and figuring out that it was the head from 
the feed that used to surround the entry.

> "Last Call." This is dangerous.
If we don't want to reorganize the spec to do this, we can create 
PaceAggregationDocument2 which does it more simply by making only four 
changes:

1) add an aggregation element
2) state that it requires an atom:head
3) state that it can contain any number of atom:feeds
4) state that it can be the document element

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 4 Feb 2005 00:45:06 -0800

> interoperability across the network, abstraction hurts.

Hmm, I am either suffering from a bad case of ennui at the moment
or I have just completely lost track of why atom needs a syndication
format in the first place, or perhaps both.  I think your summary
reflects the email discussion, not the content of those paces.

>   I would like it if the markup found in Atom documents was as close 
> as possible to a typical human mental model.

... of what?  Personally, I think that is the main problem with
all of the syndication formats -- they reify an interaction model
rather than a data model.

> The word "feed" has entered the vocabulary, even the non-geek 
> vocabulary, and the notion that there are "things" (entries, stories, 
> items, whatever) in "feeds" likewise.  Any attempt to abstract away 
> and generalize along the lines of "well, a feed is really an entry" or 
> "well, an entry is really a feed" or "really, they're all just web 
> resources and collections" is dangerously close to verging on 

Entry is a data model that easily fits the XML format, assuming
we ignore (for the moment) that entries and entry summaries are
actually quite different.  It would be nice if atom would define
a distinct data model for entry first, before it got tied up in
the details of how feeds are supposed to work.

Feed is an interaction model.  It has more to do with the pipe in
pipe-and-filter and the slash in client/server than the actual data
passed in syndication.  It carries a lot of semantic baggage, which
is fine when that is the baggage you want to define.  Other than
that, it is just data, and all data looks like an entry because
(so far) entry is completely undefined.

The purpose of PaceFeedRecursive is to simplify the feed data
format by specifying a particular method of answering the data
relationship question via XML containment.  With all due respect,
that is far more concrete a solution than what is currently in
the draft, much easier to specify, and easily implemented by
existing XML tools without the need for (or conflict with) RDF/XML.

My interest is in simplification, not abstraction.  For example,
the draft wastes a lot of text talking in the abstract about
various constructs rather than simply defining one element for
each of those constructs.  Likewise, it talks about documents
in a very abstract way, whereas it really should define entry
first (and completely) before even talking about feed documents.
It defines an element, atom:head, that serves no useful purpose
whatsoever (XML is ordered and there is nothing preventing the
syndication format from requiring that entries be provided last)
while at the same time making a mess out of atom extensibility.

Having said all that, I have no objection to withdrawing
PaceFeedRecursive and replacing it with a PaceHeadless
that simply removes atom:head.  My interest in atom is almost
all on the back-end (content management) and that is the one
piece that dirties a clean implementation via Jackrabbit
(or any CMS based on JSR 170).

I think one reason that Robert has been exploring various
alternatives is because atom is not clear on what an entry
should be, which is a separate issue that has very little
to do with the organization paces, and yet that is the one
capability that distinguishes atom from the other syndication
formats.  OTOH, if people really want just a syndication format
containing typical blog entries, then I strongly recommend
just using the proposed RSS 1.1 and move on to greener pastures.

Cheers,

Roy T. Fielding                            <http://roy.gbiv.com/>
Chief Scientist, Day Software              <http://www.day.com/>

---

From: Robert Sayre <[email-masked]>
Date: Fri, 04 Feb 2005 12:10:26 -0500

> the details of how feeds are supposed to work.

Agree. *looks for the well-formed log entry.

> relationship question via XML containment.  

Now that you've written PaceRemoveHeadElement (Note to Bob: it still 
does what you want), I think that is what will probably happen. 
Secondly, it would be easy to turn it into PaceFeedRecursive at some 
point in the future.

> each of those constructs. 

Person, Date, and Text constructs all save space because they avoid 
repetitive requirements, and they match up nicely with the Relax NG. I'm 
pretty sure the group has agreed to drop Service constructs, and mnot 
has already moved atom:id into an element definition for -06.

Robert Sayre

---

From: Tim Bray <[email-masked]>
Date: Fri, 04 Feb 2005 10:38:22 -0800

On Feb 4, 2005, at 9:10 AM, Robert Sayre wrote:

> repetitive requirements, and they match up nicely with the Relax NG. 
> I'm pretty sure the group has agreed to drop Service constructs, and 
> mnot has already moved atom:id into an element definition for -06.

Right, and Brent Simmons, one of the major implementors, published a 
blog entry talking about how much, as a programmer, he liked the 
"person" and "link" and so on constructs.  -Tim

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 4 Feb 2005 14:40:39 -0500

Robert Sayre wrote:
> Now that you've written PaceRemoveHeadElement [or, PaceHeadless]...
>(Note to Bob: it still does what you want), I think that is what
> will probably happen.
	As long as we can put the feed metadata into an Entry document or
instance of an entry I'm happy. I don't care what the tag is called. I
simply need to have an entry free and in the wild while still being able to
describe the feed from which it was cut. Atom over XMPP, aggregated feeds
ala PubSub or as search results, etc. even many forms of archiving will work
fine if PaceHeadless is accepted.

	Note: The proposal in PaceHeadless is essentially what I was
originally proposing back in the summer. (i.e. feed meta-data would be
"inherited" by entries unless that inheritance was explicitly over-ridden by
the presence of a <source-feed> or <head> element in the entry.) You may
remember that after a great deal of discussion the whole idea of hierarchy,
inheritance, and other similar metaphors was rejected as too complicated.
I'm pleased if we have now come to the conclusion that hierarchy and
"inheritance" from parent elements is a reasonable thing to specify.
However, I'm concerned that it seems like we've gotten to this point rather
casually and without as much discussion as we had when the group originally
rejected the idea. On the other hand, I think disagreed with the earlier
group consensus and think that this is where we should be. So, I won't shoot
myself in the foot by complaining any more.

		bob wyman

---

From: Graham <[email-masked]>
Date: Sat, 5 Feb 2005 01:20:14 +0000

-1

Putting everything in one group and requiring it to be first is useful, 
and also adds consistency to head-in-entry, as evidenced by the 
introduction of the feeder element. Also "feeder" is a horrible word.

Graham

---

From: Robert Sayre <[email-masked]>
Date: Fri, 04 Feb 2005 20:36:05 -0500

> introduction of the feeder element. Also "feeder" is a horrible word.

And "head" doesn't suck? I struggle to type a sentence on the subject 
without making an unintentional pun. "source-feed" is fine. The section 
on atom:head is totally confusing and bad--it's easier if you just do it 
the way PubSub does it *right now*.

Robert Sayre