Subject: NoIndex, again

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Tue, 19 Apr 2005 23:15:16 -0400

Hi,
I've recently ended up in argument about what to do with feeds that
don't want to be reproduced. I e-mailed Dave Winer in the hope of
getting some information about RSS end of things. That resulted in a
blog entry with interesting comments [1], and I now know that Creative
Commons has an RDF schema for describing licensing [2].

The only common feature I want to include, and haven't found, is the
"noindex" type of behavior (do not include in search engines). I
searched the archives of this list and found an old thread discussing
this very issue [3]. It seems to have fizzled out and I haven't found
anything more recent documents or discussions.

Was the issue simply forgotten or purposfully dropped?

In the RSS discussion, it was suggested by Roger Benningfield that
search eninges and syndication sites use atom:summary instead of
atom:content to avoid the noarchive issue. The rationale is that
summaries are meant to be reproduced, much like an abstract for a paper.

I'm not sure about nofollow, I think noindex is definitely needed. The
latter could be used to opt-out of services such as Feedster,
Technorati, and PubSub.

Thoughts and comments?

[1] http://www.reallysimplesyndication.com/2005/04/19#a445
[2] http://web.resource.org/cc/
[3] http://www.imc.org/atom-syntax/mail-archive/msg00183.html

Regards,
-Nikolas 'Atrus' Coukouma

---

From: Antone Roundy <[email-masked]>
Date: Tue, 19 Apr 2005 21:32:26 -0600

I believe we decided not to address licensing in the Atom core because 
the probability of not getting it quite right is too high, especially 
given the differences between laws in various jurisdictions around the 
world.  Having something in the core might require consuming 
applications to enforce licensing terms or make their authors liable in 
some jurisdictions, and we didn't want to create that rat's nest.

On Tuesday, April 19, 2005, at 09:15  PM, Nikolas 'Atrus' Coukouma

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Tue, 19 Apr 2005 23:50:11 -0400

Some pertinent quotes from the discussion two years ago:

The issue of bots can be deconstructed into privacy considerations
(how to specify what information should not be retained or used),
rights control (how to restrict ways in which the content can be used),
and resource use (how to specify the optimal access pattern).
It might be benefitial to analyze each issue separately.
-Misha Dynin

I think while all three are valid issues, here we should limit ourselves
to the first element only - replicating the meta tag behavior so that
the feed has a way to tell search sites what they should and should not
do with its contents.

Rights control, in my opinion, is best handled with some kind of
Creative Commons metadata. Resource use ties in to the ongoing thread
about manifests vs. full-content feeds, so is probably best discussed in
that thread.
-Maciej Ceglowski

And that's where the thread ends. I agree that rights control is
definitely something best handled by licensing data from some other
namespace (like the Creative Commons one). I'm not sure what happened
with the resource use part and it's not a concern of mine (at least not
for this).

The first issue (privacy considerations), which a few people expressed
interest in addressing, never was handled to my knowledge. If it was in
some other thread, I'd appreciate a pointer in that direction.

Regards,

Antone Roundy wrote:

> applications to enforce licensing terms or make their authors liable
> in some jurisdictions, and we didn't want to create that rat's nest.
>
> On Tuesday, April 19, 2005, at 09:15  PM, Nikolas 'Atrus' Coukouma wrote:
>
>>
>> Hi,

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Tue, 19 Apr 2005 23:50:16 -0400

The problem I'm having is with LiveJournal. They have 6.8 million
registered users, with 2.7 million active. We're talking about a
robots.txt of hundreds of megabytes. I imagine other blog hosting sites
(Blogger, Xanga) have a similar problem.

Also, robots.txt can only handle noindex. What about nofollow? Is
nofollow needed? Is the summary/content recommendation sufficient to
handle noarchive and nosnippet?

Regards,

James Robertson wrote:

> robots.txt solved this problem a long time ago
>
> At 11:15 PM 4/19/2005, you wrote:
>
>> Hi,
> <Talk Small and Carry a Big Class Library>
> James Robertson, Product Manager, Cincom Smalltalk
> http://www.cincomsmalltalk.com/blog/blogView
>

---

From: "John Panzer" <[email-masked]>
Date: Tue, 19 Apr 2005 22:09:25 -0700 (PDT)

Regarding the privacy facet:  True privacy probably requires access 
control, which requires authentication.  In fact this is one of the 
major use cases for authenticated Atom feeds in AOL Journals.

-John

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Wed, 20 Apr 2005 01:13:31 -0400

James Robertson wrote:

> At 11:38 PM 4/19/2005, you wrote:
>
> Why would it be that big?  Why wouldn't it just be a site wide policy?

We offer users a choice. There's a "bot blocking" preference to address
exactly this issue on LiveJournal: some users want to be in search
engines, others not. The discussion two years ago mentioned the figure
25% and I'm not sure how accurate that is.

When users have domains or subdomains (a paid account feature), then we
do produce an appropriate robots.txt file. However, only 1.5% of users
have that option available to them.

I can understand if Atom doesn't want to bother with a small number of
sites that have a large number of users. They are few and far between.

> well, IMHO, nofollow is pointless.

I'm not a fan myself.

>   The rest of it?  Which aggregators pay any attention?

I was under the impression that this mailing list was involved in
creating a standard that could then be implemented by developers. I've
already mentioned JournURL's policy (from Roger Benningfield).
LiveJournal has little policy in effect and it's something we are (or at
least I am) trying to figure out. LiveJournal both produces feeds (RSS
and Atom feeds are produced for all users) and consumes them (it acts as
an aggregator via "syndication accounts").

I intend to contact some of the other large sites, but wanted to know if
something already existed. I don't want re-invent the wheel and have
already done a fair amount of searching for existing ones. I also don't
want to just fabricate an element on my own when there's a mailing list
for discussing such things.

---

From: Walter Underwood <[email-masked]>
Date: Tue, 19 Apr 2005 22:18:45 -0700

A long time ago, I proposed a robots processing instruction that could
be used in any XML format. I can find that again.

An in-document robots directive is useful because it is controlled
by the document author, rather than by the webmaster.

"nofollow" is not particularly useful, because there is almost always
another path to the document. Still, it can be a polite hint to the 
robot that all the links in the doc are junk.

I would use exactly the model in the HTML robots meta tag, because:
a) that is what robots already know how to deal with, and b) it has
proven good enough.

wunder

--
Walter Underwood
Principal Architect, Verity

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Wed, 20 Apr 2005 01:41:08 -0400

>be used in any XML format. I can find that again.
This seems worth resurrecting. I'd rather see the element in a seperate
namespace so it
1) can be reused - most obviously, in other syndication formats, like a
flavor RSS
2) is not cluttering Atom

>a) that is what robots already know how to deal with, and b) it has
Speaking of reuse and HTML, in some ways I'd like to just borrow the
meta tag (from the XHTML namespace), but I think it's far to generic. An
element specifically for this purpose seems better.

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Wed, 20 Apr 2005 02:47:11 -0400

>>be used in any XML format. I can find that again.
Here it is, as saved by the Wayback Machine:
http://web.archive.org/web/20010407112036/http://homepages.go.com/~wunder0/robots-pi.html

I'm a tad concerned that it's not namespaced, but I can't think of a way
to handle that without idiosyncracies or being invalid.

It also doesn't address archiving, which isn't a large concern in a
syndication format. I'm not sure if it makes sense to delay
archive=yes/no until mid-processing.

Generally, it seems to be the right tool for the job. It should also be
discussed in detail elsewhere.

Unless someone has a good reason not to, let's move.

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 20 Apr 2005 02:56:24 -0400

Nikolas 'Atrus' Coukouma wrote:
> I now know that Creative Commons has an RDF schema for describing
> licensing
	We've been over this ground many times before. Read my post at:
http://bobwyman.pubsub.com/main/2005/03/lazyweb_query_a.html
	Creative commons licenses can only be used to *grant* permissions --
not restrict them. In fact, it is violation of the Creative Commons licenses
to implement systems that interpret CC licenses as prohibiting use. For
example, a CC "non-commercial" license does not forbid commercial use.
Rather, it simply fails to grant commercial use rights. Whether or not
commercial use is permitted depends on other factors -- such as local law,
etc. 
	To claim that something like a CC "non-commercial" license
prohibited the commercial use of content, you would have to argue that such
use was prohibited whether or not a CC license was present. You would have
to argue that NO content in RSS or Atom feeds could be used without an
explicit grant via Creative Commons or some other means. The result would be
a "poisoning of the stream" in that success of your assertion would
instantly shut down all use of RSS and Atom feeds which did not carry
Creative Commons licenses or other means of granting rights (i.e. 99.9% of
all feeds.)
	There is a strong argument that can be made that anyone who
publishes an RSS or Atom file without taking care to control access to the
feed has implicitly waived their right to restrict use of the data since the
usual and customary use of RSS and Atom files is to facilitate aggregation
and syndication.
	The Digital Rights Management space is papered with hundreds of
patents -- a number of which claim the use of XML to encode licenses and
many which make general claims concerning methods no matter what encoding
format might be used. Without evidence that someone has done a complete
patent search concerning whatever methods may be proposed to control use of
content (such as noindex extensions, etc.), it is unlikely that any
responsible developer would risk the potential patent infringements that
might result from implementing the method.

> opt-out of services such as Feedster, Technorati, and PubSub.
	To the best of my knowledge, there is no useful means by which the
services mentioned can be distinguished from any other form of aggregator.
(Note: I am CTO for PubSub.com. What have you got against us?) The PubSub
service, for instance, only reads RSS and Atom feeds (we do no HTML
scraping) and we produce feeds *only* on behalf of specific users. (with the
exception of a few "sample" feeds.) Thus, you can't distinguish what we do
for our users from what individual users do for themselves with aggregators
running on their own machines. (Of course, users often pay for personal
aggregators while PubSub is free. Which is more commercial?)

> it was suggested by Roger Benningfield that search eninges and
> syndication sites use atom:summary instead of atom:content to
> avoid the noarchive issue.
	Most feeds do not contain atom:summary elements. It is optional.
This is not a useful path to a solution. In any case, reliance on
atom:summary elements wouldn't help you with RSS files. If Atom has a rights
issue then RSS does as well. 

	If you want to have fine control over the use of your feeds, why
don't you use a blogging system that provides such control? For instance,
Yahoo!'s 360 service enables very fine control of access rights. Also,
services like LiveJournal, Typepad, etc. allow you to mark your posts as
"non-public" and as a result they don't get syndicated. The control you want
should be provided by your blogging software -- not embedded in the Atom or
RSS formats. RSS and Atom are formats for open and broad syndication. Any
effort to bring DRM into this world will only result in a poisoning of the
stream and a loss of the value of syndication to the millions who currently
rely on it.
	If you really want to focus on non-individual use of your stuff, why
don't you go after the sites that simply take RSS data and use it to build
HTML web sites? It is entirely possible that the implicit waiver concerning
content flowing through the syndication systems does not apply once the
content is removed from the system. (Note: This may be a *very* important
point... but is one for the lawyers, not us mere mortals.)
 
		bob wyman

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Wed, 20 Apr 2005 04:08:04 -0400

>http://bobwyman.pubsub.com/main/2005/03/lazyweb_query_a.html
I didn't really want to discuss the licensing mess here. The previous
discussion seemed to conclude that it was risky and out of scope. I just
wanted to note that some attempt had been made to make a
machine-readable description of licensing.

>might result from implementing the method.
Now the /success/ of Creative Commons is debatable. I don't think anyone
is going to try to implement DRM, in the usual locking sense, in a
syndication feed, but maybe I'm wrong. I'm only concerned with providing
guidelines to users so they have a better chance of knowing what they're
doing.

Thanks for pointing out the problem with even encoding licenses. I'll
definitely proceed with caution, if at all.

>(Note: I am CTO for PubSub.com. What have you got against us?)
>
I don't have anything against any of the services named, per-se. They're
just some of the largest and best search engines around, and therefore
the largest concerns of people trying to keep their content from being
indexed (for whatever reason).

> The PubSub
The area is definitely gray (definitely a maybe?). The best difference I
can come up with is that search/match services index large volumes of
content and individual users never intend to read most of it. I suppose
people could subscribe to 8,988,910 weblogs (number from Technorati)
personally and do the same thing, but it seems absurd to compare reading
70% of entries to reading 0.000001% (if that).

Similar arguments can be made with HTML search engines. Sure, you could
crawl the web yourself and "view" that information, but it's unlikely
you'll ever read a significant percentage of it (I'm sure specification
and manual authors feel that this is true for all content)

You also share the same content among all users. So do
online-aggregation services. I doubt there will ever be a definitive
line. I think there's a fuzzy philosophical one and you can decide which
one you're on.

>issue then RSS does as well.
I was unsatisfied with it as a solution. I think it makes a nice little
guidline in the rare cases it applies to and that's about it.

I've also already e-mailed Dave Winer to discuss the issue in RSS. I
even cited the discussion ;) I would certainly prefer a cross-format
solution and am quite happy to use Walter Underwoods' general solution
for XML content (mentioned elsewhere in this tree).

>stream and a loss of the value of syndication to the millions who currently
The point that "RSS and Atom are formats for open and broad syndication"
is an excellent one and something I've pointed out often. The question
was "is there any interest in providing minimal privacy elements in the
spec, comprable to what HTML provides" and apparently the answer is
"no." That's fine.

Part of what I've been wrestling with is the question of "are RSS and
Atom suited to aggregation when syndication is not desired?" LiveJournal
is what I spend time working on, and there's been an argument over there
about whether or not people want all of the content of all of their
public posts syndicated. It's being worked on at the application level,
and I wondered if there could be anything done once it's in one of these
standard XML formats. The current approach is to just not put the
information into a syndication-oriented format, and it seems to be the
right one.

>point... but is one for the lawyers, not us mere mortals.)
I concur. I'm also looking for something that helps people find (and
possibly be alerted to) copyright infringement and abuse. It's a
somewhat seperate battle.

My current feeling is that a "noarchive"-like element should be used to
indicate that the items should not persist any longer than they do in
the feed. I'm sure the issue will be explored sometime, and hopefully
before things get too unpleasant.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Apr 2005 09:32:10 -0700

We chose not to put things like this in the Atom core. Feel free to 
write an extension and discuss it here; there was certainly interest 
in many directions about grappling with the many intertwined issues 
that arise out of copyright, privacy, and so on.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Roger B." <[email-masked]>
Date: Wed, 20 Apr 2005 14:46:21 -0500

> This is not a useful path to a solution.

Bob: The fact that a summary is optional is precisely why it is a
useful approach.

It's all about working together, taking responsibility, and operating
in good faith. If a publisher provides a feed with nothing but inline
atom:content, it's reasonable for me to assume that my web-based
aggregator can display that content. If that publisher wants me to
only display an excerpt, then it is reasonable to expect said
publisher to insert an atom:summary into her feed.

> In any case, reliance on

There are two ways to approach the matter in RSS:

(1) Rely on description/content:encoded as an analog to summary/content.

(2) Evangelize the use of atom:summary within RSS.

--
Roger Benningfield
http://admin.support.journurl.com/

---

From: Kevin Marks <[email-masked]>
Date: Wed, 20 Apr 2005 21:47:28 -0700

> machine-readable description of licensing.

Note that a simpler form of expressing this is rel="license", as 
defined here:

http://developers.technorati.com/wiki/RelLicense

and adopted by Creative Commons:

http://lists.ibiblio.org/pipermail/cc-metadata/2004-February/000290.html

and generated by their license server along with the RDF form
http://creativecommons.org/license/

This seems a suitable atom-like way to express this (subject to Bob's 
caveats about ti being a positive grant rather than a restriction).

---

From: "Nikolas 'Atrus' Coukouma" <[email-masked]>
Date: Sun, 24 Apr 2005 03:56:47 -0400

I've revived the robot processing instruction at a (P)URL. It's been
rewritten/updated, too. The draft I wrote [2] attempts to address my
major concerns with the old version:
1) the "noarchive" instruction seems to be popular and is generally
useful; it is now included
2) you no longer have to specify all parameters. omitted values are
assumed to be "yes"
3) there's now room for extension by bots in specific realms (e.g.
Google's nosnippet)
4) links are no longer specified to be members of the XLink standard

I'm a bit concerned about the last two. Including un-namespaced
extensiblity creats the potential for conflicts and confusion. I don't
have a solution. At least limiting the values to "yes" and "no" prevents
people from sneaking content in.

I simply left "links" underspecified. Although the XLink standard has
stabilized,  it's not widely used. Many formats simply treat elements
and properties in their own namespaces as links (e.g. the XHTML anchor
element). My current philosophy is that links are whatever a bot follows.

Comments are not only welcome; they're encouraged.

[1] http://purl.org/net/robotspi/
[2] http://purl.org/net/robotspi/spec-20050423/