Subject: Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC

From: Michael Richardson <[email-masked]>
Date: Mon, 28 Feb 2022 12:56:42 -0500

It's good to see robots.txt coming to the IETF.
I think that this is coming via AD sponsorship.  Was there a dispatch
discussion/action that I missed?

I see that the document is Informational.
I'm unclear if this document is identical to the 1996 description, or if
there are changes against the 1996 version.

Are there plans to revise the protocol further, and this is the first step?

I wonder if 2.3.1.2.  Redirects should suggest the file be moved to
/.well-known/robots.txt with a redirect, that perhaps shouldn't be counted.
That would be a change which may be more than desired.

(I was thinking on the weekend about some extensions that would fit nicely
into robots.txt)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 28 Feb 2022 10:50:22 -0800

> discussion/action that I missed?

It's a question of inheritance; AD sponsorship was selected for this
document before my term as an Area Director started.  I'm not aware of
whether it was discussed at DISPATCH prior to that, but if so, it would've
been quite a while ago now.

-MSK

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 1 Mar 2022 08:25:58 +1100

Thanks to the authors for doing this work; having it documented in an IETF-stream RFC would indeed be a good thing.

I'd be *much* more comfortable doing so if we had public indication from other consumers of this format (especially: competing search engines) that the definition is correct and interoperable with their implementations.

Cheers,

> On 1 Mar 2022, at 4:39 am, The IESG <[email-masked]> wrote:
> 
> 
> The IESG has received a request from an individual submitter to consider the
> following document: - 'Robots Exclusion Protocol'
>  <draft-koster-rep-06.txt> as Informational RFC
> 
> The IESG plans to make a decision in the next few weeks, and solicits final
> comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2022-04-07. Exceptionally, comments may
> be sent to [email-masked] instead. In either case, please retain the beginning
> of the Subject line to allow automated sorting.
> 
> Abstract
> 
> 
>   This document specifies and extends the "Robots Exclusion Protocol"
>   method originally defined by Martijn Koster in 1996 for service
>   owners to control how content served by their services may be
>   accessed, if at all, by automatic clients known as crawlers.
> 
> 
> 
> 
> The file can be obtained via
> https://datatracker.ietf.org/doc/draft-koster-rep/
> 
> 
> 
> No IPR declarations have been submitted directly on this I-D.
> 
> 
> 
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

--
Mark Nottingham   https://www.mnot.net/

---

From: "John Levine" <[email-masked]>
Date: 28 Feb 2022 17:29:30 -0500

>It's good to see robots.txt coming to the IETF.

Agreed, also agree with Mnot's question whether we have reports from
other search engines that they follow this spec. Based on my
experience looking at my web server's log files and tweaking the
robots.txt files and looking at the web sites where they explain their
crawling practices, I think they do, but surely we know people at a few
other search engines.  I'd be particularly interested to hear who
interprets the * and $ pattern metacharacters.

Section 2.2.2 has this example of a path with a Unicode character:

   | /foo/bar/U+E38384 | /foo/bar/%E3%83%84    | /foo/bar/%E3%83%84    |

There is no U+E38384 character, but the UTF-8 version of the Japanese
character U+30C4 is hex E3 83 84 so I'm guessing that's what they meant.

The "Crawl-Delay" line is ignored by Google but followed by many other
search engines such as Bing and Yandex. I would describe it, with a
note that only some spiders use it.

Most importantly, the copyright license is broken. At the top it has
the "no derivatives" license, which is fine, but it also has code
sections in <CODE BEGINS>. The TLP specifically says that the code
license only applies RFCs that use the regular license, not any other
license. In this case the "code" sections are short snippets of sample
robots files with made up names and paths so I would take out the code flags.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 28 Feb 2022 17:35:01 -0500

>there are changes against the 1996 version.

It adds provision for encoding UTF-8 characters in URLs and says
patterns can have * and $ metacharacters and # comments.

>That would be a change which may be more than desired.

That seems like a poor idea for a widely implemented 25 year old spec.  But
see the "no derivatives" license on the draft.

R's,
John

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 1 Mar 2022 09:55:12 +1100

> On 1 Mar 2022, at 9:29 am, John Levine <[email-masked]> wrote:
> 
> the "no derivatives" license, which is fine,

Ah - I missed, that, thanks for pointing it out. 

I'm uncomfortable leaving change control for a key interoperability mechanism in the search market in the hands of one competitor, yet blessing it as part of the IETF stream. I think the IETF as a whole should be uncomfortable with that too, given current competition enforcement trends.

As such, I believe this draft should either a) drop the "no derivatives" clause (preferred), or b) seek publication on a different stream.

Cheers,

---

From: "John R Levine" <[email-masked]>
Date: 28 Feb 2022 18:10:51 -0500

> I'm uncomfortable leaving change control for a key interoperability mechanism in the search market in the hands of one competitor, yet blessing it as part of the IETF stream. I think the IETF as a…

The main author is Martijn Koster who invented robots.txt in 1996.

I see in Google's blog that they want it to be an Internet Standard, which 
is of course not compatible with a "no derivatives" license.  So I concur 
with the advice that they need to change the license.

https://developers.google.com/search/blog/2019/07/rep-id

Still think they should take out <CODE BEGINS> on the examples.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Rob Sayre <[email-masked]>
Date: Mon, 28 Feb 2022 15:24:18 -0800

On Mon, Feb 28, 2022 at 2:55 PM Mark Nottingham <[email-masked]> wrote:

>
> As such, I believe this draft should either a) drop the "no derivatives"
> clause (preferred), or b) seek publication on a different stream.
>

I agree with this sentiment, but I have an additional concern. I am not
sure the MUST/SHOULD requirements are consistent. I think the draft should
include examples that test every single requirement (looks like there are
about 20-30).

thanks,
Rob