Subject: Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC

From: S Moonesamy <[email-masked]>
Date: Tue, 01 Mar 2022 02:15:44 -0800

Hello,
At 09:39 AM 28-02-2022, The IESG wrote:
>The IESG has received a request from an individual submitter to consider the
>following document: - 'Robots Exclusion Protocol'
>   <draft-koster-rep-06.txt> as Informational RFC
>
>The IESG plans to make a decision in the next few weeks, and solicits final
>comments on this action. Please send substantive comments to the
>[email-masked] mailing lists by 2022-04-07. Exceptionally, comments may

The draft has a "no derivative work" clause.  The authors of the 
draft could consider either of those two alternatives:

   (a) Renaming the document to: Stalworthy Computing, Ltd's Robots Exclusion
       Protocol;

   (b) Removing the "no derivative work" clause.

Regards,
S. Moonesamy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 1 Mar 2022 19:14:52 -0800

On Mon, Feb 28, 2022 at 2:30 PM John Levine <[email-masked]> wrote:

> Most importantly, the copyright license is broken. At the top it has
> the "no derivatives" license, which is fine, but it also has code
> sections in <CODE BEGINS>. The TLP specifically says that the code
> license only applies RFCs that use the regular license, not any other
> license. In this case the "code" sections are short snippets of sample
> robots files with made up names and paths so I would take out the code
> flags.
>

Thanks, and thanks to SM, for pointing this out.  I missed that it was in
there.  I'll take it up with the authors and come to a resolution before we
proceed beyond Last Call.

-MSK

---

From: Ted Hardie <[email-masked]>
Date: Wed, 2 Mar 2022 08:34:18 +0000

On Mon, Feb 28, 2022 at 5:56 PM Michael Richardson <[email-masked]>
wrote:

>
> It's good to see robots.txt coming to the IETF.
> I think that this is coming via AD sponsorship.  Was there a dispatch
> discussion/action that I missed?
>

It came to the ART list for discussion in 2019, if you're looking for some
of the previous background.

regards,

Ted

>
> I see that the document is Informational.
> I'm unclear if this document is identical to the 1996 description, or if
> there are changes against the 1996 version.
>
> Are there plans to revise the protocol further, and this is the first step?
>
> I wonder if 2.3.1.2.  Redirects should suggest the file be moved to
> /.well-known/robots.txt with a redirect, that perhaps shouldn't be counted.
> That would be a change which may be more than desired.
>
> (I was thinking on the weekend about some extensions that would fit nicely
> into robots.txt)
>
> --
> Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
>            Sandelman Software Works Inc, Ottawa and Worldwide
>
>
>
>
>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 8 Mar 2022 09:18:32 +0000

Hi Mark,

On Mon, Feb 28, 2022 at 10:56 PM Mark Nottingham <[email-masked]> wrote:

>
>
> > On 1 Mar 2022, at 9:29 am, John Levine <[email-masked]> wrote:
> >
> > the "no derivatives" license, which is fine,
>
> Ah - I missed, that, thanks for pointing it out.
>
> I'm uncomfortable leaving change control for a key interoperability
> mechanism in the search market in the hands of one competitor, yet blessing
> it as part of the IETF stream. I think the IETF as a whole should be
> uncomfortable with that too, given current competition enforcement trends.
>

Having the original author of the spec be the principal author here is a
bit of a bulwark against that, as I don't believe he is or would be
interested in handing change control over to Google. I also believe Gary
and the other authors have reached out to the rest of the relevant
community (though my change in employer means I know longer have the
relevant e-mails to cite).

On the more general topic of why this has the "no derivatives" clause,  I
understand your reluctance, but I think this is a case where the
combination is valid.  First, it's important to note that the specification
was brought to the IETF for substantive review, to make sure that the
elements it uses (like ABNF) were being used in the right way and to
eliminate any possibility of ambiguity.  From my perspective, that's been
very useful and it would not have occurred to the same extent had this gone
directly to the ISE.

However, this spec reflects operations which have been stable/backwards
compatible for a very long time.  Given that, it is important to the
community which deploys this that it be fairly difficult to amend.  One way
to achieve that would have been to make this standards track; that would
require standards action to update or obsolete it later.  When we discussed
that back at the beginning of this process, though, it was pretty clear
that some folks would use the working group discussion around that to try
to insert functionality that would result in breaking changes.  While it
would have been kind of unlikely for any of those to win out against the
need for maintaining interoperability, the result would have been a pretty
big increase in the amount of effort needed to get this published.

Another option for getting an archival spec with a high bar for change was
this one:  an IETF informational with a no-derivatives clause. That gave
the full benefit of IETF review and made the bar for amendment high enough
to allay the concerns of the original author and the relevant community.
It had this clause when Adam agreed to sponsor it and it has had it in
every iteration since, so I thought this was well understood.  As shepherd,
my apologies if it was not.

There is another option that gets the full set of characteristics needed:
AD sponsored on the standards track.  At the time this went through the
first set of discussions that was something folks had become very reluctant
to do.  If it is on the table, I personally believe that a standards track
document with the usual clauses would work as well.  Those can't be
superseded or amended without serious work and plenty of time for the
relevant community to chip in.

But, absent that, I think this kind of document is why BCP78 permits this
combination: documents which need and have received significant IETF review
but which also have a significant external community for whom the usual
clauses result in a risk of inappropriate later amendments. To put this
slightly differently, I think you'll see that this falls under the logic in
RFC 5378, Section 3, in the penultimate paragraph.

If you and the broader community prefer the standards track approach, now
would be a good time to let the sponsoring AD know.

regards,

Ted Hardie

>
> As such, I believe this draft should either a) drop the "no derivatives"
> clause (preferred), or b) seek publication on a different stream.
>
> Cheers,
>
> --
> Mark Nottingham   https://www.mnot.net/
>
> --
> last-call mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/last-call
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 8 Mar 2022 22:20:14 +1100

Hi Ted,

Thanks for the response, especially since this is not strictly your problem any more. Responses below.

> On 8 Mar 2022, at 8:18 pm, Ted Hardie <[email-masked]> wrote:
> 
> Hi Mark,
> 
>> I'm uncomfortable leaving change control for a key interoperability mechanism in the search market in the hands of one competitor, yet blessing it as part of the IETF stream. I think the IETF as a whole should be uncomfortable with that too, given current competition enforcement trends.
> 
> Having the original author of the spec be the principal author here is a bit of a bulwark against that, as I don't believe he is or would be interested in handing change control over to Google. I also believe Gary and the other authors have reached out to the rest of the relevant community (though my change in employer means I know longer have the relevant e-mails to cite).

Hmm. Because Google employees are co-authors in a joint work, my understanding is that they have the ability to publish derivative works in the future, at least in the US. If that's true, it does give them a form of change control -- or at least significant privilege (regarding the future of the spec) over other members of the community.

 (Obviously, one would need to talk to a copyright lawyer -- my understanding here is informed by a similar situation in a different venue)

> On the more general topic of why this has the "no derivatives" clause,  I understand your reluctance, but I think this is a case where the combination is valid.  First, it's important to note that the specification was brought to the IETF for substantive review, to make sure that the elements it uses (like ABNF) were being used in the right way and to eliminate any possibility of ambiguity.  From my perspective, that's been very useful and it would not have occurred to the same extent had this gone directly to the ISE. 

I find this a bit surprising -- surely it's possible to get adequate review for ISE documents without putting them into the IETF stream? Otherwise, the Independent stream would jeopardise the quality of the RFC Series overall... Was the ISE involved in this discussion?

> However, this spec reflects operations which have been stable/backwards compatible for a very long time.  Given that, it is important to the community which deploys this that it be fairly difficult to amend.  One way to achieve that would have been to make this standards track; that would require standards action to update or obsolete it later.  When we discussed that back at the beginning of this process, though, it was pretty clear  that some folks would use the working group discussion around that to try to insert functionality that would result in breaking changes.  While it would have been kind of unlikely for any of those to win out against the need for maintaining interoperability, the result would have been a pretty big increase in the amount of effort needed to get this published.

This is the rub -- depending on your definitions of "the community" and "some folks" in the statement above, the outcome might be completely reasonable and justified, or blatantly illegitimate. 

It's also notably out of step with the direction that pretty much all other Internet and Web standards are taking. HTML, DOM, and many other aspects of the platform have considerable requirements for stability and backwards compatibility, and yet they are not locked behind a no-derivatives clause. 

> Another option for getting an archival spec with a high bar for change was this one:  an IETF informational with a no-derivatives clause. That gave the full benefit of IETF review and made the bar for amendment high  enough to allay the concerns of the original author and the relevant community.  It had this clause when Adam agreed to sponsor it and it has had it in every iteration since, so I thought this was well understood.  As shepherd, my apologies if it was not.
> 
> There is another option that gets the full set of characteristics needed:  AD sponsored on the standards track.  At the time this went through the first set of discussions that was something folks had become very reluctant to do.  If it is on the table, I personally believe that a standards track document with the usual clauses would work as well.  Those can't be superseded or amended without serious work and plenty of time for the relevant community to chip in.  
> 
> But, absent that, I think this kind of document is why BCP78 permits this combination: documents which need and have received significant IETF review but which also have a significant external community for whom the usual clauses result in a risk of inappropriate later amendments. To put this slightly differently, I think you'll see that this falls under the logic in RFC 5378, Section 3, in the penultimate paragraph. 

Assuming that you're referring to this paragraph in s 3.3:

~~~
   The IETF has historically encouraged organizations to publish details
   of their technologies, even when the technologies are proprietary,
   because understanding how existing technology is being used helps
   when developing new technology.  But organizations that publish
   information about proprietary technologies are frequently not willing
   to have the IETF produce revisions of the technologies and then
   possibly claim that the IETF version is the "new version" of the
   organization's technology.  Organizations that feel this way can
   specify that a Contribution be published with the other rights
   granted under this document but may withhold the right to produce
   derivative works other than translations.
~~~

... then the question is whether the robots.txt format is really 'proprietary' technology, or whether it's a public good. Given its wide deployment and use as what amounts to an API for search engines to interoperate with Web sites, I struggle to see it as the former.

> If you and the broader community prefer the standards track approach, now would be a good time to let the sponsoring AD know.

To be clear, I think that this document is almost certainly a reasonable record of how the robots.txt format works today, and that its authors are acting in good faith. However, given the circumstances I'm concerned that from the 'outside', publishing the document in this manner won't look legitimate -- and therefore call into question the legitimacy of the IETF itself, in some eyes.

I think there are a few different ways we could address that while meeting the authors' goals.

0) I still believe that removing the no-derivatives clause is the most straightforward to do so. TCP, QUIC, HTTP, and many other Internet specifications remain stable and backwards compatible without the benefit of a no-derivatives clause; I don't see how robots.txt is different.

1) Alternatively, statements from at least some other search engines that they are aware of this work and do not object to it being published would change how this action is perceived considerably. Ideally, this would be represented by adding authors from other search engines to the document.

2) Or, since the document has now been reviewed for ABNF, nothing stops it from being switched to the Independent Stream with a title like "The Google Robots Exclusion Protocol" (to reflect its 'proprietary' nature).

Cheers,

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 8 Mar 2022 12:14:59 +0000

I normally would not have read this draft.

"You can't modify this document" goes so far against IETF practices and history that in my view it is enough to tell people to take it somewhere else if they are not willing to change it.

---

From: Ted Hardie <[email-masked]>
Date: Tue, 8 Mar 2022 16:04:41 +0000

Hi Mark,

Some comments in-line.

On Tue, Mar 8, 2022 at 11:20 AM Mark Nottingham <[email-masked]> wrote:

> Hmm. Because Google employees are co-authors in a joint work, my
> understanding is that they have the ability to publish derivative works in
> the future, at least in the US. If that's true, it does give them a form of
> change control -- or at least significant privilege (regarding the future
> of the spec) over other members of the community.
>
>  (Obviously, one would need to talk to a copyright lawyer -- my
> understanding here is informed by a similar situation in a different venue)
>

I am also not a copyright lawyer, and my understanding of joint authorship
is probably poor.  But I am pretty sure that deriving the right to create
derivative works isn't the aim here.  The aim was, in fact, to get a stable
reference for this and to make sure the bar for changing it was
significant.  The ISE generally does not publish updates to IETF-issued
documents (though they do publish commentary), and so going with the IETF
stream here is actually likely to make it harder for any later modification
to go forward without community attention (without leaving the RFC
publication process entirely, obviously).

> I find this a bit surprising -- surely it's possible to get adequate
> review for ISE documents without putting them into the IETF stream?
> Otherwise, the Independent stream would jeopardise the quality of the RFC
> Series overall... Was the ISE involved in this discussion?
>
>
I didn't mean to imply that ISE documents weren't generally adequate, but
is the ISE's choice about what review to seek.  Seeking public review was
part of the point here, and I continue to believe that's beneficial.

> This is the rub -- depending on your definitions of "the community" and
> "some folks" in the statement above, the outcome might be completely
> reasonable and justified, or blatantly illegitimate.
>
> As I said, I don't have the ability to cite previous discussions, but I
have asked the authors to reach out to the folks from that community so
that they can comment directly.  I hope that their doing so resolves this
issue for you.

> It's also notably out of step with the direction that pretty much all
> other Internet and Web standards are taking. HTML, DOM, and many other
> aspects of the platform have considerable requirements for stability and
> backwards compatibility, and yet they are not locked behind a
> no-derivatives clause.
>
>
Of course, the parallel to the robots.txt spec may or may not be strong,
given both when it was developed and the extent to which it was fostered by
a single individual.

>
> > Another option for getting an archival spec with a high bar for change
> was this one:  an IETF informational with a no-derivatives clause. That
> gave the full benefit of IETF review and made the bar for amendment high
> enough to allay the concerns of the original author and the relevant
> community.  It had this clause when Adam agreed to sponsor it and it has
> had it in every iteration since, so I thought this was well understood.  As
> shepherd, my apologies if it was not.
> >
> > There is another option that gets the full set of characteristics
> needed:  AD sponsored on the standards track.  At the time this went
> through the first set of discussions that was something folks had become
> very reluctant to do.  If it is on the table, I personally believe that a
> standards track document with the usual clauses would work as well.  Those
> can't be superseded or amended without serious work and plenty of time for
> the relevant community to chip in.
> >
> > But, absent that, I think this kind of document is why BCP78 permits
> this combination: documents which need and have received significant IETF
> review but which also have a significant external community for whom the
> usual clauses result in a risk of inappropriate later amendments. To put
> this slightly differently, I think you'll see that this falls under the
> logic in RFC 5378, Section 3, in the penultimate paragraph.
>
> ... then the question is whether the robots.txt format is really
> 'proprietary' technology, or whether it's a public good. Given its wide
> deployment and use as what amounts to an API for search engines to
> interoperate with Web sites, I struggle to see it as the former.
>
>
I think your reading of this is stricter than mine.  The paragraph says
"even when the technologies are proprietary", but I do not believe that
restricts the scope of the exception to those which are.  In this case, as
in the proprietary case, there is a concern about new versions which are
not backwards compatible without a pretty high bar for review.  This
paragraph makes it possible to use the no-derivatives clause in cases like
this. As I said in my previous message, I believe that a standards-track
approach with IETF change control would have a sufficiently high bar, so I
suspect that it would be fine as well.

>
> > If you and the broader community prefer the standards track approach,
> now would be a good time to let the sponsoring AD know.
>
> To be clear, I think that this document is almost certainly a reasonable
> record of how the robots.txt format works today, and that its authors are
> acting in good faith.

That's good to hear, and I certainly agree. I continue to shepherd the
document, despite my change in employer, because I am personally convinced
that getting this published as an RFC is a good thing.

However, given the circumstances I'm concerned that from the 'outside',
> publishing the document in this manner won't look legitimate -- and
> therefore call into question the legitimacy of the IETF itself, in some
> eyes.
>
> 0) I still believe that removing the no-derivatives clause is the most
> straightforward to do so. TCP, QUIC, HTTP, and many other Internet
> specifications remain stable and backwards compatible without the benefit
> of a no-derivatives clause; I don't see how robots.txt is different.
>
> TCP, QUIC, HTTP and many others are developed within the context of the
IETF.  This has not historically been developed here, and so I believe that
these are not the strong parallels you see.

1) Alternatively, statements from at least some other search engines that
> they are aware of this work and do not object to it being published would
> change how this action is perceived considerably. Ideally, this would be
> represented by adding authors from other search engines to the document.
>
> I have asked the authors to make sure their colleagues who have reviewed
the work chime in here.  I think adding authors is not necessary and it
would go against what the RFC Series editor has previously stated for the
role of an author:

https://www.rfc-editor.org/pipermail/rfc-interest/2015-May/008869.html

2) Or, since the document has now been reviewed for ABNF, nothing stops it
> from being switched to the Independent Stream with a title like "The Google
> Robots Exclusion Protocol" (to reflect its 'proprietary' nature).
>
>
As I noted before, I think you are keying off the word proprietary a bit
too much, and it absolutely is not Google's Robots Exclusion Protocol;
Martijn's work on this was released two years before Google existed, as I
am sure you know.

regards,

Ted Hardie

---

From: Ted Hardie <[email-masked]>
Date: Tue, 8 Mar 2022 16:11:47 +0000

It was rapidly pointed out to me that my statement here:

> these are not the strong parallels you see.
was inappropriately vague about the meaning of "are developed".  I meant
"continued work on these is within the context of the IETF."  I apologize
for any lack of clarity.

regards,

Ted

---

From: "John R Levine" <[email-masked]>
Date: 8 Mar 2022 11:59:20 -0500

>> uncomfortable with that too, given current competition enforcement trends.

Putting on my trustee hat, I don't think this can be an IETF document 
without IETF change control.  RFC 5378 says

       The right to produce
    derivative works, in addition to translations, is required for all
    IETF Standards Track documents and for most IETF non-Standards Track
    documents.  There are two exceptions to this requirement: documents
    describing proprietary technologies and documents that are
    republications of the work of other standards organizations.

If it's a proprietary technology, Mark is right.  If it's not, we need
change control.

Another possibility would be to move it to the Independent stream if Eliot agrees.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Ted Hardie <[email-masked]>
Date: Tue, 8 Mar 2022 17:33:49 +0000

On Tue, Mar 8, 2022 at 4:59 PM John R Levine <[email-masked]> wrote:

> I took that particular trustee hat off some time ago, but in my personal
opinion, I think you are construing this too narrowly.

One, this is not requesting a standards track document (if it were, the
language would be different, as has been discussed).   Second, the
penultimate paragraph and the ultimate paragraph of that same section go
into considerable additional detail on the motivations for permitting the
non-standards track case.   I believe that this document falls within
them.  I've already discussed with Mark why I think the first applies, but
I'll also note that there are aspects which are closer to the second case
than the first.  This RFC is a restatement of the work that was maintained
by Martijn at https://www.robotstxt.org/ for a couple of decades (
https://www.greenhills.co.uk/posts/robotstxt-25/ for a bit of history).  It
is in use by over 500 million websites, and having a stable, archival
reference would be a useful thing.  Making it available as an RFC does
that, and I hope you agree.

We could argue about whether robotstxt.org is an SDO or whether, if it is
not, whether Martijn is the owner of the specification (case 2 or case 1),
but I think that's not very productive.  We have this exception to deal
with cases where it is useful to get IETF review and approval of documents
which originated from outside and which serve the common interest.  We've
had the IETF review, and the specification has improved as a result.  Going
through with the rest of the process makes the most sense to me, but, as
I've reiterated, I think the standards track would also do so.

If you support that option, it might be useful to say so (obviously, not
with a Trustee hat on).

regards,

Ted

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 8 Mar 2022 17:38:40 +0000

Can someone – probably one of the authors, not Ted – why they feel it is necessary to not allow derivative works?

In the past there have been many RFC’s that documented existing cryptographic algorithms: Blake2, ChaCha/Poly, etc. None of them say no derivatives.

---

From: Scott Bradner <[email-masked]>
Date: Tue, 8 Mar 2022 12:40:51 -0500

fwiw - basically supporting what John said

the intent was to be able to publish, for information only, company/SDO documents within the IETF process -
of course the IETF would not have change control over most such documents 
but the intent was to not require that all such documents go through the independent stream  

and the intent was not that just any document would qualify but that would be up to the WG/IESG

in any case, all standards track documents must be under IETF change control and cannot include the 
"no derivative works" label

Scott

> On Mar 8, 2022, at 11:59 AM, John R Levine <[email-masked]> wrote:
> 
> Putting on my trustee hat, I don't think this can be an IETF document without IETF change control.  RFC 5378 says
> 
>      The right to produce

---

From: Scott Bradner <[email-masked]>
Date: Tue, 8 Mar 2022 12:45:32 -0500

my previous note sent before seeing Ted's - but they seem to b win general agreement - an IETF WG (or other IETF process)
can publish a "no derivative works" if they think its the right thing to do & the IESG agrees

Scott

> On Mar 8, 2022, at 12:40 PM, Scott Bradner <[email-masked]> wrote:
>

---

From: John C Klensin <[email-masked]>
Date: Tue, 08 Mar 2022 12:59:05 -0500

Scott, 

You (or John) might have added one thing, which is that at least
as things have evolved, when we publish a company document under
that exception (whether it received IETF review and changes were
made during the process or not), those documents not only get
very clear text in the Abstract and/or Introduction explaining
what they are but get titles like "BigCo's Protocol for ..." as
another way to show who has the final say on the spec and where
change control lies.  And, whether it might apply here or not
and AFAIK, we have never recognized either an informal
organization that consists of people working together on one
protocol or an industry consortium with no outside members as an
SDO.

   john

--On Tuesday, March 8, 2022 12:40 -0500 Scott Bradner
<[email-masked]> wrote:

> the intent was to be able to publish, for information only,
> company/SDO documents within the IETF process - of course the
> IETF would not have change control over most such documents 
> in any case, all standards track documents must be under IETF
> change control and cannot include the  "no derivative works"
> label
> 
> Scott
> 
>>>> I'm uncomfortable leaving change control for a key
>>>> interoperability mechanism in the search market in the
>>>> hands of one competitor, yet blessing it as part of the
>>>> IETF stream. I think the IETF as a whole should be
>> Putting on my trustee hat, I don't think this can be an IETF
>> document without IETF change control.  RFC 5378 says
>> 
>>      The right to produce
>>   derivative works, in addition to translations, is required
>>   for all IETF Standards Track documents and for most IETF
>>   non-Standards Track documents.  There are two exceptions to
>>   this requirement: documents describing proprietary
>>   technologies and documents that are republications of the
>>   work of other standards organizations.
>> 
>> If it's a proprietary technology, Mark is right.  If it's
>> not, we need change control.
>> 
>> John Levine, [email-masked], Taughannock Networks,
>> Trumansburg NY Please consider the environment before reading
>> this e-mail. https://jl.ly
>> 
>> -- 
>> last-call mailing list
>> [email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 8 Mar 2022 14:51:59 -0500

> Can someone – probably one of the authors, not Ted – why they feel it is necessary to not allow derivative works?

This seems like a reasonable next step.

I understand about documenting proprietary formats, I did that in RFC 
6143, but this doesn't feel like the same situation.

Regards,

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 9 Mar 2022 10:09:09 +0900

> In the past there have been many RFC’s that documented existing cryptographic algorithms: Blake2, ChaCha/Poly, etc. None of them say no derivatives.

Yes indeed. I think the IETF is quite good at distinguishing when it 
makes sense to describe existing practice (as in this case) and when it 
makes sense to put in a lot of work and discussions (e.g. quic, WEBRTC,...).

The reason it makes sense to allow derivative works is that sometimes 
(not necessarily in this case, but we don't know in advance), things can 
change.

I don't like to guess, but it could be that the authors some experience 
(or hearsay from friends) with another technology standardized in the 
IETF where there was a lot of discussion, and incorrectly generalized 
from that.

Another guess would be that there are actual differences between search 
engines, and the authors were afraid to open this Pandora's box. I don't 
think this is the case, but if it were, it wouldn't be the IETF's job to 
"take sides". Some confirmation from other search engines would be 
desirable.

There's also the point that while the spec text is affected by 
copyright, the technical content isn't. This is an argument that works 
both ways, but the most important part I see in it is that it's not 
worth for the IETF to make an exception to its general policy of 
allowing derivative works.

Regards,   Martin.

---

From: "John R Levine" <[email-masked]>
Date: 8 Mar 2022 21:28:12 -0500

> The reason it makes sense to allow derivative works is that sometimes (not 
> necessarily in this case, but we don't know in advance), things can change.

There are in fact some widely used extensions to the format described in 
the draft.  Many search engines recognize "Sitemap:" with the location of 
a site map, and "Crawl-delay:" to slow down crawling.  On the other hand, 
a lot don't recognize # $ * in sec 2.2.3 as special.  I don't think we 
need to fix these to publish the draft, but it wouldn't be absurd to 
update it in light of experience.

> I don't like to guess, but it could be that the authors some experience (or 
> hearsay from friends) with another technology standardized in the IETF where 
> there was a lot of discussion, and incorrectly generalized from that.

Someone already suggested moving the file to a different location, which 
considering that all robots.txt files have been in the same place for 25 
years, seems like a bad idea.

Regards,

---

From: George Michaelson <[email-masked]>
Date: Wed, 9 Mar 2022 12:49:23 +1000

Limit copyright restricted republication to informative/experimental.
Never PS, STD, arguably never normative directions. If the document
has normative form, then some kind of prequel stating the
non-normative outcome due to copyright?

If you want to provide normative protocol conforming advice to a
community about structure of bits, and bits on a wire, then requiring
no change in the standards body is nonsensical.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 9 Mar 2022 14:15:05 +0000

> I don't like to guess, 

Let's not guess, let's ask the authors directly.  

> In the past there have been many RFC’s that documented existing cryptographic algorithms: Blake2, ChaCha/Poly, etc. None of them say no derivatives.

Why do you want no-derived-changes?  Given that the IETF almost never does this, I would expect this to be a hard requirement to justify.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 09 Mar 2022 13:32:35 -0500

Mark Nottingham <[email-masked]> wrote:
    ted> However, this spec reflects operations which have been
    ted> stable/backwards compatible for a very long time.  Given that, it is
    ted> important to the community which deploys this that it be fairly
    ted> difficult to amend.  One way to achieve that would have been to make
    ted> this standards track; that would require standards action to update
    ted> or obsolete it later.  When we discussed that back at the beginning
    ted> of this process, though, it was pretty clear  that some folks would
    ted> use the working group discussion around that to try to insert
    ted> functionality that would result in breaking changes.  While it would
    ted> have been kind of unlikely for any of those to win out against the
    ted> need for maintaining interoperability, the result would have been a
    ted> pretty big increase in the amount of effort needed to get this
    ted> published.

    > reasonable and justified, or blatantly illegitimate.

I agree with Mark.

Ted seems to be describing some kind of tyranny of full consensus.

Had this gone into a WG, and the WG has been chartered not to break anything,
then I think that we would respect that, and the WG, and WG chairs refused,
then the IESG would make it the case.

So I don't really buy Ted's argument, sorry.

Based upon Ted's comments, I think that maybe we shouldn't publish this.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 09 Mar 2022 15:47:19 -0500

> years, seems like a bad idea.

Since redirects are explicitely supported (but limited), we could trivially encourage
people to change the location, provided they leave a redirect.

---

From: Martijn Koster <[email-masked]>
Date: Fri, 11 Mar 2022 10:23:18 +0000

> On 9 Mar 2022, at 14:15, Salz, Rich <[email-masked]> wrote:
> 
>> I don't like to guess, 
> 

Gary, why was this added? Having looked at the discussion in RFC3667 sections 5.2 and 7.3 it seems that this is appropriate for re-publishing documents from other standard bodies, and to protect proprietary technologies. The robots.txt is based on a long-standing industry practice, defined by a historical specification and de-facto (diverging) implementations, but not a standard body. And the proprietary technology protection doesn’t apply either.
So I agree it should not be there.

— Martijn

---

From: S Moonesamy <[email-masked]>
Date: Fri, 11 Mar 2022 04:52:10 -0800

Hi Ted,
At 01:18 AM 08-03-2022, Ted Hardie wrote:
>On the more general topic of why this has the "no derivatives" 
>clause,  I understand your reluctance, but I think this is a case 
>where the combination is valid.  First, it's important to note that 
>the specification was brought to the IETF for substantive review, to 
>make sure that the elements it uses (like ABNF) were being used in 
>the right way and to eliminate any possibility of ambiguity.  From 
>my perspective, that's been very useful and it would not have 
>occurred to the same extent had this gone directly to the ISE.

I read that the ISE did not have the expertise to review elements 
such as ABNF.  It took me a few minutes to find the ABNF is Section 2 
of RFC 8905.

>However, this spec reflects operations which have been 
>stable/backwards compatible for a very long time.  Given that, it is 
>important to the community which deploys this that it be fairly 
>difficult to amend.  One way to achieve that would have been to make 
>this standards track; that would require standards action to update 
>or obsolete it later.  When we discussed that back at the beginning 
>of this process, though, it was pretty clear that some folks would 
>use the working group discussion around that to try to insert 
>functionality that would result in breaking changes.  While it would 
>have been kind of unlikely for any of those to win out against the 
>need for maintaining interoperability, the result would have been a 
>pretty big increase in the amount of effort needed to get this published.

The was an email on 24 February with the following paragraph: "The 
IETF prides itself on its open process and transparent communication, 
with one of our core principles being our commitment to making all 
materials related to our standards process and other activities 
publicly available."  I was unable to find the discussion which 
occurred at the beginning of the process.

>Another option for getting an archival spec with a high bar for 
>change was this one:  an IETF informational with a no-derivatives 
>clause. That gave the full benefit of IETF review and made the bar 
>for amendment high enough to allay the concerns of the original 
>author and the relevant community.  It had this clause when Adam 
>agreed to sponsor it and it has had it in every iteration since, so 
>I thought this was well understood.  As shepherd, my apologies if it was not.

 From what I understand, the "IETF informational" imprimatur isn't 
about creating archival specifications.

>this combination: documents which need and have received significant 
>IETF review but which also have a significant external community for 
>whom the usual clauses result in a risk of inappropriate later 
>amendments. To put this slightly differently, I think you'll see 
>that this falls under the logic in RFC 5378, Section 3, in the 
>penultimate paragraph.
>
>If you and the broader community prefer the standards track 
>approach, now would be a good time to let the sponsoring AD know.

The following sentence is from RFC 5378: "The IETF has historically 
encouraged organizations to publish details of their technologies, 
even when the technologies are proprietary, because understanding how 
existing technology is being used helps when developing new 
technology."  There is information on the web to understand how 
robots.txt is being used.  Why is an IETF RFC with a "no derivative" 
clause useful in this context?

Regards,
S. Moonesamy

---

From: Gary Illyes <[email-masked]>
Date: Mon, 14 Mar 2022 22:04:53 +0100

All, we had this boilerplate since the very first draft, so I'm rather
amused it only surfaced now as an issue. We thought we need the
current "no modifications, no derivatives" boilerplate so consumers
and users of the Robots Exclusion Protocol (REP), in particular
webmasters and search engines, don't end up with
backwards-incompatible changes in the draft.  As Martijn
mentioned, the REP has been around for a very long time, over
450 million hosts rely on it in addition to search engines, so we
feel rather protective about it.

With all that said, it's clear that the current boilerplate is a no-go
and, based on some sidebar chats, our worries perhaps unfounded,
so we can switch the status section to the "IETF Informational w/
consensus" one
(https://www.rfc-editor.org/materials/status-memos.txt).

Additionally, addressing Mark's comment about an ACK from other
consumers of the REP, I'll put out a public request on an official
Google social media account inviting other search engines to reply
here. We did check in with them before submitting the draft as well
as when we made notable changes (v5 -> v6), but I agree that the
other consumers of the REP should also at least acknowledge the
publication of this draft, considering how important it is in general.

Let me know if I missed some comments that need my attention.

Gary

On Fri, Mar 11, 2022 at 11:23 AM Martijn Koster <[email-masked]>
wrote:

> >> Can someone – probably one of the authors, not Ted – why they feel it
> is necessary to not allow derivative works?
> >
> >> In the past there have been many RFC’s that documented existing
> cryptographic algorithms: Blake2, ChaCha/Poly, etc. None of them say no
> derivatives.
> >
> > Why do you want no-derived-changes?  Given that the IETF almost never
> does this, I would expect this to be a hard requirement to justify.
>
> Gary, why was this added? Having looked at the discussion in RFC3667
> sections 5.2 and 7.3 it seems that this is appropriate for re-publishing
> documents from other standard bodies, and to protect proprietary
> technologies. The robots.txt is based on a long-standing industry practice,
> defined by a historical specification and de-facto (diverging)
> implementations, but not a standard body. And the proprietary technology
> protection doesn’t apply either.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 15 Mar 2022 09:52:15 -0700

> publication of this draft, considering how important it is in general.

If the authors are prepared to make this change, I'd like to hear why "IETF
Standards Track w/consensus" is (or should be) off the table, especially if
other search engines are going to indicate support.

-MSK

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Mar 2022 08:24:19 +1100

> On 16 Mar 2022, at 3:52 am, Murray S. Kucherawy <[email-masked]> wrote:
> 
> If the authors are prepared to make this change, I'd like to hear why "IETF Standards Track w/consensus" is (or should be) off the table, especially if other search engines are going to indicate support.

That's reasonable, if support eventuates. If it doesn't, it would be pretty awkward -- I think we'd want at least some level of review or at least acknowledgement by other major implementers before declaring it Standards Track.

AIUI a status change would necessitate another IETF LC, so that might be the best way to solicit that support. Make it a long one to give other implementations a meaningful way to respond -- say, four weeks. 

If the IESG doesn't hear from at least two or three other stakeholder implementations (in its estimation), it could still be published at Informational, and transitioned to Standards Track if/when that support eventuates.

Make sense?

Cheers,

---

From: "John Levine" <[email-masked]>
Date: 15 Mar 2022 22:05:13 -0400

It appears that Mark Nottingham  <[email-masked]> said:
>> If the authors are prepared to make this change, I'd like to hear why "IETF Standards Track w/consensus" is (or
>should be) off the table, especially if other search engines are going to indicate support.
>
>That's reasonable, if support eventuates. If it doesn't, it would be pretty awkward -- I think we'd want at least
>some level of review or at least acknowledgement by other major implementers before declaring it Standards Track.

A lot of search engines describe how they handle robots.txt. For example,

Yahoo https://help.yahoo.com/kb/SLN22600.html

Bing https://www.bing.com/webmasters/help/how-to-create-a-robots-txt-file-cb7c31ec

Yandex https://yandex.com/support/webmaster/controlling-robot/robots-txt.html

Baidu https://www.baidu.com/search/robots_english.html

DataForSEO https://dataforseo.com/dataforseo-bot

Petal Search https://webmaster.petalsearch.com/site/petalbot

Mojeek https://www.mojeek.com/bot.html

(Those last three I never heard of but their spiders show up in my logs.)

>From what I can see, the current draft is OK, but I would make it clearer that the * and $ metacharacters
aren't widely supported, and I would add Crawl-Delay and Siteindex directives which are.

R's,
John

---

From: Gary Illyes <[email-masked]>
Date: Fri, 18 Mar 2022 11:11:22 +0100

Adding to this thread Andrey Zosimov (Yandex) and Fabrice Canel (Bing) to
confirm the draft is acceptable for their respective search engines.

https://datatracker.ietf.org/doc/html/draft-koster-rep-06

Based on the feedback we received so far, we're not going to  have more
changes to the draft other than the boilerplate change discussed in this
thread, and the example and language corrections in the ART review.

On Tue, 15 Mar 2022 at 22:24, Mark Nottingham <[email-masked]> wrote:

> > If the authors are prepared to make this change, I'd like to hear why
> "IETF Standards Track w/consensus" is (or should be) off the table,
> especially if other search engines are going to indicate support.
>
> That's reasonable, if support eventuates. If it doesn't, it would be
> pretty awkward -- I think we'd want at least some level of review or at
> least acknowledgement by other major implementers before declaring it
> Standards Track.
>
> AIUI a status change would necessitate another IETF LC, so that might be
> the best way to solicit that support. Make it a long one to give other
> implementations a meaningful way to respond -- say, four weeks.
>
> If the IESG doesn't hear from at least two or three other stakeholder
> implementations (in its estimation), it could still be published at
> Informational, and transitioned to Standards Track if/when that support
> eventuates.
>
> Make sense?
>
> Cheers,
>
>
> --
Thanks,
Gary