Subject: [rfc-i] Wrong Internet search results for new RFCs

From: Toerless Eckert <[email-masked]>
Date: Tue, 3 May 2022 03:23:20 +0200

When i search for RFC9148 on google or bing, i do only get

https://www.rfc-editor.org/rfc/authors/rfc9148.html

And of course the URL already does not exist anymore.
And yes, i did at least send a note to google.

I assume that google/bing get these URLs from the searchable directory

https://www.rfc-editor.org/rfc/authors/

Alas, i am not on top of all those HTTP headers that could be set
to prohibit or discourage caching information from this directy
in search engines. But if there i an easy fix by better config
on the www.rfc-editor.org webserver, that would be nice.

Cheers
    toerless

_______________________________________________
rfc-interest mailing list
[email-masked]
https://mailman.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: "John Levine" <[email-masked]>
Date: 2 May 2022 22:17:18 -0400

>https://www.rfc-editor.org/rfc/authors/

No, that's not how search engines work. They only add working URLs to
their index, and this is particularly strange since I helped the RPC
put site maps on the rfc-editor site that have the right URLs for all
of the RFCs.

When I do a search I see the nonexistent authors page and also a bunch
of pages at https://sandbox-ng.ietf.org/.  This suggests that something got
spidered very strangely during the datatracker upgrade.

I can see if I can ask the RPC to resubmit the site index which should help
it clean out the bad rfc-editor URLs.

R's,
John

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 3 May 2022 05:42:24 +0200

On Mon, May 02, 2022 at 10:17:18PM -0400, John Levine wrote:
> their index,

Sure, but they need to find the URL first, and given how these "authors"
URLs are typically private-email used, i wouldn't know where the
search engines would find he URLs other than from the index URL.

> and this is particularly strange since I helped the RPC

The site-map would list a URL like https://www.rfc-editor.org/rfc/authors/rfc9148.html
given how its temporary... ?!

> it clean out the bad rfc-editor URLs.

Thanks!
    Toerless

---

From: Julian Reschke <[email-masked]>
Date: Tue, 3 May 2022 07:20:48 +0200

Am 03.05.2022 um 05:42 schrieb Toerless Eckert:
> ...

FWIW, this shows that it is a bad idea to name a publicly readable
resource RFCNNNN until the moment it get's published (so I believe the
RFC Editor should modify their naming convention).

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 May 2022 07:28:41 +0200

> RFC Editor should modify their naming convention).

+1

➔

>> https://www.rfc-editor.org/rfc/authors/rfc9148-to-be.html

And, after publication, this needs to have a permanent redirect to

>> https://www.rfc-editor.org/rfc/rfc9148.html

(Similar for the other presentation forms.)

Grüße, Carsten

---

From: Christian Huitema <[email-masked]>
Date: Mon, 2 May 2022 23:06:50 -0700

> (Similar for the other presentation forms.)

Uh, No. The spiders and indexers are still going to pick "rfc9148" from 
"https://www.rfc-editor.org/rfc/authors/rfc9148-to-be.html". If you want 
safety, you need to obfuscate that some more. Maybe you can get away with:

https://www.rfc-editor.org/rfc/authors/nine-one-four-eight-to-be.html

Maybe. But the ways of the spiders are unpredictable.

-- Christian Huitema

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 May 2022 09:26:45 +0200

On 2022-05-03, at 08:06, Christian Huitema <[email-masked]> wrote:
> 
> Uh, No. The spiders and indexers are still going to pick "rfc9148" from "https://www.rfc-editor.org/rfc/authors/rfc9148-to-be.html”.

But that’s OK, because there will be a permanent redirect as soon as the real thing is available.

> If you want safety, you need to obfuscate that some more. Maybe you can get away with: 

Safety?
Given that drafts can be sitting in that state for a year or so, it is totally appropriate for people to find them under the new name.

Grüße, Carsten 
(who has three drafts in AUTH48-DONE right now, one approaching 10 weeks there)

---

From: Eliot Lear <[email-masked]>
Date: Tue, 3 May 2022 11:49:16 +0200

Unless someone made a recent change robots.txt, someone is harvesting 
the exclusions.  So we need to do something and fast. Changing the 
filename seems like it might cause some challenges for tooling in the 
short term, at least, and will definitely cause confusion for people 
like me who have to look at lots of RFC diffs.

Is there a reason to provide a listing of documents at /authors?

Eliot

---

From: Julian Reschke <[email-masked]>
Date: Tue, 3 May 2022 11:55:30 +0200

> Is there a reason to provide a listing of documents at /authors?

It helps me tracking what is in AUTH48, and what changes are being made
(and when). (*)

Of course that wouldn't be needed if these files lived in a public
source code repository.

Best regards, Julian

(*) And yes, that *is* important. Sometimes there seems to be confusion
about what kind of changes are acceptable *after* the authors approved
publication.

---

From: Robert Sparks <[email-masked]>
Date: Tue, 3 May 2022 09:23:44 -0500

> spidered very strangely during the datatracker upgrade.
This was broken configuration on the sandbox machine (for both sandbox 
and sandbox-ng) that is now repaired.

Fwiw, sandbox-ng has served its purpose and will be taken down soon (at 
the moment it is effectively an alias for sandbox).

RjS

---

From: Michael Richardson <[email-masked]>
Date: Tue, 03 May 2022 12:41:51 -0400

It seems to me that the AUTH48 copies are forever going to leak if some
search engines are ignoring the robots.txt

Wouldn't an HTTP AUTH help?  Even if the username/password is ietf/ietf?

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: "Salz, Rich" <rsalz=[email-masked]>
Date: Tue, 3 May 2022 16:48:23 +0000

>    Wouldn't an HTTP AUTH help?  Even if the username/password is ietf/ietf?

We don't want any barriers to our stuff being globally accessible, even if it means a search engine, or us, sometimes get it wrong.

---

From: "John R Levine" <[email-masked]>
Date: 3 May 2022 13:18:09 -0400

> search engines would find he URLs other than from the index URL.

They only add URLs that work.

> given how its temporary... ?!

I wrote the script that generates the site map and it never included 
anything in the authors directory.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 3 May 2022 19:56:05 +0200

> They only add URLs that work.

Sure. Absent of a sitemap, google would have had to find some place that
included a URL https://www.rfc-editor.org/rfc/authors/, then it finds the list
of URLS there, check them and adds them - and expires them only periodically.
And at that point in time, rfc9148.html was valid

> anything in the authors directory.

Well, either there is something wrong with the sitemap, or google _also_
looks up URLs through the non-sitemap ways described above, and some place
is pointing to https://www.rfc-editor.org/rfc/authors/. I just can't
find which place that would be through google itself.

Cheers
    Toerless

---

From: Michael Richardson <[email-masked]>
Date: Tue, 03 May 2022 17:51:27 -0400

> We don't want any barriers to our stuff being globally accessible, even
    > if it means a search engine, or us, sometimes get it wrong.

This is the pre-release rfc-editor AUTH48 copies of RFCs.
We actively don't want it globally accessible!
We might want to go as far as everyone in the WG accessing it.

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 3 May 2022 15:07:00 -0700

> We actively don't want it globally accessible!

I don't think I agree with this. In an ideal world, every step along the
way to final publication would be world accessible in perpetuity so that
people could see what had happened.

What we need to do is ensure that these are properly labelled and that
search engines find the right one, but that shouldn't make it impossible to
be transparent.

-Ekr

---

From: "John Levine" <[email-masked]>
Date: 3 May 2022 22:27:00 -0400

It appears that Eliot Lear  <[email-masked]> said:
>the exclusions. ...

The bogus files are under /rfc/authors which wasn't in the robots.txt
until I noticed it and asked the RPC to update it, which they did a
few hours ago, around 22:50 UTC.

I also explained how to add a <meta> tag to the auto-generated index
pages, like the one for /rfc/authors, to tell spiders to go away,
which I presume they will do shortly.  That shouldn't be needed for this
particular directory now that the robots file is updated, but it should
keep any other folders with auto-generated indices from leaking into search
results.

Legitimate search engines like Google and Bing all follow the advice
in robots.txt, so the main challenge is to make sure what's in the
robots file and <meta> tags matches what we want to index. There are
also lots of junk spiders with names like BLEXbot and Ahrefs that seem to
be for the SEO crowd. Some follow robots.txt, some don't, but I don't
care what their users see.

R's,
John

---

From: "Salz, Rich" <rsalz=[email-masked]>
Date: Wed, 4 May 2022 11:39:35 +0000

>    We might want to go as far as everyone in the WG accessing it.

Can you explain why you think this is different than any other draft?

---

From: Michael Richardson <[email-masked]>
Date: Wed, 04 May 2022 09:46:18 -0400

> Can you explain why you think this is different than any other draft?

First, I'm all for all of the record being public, and the documents that go
along with them.  I'll point out that the AUTH48 documents mutate in place,
as edits are being made, so if you want a public record, then you need all of
the history.  It's meaningless for a reviewer to see "change X to Y", if they
can never see X.  So, what we really don't want is for some of the contents
to get cached by search engines (or other robots) and not other content.

What we have in the AUTH48 step is essentially a shared, distributed editor.
At present, we don't require authors to make their editor contents public,
only the IDs that they actually post.

Second, the current process is actually really annoying and hard to use.
I'd like for all of it to happen in the form of git pull requests.
At which point, if you wanted to mirror/cache/index that repo, I'd have no
problem with it, because the record would be complete.

---

From: Eliot Lear <[email-masked]>
Date: Wed, 4 May 2022 16:56:32 +0200

On this:

> problem with it, because the record would be complete.

This is probably worth discussing once the RSWG gets formed. There's a 
whole work flow tied to all of this that could be reviewed.

Eliot

---

From: Jay Daley <[email-masked]>
Date: Wed, 4 May 2022 16:14:22 +0100

> On 4 May 2022, at 15:56, Eliot Lear <[email-masked]> wrote:
> 
> Signed PGP part
> On this:
> 
> This is probably worth discussing once the RSWG gets formed. There's a whole work flow tied to all of this that could be reviewed.

A small alarm bell just went off when I read that. From my understanding of the role of the RSWG it could define a policy that says something like "the AUTH48 process should be transparent, all changes (proposed, accepted, rejected) be public, all changes tracked so that a diff can be seen, etc" but it could not define a policy that says "The RPC must do everything in GitHub".  The former is policy, the latter is operations and a line was drawn between the two in the new model.

Jay

> 
> Eliot
> 
> <OpenPGP_0x87B66B46D9D27A33.asc>
> 
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Wed, 04 May 2022 11:37:22 -0400

> This is probably worth discussing once the RSWG gets formed. There's a whole
    > work flow tied to all of this that could be reviewed.

Maybe.  I have no opinion about where the responsability to make the decision
lies.  I think that until such a decision is made that we need to address the
immediate issues which were:

1) robots.txt was inaccurate and some bots are caching AUTH48 documents and
returning them in searches.  We all agree that this is wrong and potentially misleading.
John says that this might be fixed.

2) there are obscure crawlers which do not respect robots.txt.  John feels that this
isn't our problem, but I somewhat disagree.  I think that until we have
defined the public workflow, that we should make sure that our current
workflow is actually is as private as authors and editors expect.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 May 2022 19:54:21 +0200

Am 04.05.2022 um 17:37 schrieb Michael Richardson:
 > ...> 2) there are obscure crawlers which do not respect robots.txt.
John feels that this
> workflow is actually is as private as authors and editors expect.

FWIW, I don't expect the workflow to be private at all.

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 4 May 2022 23:51:51 -0400

On Tue, 3 May 2022, John Levine wrote:
> pages, ...

The index pages now have the <meta> tags so between the robots.txt and 
the tags I expect the funky pages will age out of Google and Bing.

R's,
John