Subject: [rfc-i] Access to BibXML collections by wget/rss/rsync

From: Jay Daley <[email-masked]>
Date: Fri, 23 Apr 2021 08:44:47 +1200

I am currently supporting the Tools Team PM with planning a replacement for the existing BibXML service and we have a question about the current usage.  The BibXML service is document under the Citation Libraries section of 

	http://xml2rfc.tools.ietf.org

and that notes that it is possible to download a collection of BibXML files by wget, rss, rsync and in zip/tgz files.  As far as we are aware the collections stopped updating in January and we are not aware of anyone that maintains a separate public repository that pulls from this one.  Consequently we are considering dropping wget/rss/rsync/zip from the requirements for a replacement service as that would enable us to make more use of generation on demand for some of the less frequently used collections.

Is there anyone out there pulling any of the BibXML collections by any of these mechanisms or do you know of anyone that publishes mirrors of these?

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 22 Apr 2021 22:35:15 +0100

On 22/04/2021 21:44, Jay Daley wrote:
> Is there anyone out there pulling any of the BibXML collections by
> any of these mechanisms or do you know of anyone that publishes
> mirrors of these?

For me: no and no.

But answering a different related question as an FYI:

I've generally gotten RFC bibtex from [1] and use that for
papers written using latex. I make an i-d.bib file using a
python script that Elwyn Davies wrote ages ago that does it's
stuff based on [2].

My requirement there is to end up with rfcs.bib and i-d.bib files with 
up-to-date bibtex for all RFCs and I-Ds. Those are
v. handy and really increase the probability I reference I-Ds
and RFCs and get that right.

Cheers,
S.

[1] https://www.tm.uka.de/~bless/bibrfcindex.html
[2] http://www.ietf.org/id/all_id2.txt

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 22 Apr 2021 23:37:37 +0200

On 22. Apr 2021, at 22:44, Jay Daley <[email-masked]> wrote:
> Is there anyone out there pulling any of the BibXML collections by any of these mechanisms

Yes.  (All of them, via rsync, several times per day.)

Since the rsync access became less useful for the I-D bibxml, I wrote (*) a tool kramdown-rfc-cache-i-d-bibxml that is part of the kramdown-rfc distribution.
That gives you a copy of the datatracker bibxml for active I-Ds (1786 of them right now) that is stale only by an hour or less, in a reasonably efficient way (except for the first run, usually finishes in a dozen seconds or so, but does fetch a new copy of an 8 MB file each time).

So far there has not been much of a need to do this for RFCs (those happen less frequently and still seem to be updated in a timely way), but maybe that is the next step.
The other collections are not that useful at all unless you have a local copy.
(DOI and IANA are of course a different game, but IANA does offer rsync access, too.)

> or do you know of anyone that publishes mirrors of these?

Since there was the public rsync server, there hasn’t really been a need (I have a university-internal mirror that is internally published only).

Grüße, Carsten

(*) Kudos to Robert Sparks for his help getting this done, and to Russ Housley for fixing the datatracker limitation that this happened to uncover.
Details about using the tool and why you might want to use it are in <https://mailarchive.ietf.org/arch/msg/rfc-markdown/FJkHJJrj0ci_uemB5F5sSEP7Wyg>

---

From: Jay Daley <[email-masked]>
Date: Fri, 23 Apr 2021 10:40:16 +1200

> On 23/04/2021, at 10:33 AM, Larry Masinter <[email-masked]> wrote:
> 
> Is there a document you are developing for requirements?

I will reply off-list.

Jay

> I've been working with Zotero to unmangle IETF citations (among others).
> --
> https://LarryMasinter.net https://interlisp.org 
> 
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 23 Apr 2021 10:01:04 +1000

Adding Tobie --

Tobie, what does Specref use for IETF specs?

(for others: <https://www.specref.org>)

Cheers,

> On 23 Apr 2021, at 6:44 am, Jay Daley <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Ronald Tse <[email-masked]>
Date: Fri, 23 Apr 2021 02:58:16 +0000

Like Carsten, Metanorma/Relaton syncs this data but to a data mirror:
https://github.com/relaton/relaton-data-ietf

We maintain this mirror because of speed reasons and outages we have experienced with the official service. BibXML files are fetched individually since the archives stopped updating.

Kind regards,
Ron

_____________________________________

Ronald Tse
Ribose Inc.

On 23 Apr 2021, at 5:37 AM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:

On 22. Apr 2021, at 22:44, Jay Daley <[email-masked]<mailto:[email-masked]>> wrote:

_______________________________________________
rfc-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Jay Daley <[email-masked]>
Date: Fri, 23 Apr 2021 15:18:38 +1200

> On 23/04/2021, at 2:09 PM, Randy Bush <[email-masked]> wrote:
> 
>> we are not aware of anyone that maintains a separate public repository
>> that pulls from this one.
> 
> i do
> 
> RSY="/usr/bin/rsync -vaHx --delete"
> 
> $RSY xml2rfc.tools.ietf.org::xml2rfc.bibxml/bibxml $HOME/refs/rfc
> $HOME/bin/do-allid
> #$RSY xml2rfc.tools.ietf.org::xml2rfc.bibxml/bibxml3 $HOME/refs/id
> 
> do we need to sign an nda?  what ever happened to open process in
> the ietf?

No NDA but I’m reluctant to discuss the contents of an RFP with a potential bidder before the RFP is issued so I just wanted to check that with Larry before sharing them.

> 
> randy, a second class citizen

I can’t comment on your citizenship, but if you want to see the requirements then I am happy to provide them.

Jay

> 
> ---
> [email-masked]
> `gpg --locate-external-keys --auto-key-locate wkd [email-masked]`
> signatures are back, thanks to dmarc header butchery
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Fri, 23 Apr 2021 10:44:39 -0400

Jay Daley <[email-masked]> wrote:
    > Is there anyone out there pulling any of the BibXML collections by any
    > of these mechanisms or do you know of anyone that publishes mirrors of
    > these?

I was/am using rsync.  I was doing this only because of historic cases of the entire
system being unavailable.

I have no interest in continuing, or where my cron job is.
(looking... I think that I stopped)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide