Subject: [Xml-sg-cmt] Please consider adding a way to speak about references as a token to the grammar

From: Robert Sparks <[email-masked]>
Date: Fri, 4 Mar 2022 16:15:41 -0600

See https://github.com/ietf-tools/datatracker/issues/3577

When we went from v2 to v3, we lost the token parsing at <?rfc 
include="I-D.name-bits">.

I strongly suggest we add these back explicitly to the grammar with 
something like (this is a swag for meaning - fix the details after the 
thought's been transferred):

<reference bibxml_handle="I-D.sparks-sipcore-multiple-reasons"/>

That tools could use to then make queries directly into the upcoming 
bibxml api.

Is this too great an ask? It would, I think, have a significant easing 
aspect on our authors.

RjS

---

From: "John R Levine" <[email-masked]>
Date: 4 Mar 2022 21:31:18 -0500

> That tools could use to then make queries directly into the upcoming bibxml api.

What's the advantage of that rather than allowing Xinclude only to places 
we control and accept bibxml handles as the URL?  It amounts to the same 
thing with slightly different syntax.

<references xml:base="https://bibxml.ietf.org/">
   <xi:include href="I-D.sparks-sipcore-multiple-reasons"/>
</references>

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Robert Sparks <[email-masked]>
Date: Sat, 5 Mar 2022 09:00:29 -0600

Places change over time and it would be useful to abstract that away from something the author has to know about. 

Sent from my iPhone

> On Mar 4, 2022, at 8:31 PM, John R Levine <[email-masked]> wrote:
> 
> 
>> 
> What's the advantage of that rather than allowing Xinclude only to places we control and accept bibxml handles as the URL?  It amounts to the same thing with slightly different syntax.
>

---

From: "John R Levine" <[email-masked]>
Date: 5 Mar 2022 10:42:53 -0500

> Places change over time and it would be useful to abstract that away from something the author has to know about.

So take out the xml:base and say that in xmlrfc, the default 
prefix for xi:include is the bibxml library.

No matter how we do this it is important to make it clear when the 
references are imported so published RFCs don't depend on external 
libraries.

Regards,

---

From: Julian Reschke <[email-masked]>
Date: Sat, 5 Mar 2022 16:47:45 +0100

Am 05.03.2022 um 16:42 schrieb John R Levine:
>> Places change over time and it would be useful to abstract that away
>> from something the author has to know about.
>
> So take out the xml:base and say that in xmlrfc, the default prefix for
> xi:include is the bibxml library.

No.

x:include is a generic mechanism; don't break it.

When 7991 was designed, we intentially used just a generic mechanism. If
the concern is that base URIs are changing then the right fix is not to
keep moving things around.

If you want to bake something IETF-specific into the vocabulary, by all
means do not break the generic mechanism.

> references are imported so published RFCs don't depend on external

Of course.

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 5 Mar 2022 10:54:26 -0500

> keep moving things around.

I agree that it seems the simplest approach is for the IETF to have enough 
discipline that we can say the the references will stay at bibxml.ietf.org 
or wherever.

If we let them move and we put the location info into the tools, that 
makes something else that will break mysteriously when people use out of 
date versions of the tools, as they always do.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Sat, 5 Mar 2022 10:34:43 -0600

>> xi:include is the bibxml library.
That bare imperative is aggressive. It's not clear what you're saying no 
to. Establishing tone in email is hard, and a gentler phrase here would 
have helped.
> x:include is a generic mechanism; don't break it.

I am not proposing removing xi:include. I am talking about a different 
mechanism, specific to bibxml, that allows reference by well-known token 
rather than an arbitrary URL. We do this already, for example, in places 
where we refer to swaths of boilerplate text by token rather than using 
xi:include to pull it in from "somewhere".

And a related, but separate topic: While the grammar should continue to 
support generic xi:include, and the processors like xml2rfc should allow 
following arbitrary urls on someone's personal machine, the places where 
we operate on documents using the grammar at our servers (like the 
datatracker draft submission point) must limit what urls it will follow. 
We have discovered a range of security vulnerabilities if we don't.

> keep moving things around.
Again, I'm not proposing taking the generic mechanism away, but the IETF 
community has a specific use case at references that is _very_ heavily 
used that would benefit from regularization. This would simplify author 
workflows in the long run.

---

From: "John R Levine" <[email-masked]>
Date: 5 Mar 2022 13:08:10 -0500

> xi:include to pull it in from "somewhere".

The boilerplate is in strings in the code in xml2rfc.  It seems unlikely 
that we would build the contents of the bibxml library into xml2rfc.

I don't understand why it would be impractical to declare that bibxml will 
always live at a fixed URL that we control.  If we expect it to move, we 
need to keep track of all the tools that are supposed to know where it is, 
have a plan to update them, and to tell all of their users that they need 
to use the new version.  Keeping the refs in a known fixed place seems a 
lot easier.

> following arbitrary urls on someone's personal machine, the places where we 
> operate on documents using the grammar at our servers (like the datatracker 
> draft submission point) must limit what urls it will follow. We have 
> discovered a range of security vulnerabilities if we don't.

I think we all agree about that.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Sat, 5 Mar 2022 12:35:00 -0600

On 3/5/22 12:08 PM, John R Levine wrote:
>> I am not proposing removing xi:include. I am talking about a 
>> different mechanism, specific to bibxml, that allows reference by 
>> well-known token rather than an arbitrary URL. We do this already, 
>> for example, in places where we refer to swaths of boilerplate text 
>> by token rather than using xi:include to pull it in from "somewhere".
>
> The boilerplate is in strings in the code in xml2rfc.  It seems 
> unlikely that we would build the contents of the bibxml library into 
> xml2rfc.
No, but we can have a token set that follows a pattern that is well 
understood (we already have one) with defined resolution mechanics.
>
> I don't understand why it would be impractical to declare that bibxml 
> will always live at a fixed URL that we control.  If we expect it to 
> move, we need to keep track of all the tools that are supposed to know 
> where it is, have a plan to update them, and to tell all of their 
> users that they need to use the new version. Keeping the refs in a 
> known fixed place seems a lot easier.

I think that's an orthogonal discussion. _Just_ having that anchor 
doesn't address the full benefit I think I see from not having such an 
explicit path - particularly when we get to trying to make it easy for 
working copies to continue to have pointers while the repository has 
resolved content.

Sure, we can work really hard to keep things in one place, but history 
shows that has worked badly, and I'm not confident that the future won't 
put irresistible pressure on us to change again.

>
>> And a related, but separate topic: While the grammar should continue 
>> to support generic xi:include, and the processors like xml2rfc should 
>> allow following arbitrary urls on someone's personal machine, the 
>> places where we operate on documents using the grammar at our servers 
>> (like the datatracker draft submission point) must limit what urls it 
>> will follow. We have discovered a range of security vulnerabilities 
>> if we don't.
>

---

From: "John R Levine" <[email-masked]>
Date: 5 Mar 2022 20:10:32 -0500

On Sat, 5 Mar 2022, Robert Sparks wrote:
> I think that's an orthogonal discussion. _Just_ having that anchor doesn't 
> address the full benefit I think I see from not having such an explicit path 
> - particularly when we get to trying to make it easy for working copies to 
> continue to have pointers while the repository has resolved content.

Something is going to have to know where the bibxml files are, and either 
we figure out how to maintain a list of stable URLs or we need to manage 
the list of tools that know where they are, and how we are going to update 
them all and push out the updates to all of the users when the tools 
change.

Doing it this way at least makes it obvious what is broken 
and what you need to fix if they move:

<references xml:base="https://bibxml.ietf.org/">
   <xi:include href="I-D.sparks-sipcore-multiple-reasons"/>
   <xi:include href="RFC2549"/>
</references>

In your proposal with the link in the reference, would it automagically 
replace the link with the contents when an I-D is submitted, or when it's 
rendered, or when it's prepped,, or add the contents and keep the link so 
it could update later if the library changes, or something else?

R's,
John

---

From: Robert Sparks <[email-masked]>
Date: Mon, 7 Mar 2022 09:22:38 -0600

On 3/5/22 7:10 PM, John R Levine wrote:
>> doesn't address the full benefit I think I see from not having such 
>> an explicit path - particularly when we get to trying to make it easy 
>> for working copies to continue to have pointers while the repository 
>> has resolved content.
>
> Something is going to have to know where the bibxml files are, and 
> either we figure out how to maintain a list of stable URLs or we need 
> to manage the list of tools that know where they are, and how we are 
> going to update them all and push out the updates to all of the users 
> when the tools change.
A much more tractable problem (tens of these things) than the current 
state of managing this information in the thousands of author's heads.
>
> Doing it this way at least makes it obvious what is broken and what 
> you need to fix if they move:
>
> In your proposal with the link in the reference, would it 
> automagically replace the link with the contents when an I-D is 
> submitted, or when it's rendered, or when it's prepped,, or add the 
> contents and keep the link so it could update later if the library 
> changes, or something else?

It would expand the reference pointer , but leave it in the source as a 
comment, structured such that it makes it easy for another program to 
remove what resulted from the expansion, and restore the pointer.

The biggest goal here is to make it easier for authors to continue to 
maintain their source in a way that has indirection to references while 
allowing us to make draft-submission more accurate than it currently is.

There are secondary, but large goals.

1) Right now the URL structure the older bibxml service uses suffers 
from having to build a really arcane bit of structure into the tag 
that's used to distinguish between "give me a particular version of a 
draft" and "give me the latest version of the draft". At 
xml2rfc.tools.ietf.org, references that start 'I-D.draft-' are always 
references to a specific version, and references that start 'I-D.ietf-' 
or 'I-D.name-' are to the latest version. The datatracker's bibxml3 
service and the new bibxml service are currently different - they will 
take the token and look for a version match and return that if there is 
one, and then look to see if it matches a draft name without a version, 
returning the latest if there is a match. In both ways of doing things, 
there is a level of complexity that is unkind to the authors (and the 
coders - the guessing heuristics for finding 'latest' are ugly - people 
have submitted drafts that look like draft-foo-bar-21-22-23). I would 
_much_ rather that we provided the authors a way to explicitly say what 
they want rather than relying on side effect. Mocking up an alternate 
future again, I would like to see something like:

<reference type='I-D' name='draft-foo-bar'> for 'current' and <reference 
type='I-D' name='draft-foo-bar' rev='08'> for a specific version.

We could encode this things back into URL paths and do something like 
what you have above, but then people have to learn the conventions of 
what bit of info lives in what place in the url path, where something 
like the above makes the statement very explicit.

2) For xml2rfc, internally, we plan to use the API that's been developed 
(https://dev.bibxml.org/api/v1/) rather than the current url based form 
for retrieval. We'll be converting URL when given. Yes, I realize that 
we'll always need to be able to recognize the older hosts and paths, so 
that code will be with us anyhow, but I still think we'd be much better 
off going forward with something that allowed the kind of explicit and 
self-documenting statement in 1).

RjS

---

From: Julian Reschke <[email-masked]>
Date: Mon, 7 Mar 2022 16:33:15 +0100

Am 07.03.2022 um 16:22 schrieb Robert Sparks:
> ...
> ...

Out of curiosity: why do we need an "API" when we could just use HTTP
GET + predictable URIs?

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Mon, 7 Mar 2022 09:41:15 -0600

> Out of curiosity: why do we need an "API" when we could just use HTTP

Tools for both users and software - it allows an api that's well 
documented in a way that software can read.

It will allow us to build a more testable infrastructure, and one that 
can be extended in more sensible, versioned, and documented ways.

It is a much easier thing to add "search" to such an API than it would 
be to build it into the URIs.

RjS

---

From: Julian Reschke <[email-masked]>
Date: Mon, 7 Mar 2022 16:46:29 +0100

> be to build it into the URIs.

FWIW, I disagree with all of these points; I believe this transforms
something that can (and has been) very simple into something that is way
more complex than needed.

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 7 Mar 2022 13:35:20 -0500

> would like to see something like:
>

Now we're just in the aesthetics.  I'd write it this way:

<references xml:base="https://bibxml.ietf.org/">
    <xi:include href="/type/i-d/name/draft-foo-bar" />
    <xi:include href="/type/i-d/name/draft-foo-bar/rev/08" />
</references>

I agree that the old URLs are awful, but with a little planning you can
put whatever sensible structure you want on the URLs.

For the <references ... > version I would want a really clear spec about 
when they're resolved and what resolving them means, e.g., a it 
goes through and replaces every refrerence with bibxml tags with the 
current version, and we do it when an I-D is uploaded and when it's 
prepped.

Or I suppose when you run xml2rfc and it finds an unexpanded one and you 
tell it expanding is OK.  The semantics are messy.

It occurs to me that if the data is coming from Ribose's API, it's an 
afternoon's work to write a proxy that responds to Xinclude requests with 
answers from that API, at least so long as the API holds still.  It's not 
a big deal to allow both.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Mon, 7 Mar 2022 13:24:34 -0600

On 3/7/22 12:35 PM, John R Levine wrote:
>> <reference type='I-D' name='draft-foo-bar'> for 'current' and 
>> <reference type='I-D' name='draft-foo-bar' rev='08'> for a specific 
>> version.
>

And the document it somewhere _other_ than in the url itself.

I think that distinction is more than aesthetics.

Explain those different things to a new person, and I think you'd see 
the difference?

>
> For the <references ... > version I would want a really clear spec 
> about when they're resolved and what resolving them means, e.g., a it 
right
>
> Or I suppose when you run xml2rfc and it finds an unexpanded one and 
> you tell it expanding is OK.  The semantics are messy.
>
> afternoon's work to write a proxy that responds to Xinclude requests 
> with answers from that API, at least so long as the API holds still.  
> It's not a big deal to allow both.
We are requiring that we continue to provide the legacy URL formats.

---

From: "John R Levine" <[email-masked]>
Date: 7 Mar 2022 14:44:18 -0500

> And the document it somewhere _other_ than in the url itself.

Yes, of course.  The attributes in the <reference> would have to be 
documented too.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Mon, 7 Mar 2022 13:51:50 -0600

> Yes, of course.  The attributes in the <reference> would have to be 
Yet a new person would probably navigate filling in named attributes 
(which, if well chosen, self-document) than having to learn a 
secret-handshake about what url segment means what thing.

---

From: "John R Levine" <[email-masked]>
Date: 7 Mar 2022 15:25:45 -0500

On Mon, 7 Mar 2022, Robert Sparks wrote:
> Yet a new person would probably navigate filling in named attributes (which, 
> if well chosen, self-document) than having to learn a secret-handshake about 
> what url segment means what thing.

I am reminded of a famous paper by Don Norman complaining about how arcane 
and unnatual the Unix command language was, in which it was obvious that 
his problem was that it was different from the PDP-10 he was used to.

I don't see that it would be harder or easier to learn about XML 
attributes rather than about URL segments.  If they're reasoanbly well 
documented, they should be about the same, and if they use the same 
labels they are equally self-documenting or not.

The interesting difference is in tooling.  If you want to use XML tools or 
XSLT scripts other than xml2rfc, you get Xinclude for free, while you have 
to do <reference> expansion yourself.  That's not a killer but it is a 
tradeoff.

As I think I've said, I'm not strongly opposed to <reference> tags but I 
would want a clear plan for when they're expanded and by what.

R's,
John