Subject: [rfc-i] Evolving document sources over a long time (Re: Comments on draft-roach-bis-documents-00)

From: Carsten Bormann <[email-masked]>
Date: Sat, 11 May 2019 08:10:03 +0200

Hi David,

Thank you for your article in [email-masked], it is really a little treasure-trove.

Let me extract those parts that pertain to your experiences with the evolution of RFCXML and add rfc-interest; please continue discussion on rfc-interest (which you may want to subscribe if RFC authoring is part of your life).

> On May 11, 2019, at 01:31, David Noveck <[email-masked]> wrote:
> 
> Apparently an .xml file used to prodtce an rfc is not necessarily acceptable to later versions of xml2rfc.  You can get an idea of the issues by looking at rc5661Base.xml and the file I wound up with, rfc5661Ready.xml; both are attached.

Most of your changes have to do with the better validation in the current tools, namely:

— validating the syntax of anchors (no spaces, plus signs)
— validating artwork.

For the latter, doing a wholesale <![CDATA[ … ]]> is always a better approach than sprinkling &amp;/&lt; (you almost never need &gt;, by the way).  (See also authoring tools below.)

The anchor syntax — it is just too bad the v1 tool didn’t check that.  

> I was able to get a an .xml from which a .txt file could be generated, but despite my best efforts there were differences (more than a few) between it and rfc5661, which your document would consider to be "spurious" but appear to be unavoidable. In any case they are not gratuitous changes and should not interfere with consideration of the document.   The majority of the diffs arise from the following  issues:
> 	• Despite the fact that the xml for the reference sections of both xml files are identical and the processing options are identical (symrefs="no" sortrefs="yes") the reference ids in rfc5661Ready.txt and in rfc5661 are different so that rfcdiff shows every line containing a reference as part of a diff.  Apparently, different versions of xml2rfc use different approaches to sorting references. 

Protip: DO NOT USE numeric references.  Ever.  This was stylistically appealing for some tiny documents, but rarely is appropriate for actual specifications (and certainly not for NFSv4-sized ones!).

> 	• There are a fair number of difference that seem to have arisen because the RFC edtitor made minor corrections directly on the .txt file so that each such correction (while valid) is reported by rfcdiiff as a difference.

Right.  That will be less of a problem in the future, but it does require tediously porting back changes to the document source.  

> 	• The reference sections are exposed to the same sorting issues as the reference id's.  To the naked eye, and to rfcdiff they look very different, despite that fact that they contain the same references.

The sorting issue should be taken care of by not using numeric references (really, for the sake of your readers, please don’t).
Since RFC 5661, we also got DOIs on RFCs, so it is inevitable there are a lot of diffs.  Again, tedious, but not really avoidable.

Of course, I would not recommend directly authoring in XML these days (there are now good markdown and asciidoc choices, as well as an org-mode one if that is your thing), but that was the way things were done in 2010.  (If you do want to make the transition, there are some conversion tools available; I may be able to help.)

Grüße, Carsten

_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 12 May 2019 18:16:34 +0200

> It is not clear to me, given that anchors are always in quotes, why these restrictions were added

RFC 2629 provides a DTD for RFCXML.  This enables XML-based tools to meaningfully process RFCXML.
In this DTD, anchors were supposed to be XML “ID” type strings.  Using non-conforming strings here created problems for some tools.  So the v1 tool’s overly enthusiastic “be liberal what you accept” wan’t really helpful.  This was fixed in v2.

> In any case, they were added with no concern about the fact that many existing .xml files would be invalidated.  

(It would have been easy to create a transition tool.  I have no idea why this wasn’t done — maybe people finished their work on existing v1 documents with the v1 tool, so the problem never occurred in practice.  Or it was simply too easy to fix by hand — I remember doing this for a few documents before I completely stopped using XML as an authoring format.)

> I hope future instances of such changes can be avoided.

Yes (preferably by following the specification in the first place, or by providing transition tools, as we’ll have for v2 to v3).
 
> > — validating artwork.
> 
> Artwork, by its nature, does not need to be validated.  

But it still needs to be encoded in well-formed XML — which requires CDATA or &amp; etc.
Again, the v1 tool was more “liberal” (broken) here than it should have been.

> Since we can’t go back in time and change the v1 tool, a better way to provide compatibility would be for the existing tool to get a processing option, allowing it to accept formerly acceptable anchors that have been forbidden in recent times.

A better option is to provide a migration tool, so previously invalid (and incorrectly accepted) input documents can be transformed into well-formed input documents that have the intended effect — these can then be processed by other XML-based tools as well.

[…]
> The actual .xml file was generated by make

That is actually a good way to generate a non-trivial document.
In many cases, today’s Makefiles generate markdown documents, or generate pieces that are then included from the main markdown document.  The Makefile then translates into XML whole-sale.

> > Right.  That will be less of a problem in the future, 
> 
> I hope so.  Why do you think that this will be getting better?

With the move to v3, an XML document will be the main publication format.
(It still isn’t the same as that submitted by the authors, but I think one could continue authoring from that.)

> > (really, for the sake of your readers, please don’t).
> 
> For RFC5661 and documents derived from it using Adam’s procedure, that ship has already sailed :-(.

As the references section needs to be updated anyway (for the DOIs), I’m not sure this is really true.  Or, if it is, RFC 5661 maybe isn’t really a candidate for this process, because it may be impractical to re-generate the exact numbering that RFC 5661 used.

> > Since RFC 5661, we also got DOIs on RFCs, so it is inevitable there are a lot of diffs.  
> 
> It is not inevitable as shown by the fact that I didn't run into that issue.   It's kind of nice to know that there was an issue out there that I didn't run into :-)
> 
> For reasons I  really don't understand, the xml for rfc5661 does not include rfc reference from external libraries.   It includes them inline, so a new rfc derived from that xml  file will not include DOIs.  

Yes.  All these RFC references would be updated by the RFC editor into current references.

> That is not a problem for Adam's procedure, but it may be for the IESG or the RFC editor.   I hope that, in processing RFC’s using Adam's procedure, people will overlook the lack of DOIs in the same way that they overlook other aspects of the document that would prevent a new document of that form from being published.

AFAICT, they can’t, as the RFC editor has committed to providing DOIs.

> > Of course, I would not recommend directly authoring in XML these days 
> 
> Why not?

Because:

> > (there are now good markdown and asciidoc choices, as well as an org-mode one if that is your thing), 
> 
> Where are these documented?

These are not maintained by the IETF, so they are documented at the place wherever the authors liked to document them.

Try http://rfc.space and the links emanating from that (well, maybe re-consult in a day or so; I just noticed the links are rather incomplete as of today).

> > but that was the way things were done in 2010.  
> 
> I’m prepared to stick with that, unless there is something better about the alternatives.

Right, for a minor update, digging out the v1 tools and finding a platform where they can still run may actually be the best way to proceed.

> > (If you do want to make the transition, there are some conversion tools available; I may be able to help.)
> 
> I would only make a transition for new documents.

I have helped people transition the authoring format of quite well-advanced documents, and it seems it actually helped them complete them.  (A routine check after transition is whether the formatted output is the same, and it typically is, except occasionally for minor differences in hyphenation and spacing caused by different versions of xml2rfc being in use.)

> For documents to be processed according to Adam’s procedure, the likelihood of minor diffs arising is such that I don't think a transition is possible.

There is no way the v2 tool will create the same formatting that the v1 tool did, so I think minor diffs are nearly impossible to avoid anyway.

> For a later full bis, I would be replacing major sections of an existing xml-written document which would make any transition especially difficult.
> 
> For completely new documents,my concerns would be to make sure that the needs of those who might later be called upon to update the document were taken account of.   Relevant issues: 
> 	• Could we be sure that the new source could be processed at a later time, and give rise to the same xml?

Different authoring tools may differ in their priority on forward compatibility.
The one I can talk about authoritatively has created the same formatted documents for more than 8 years now, but not necessarily exactly the same xml.

> 	• Would the xml produced by a new tool be easily human-editable? 

For all of them that I have looked at in some detail: yes.
But you wouldn’t want to do this (as with any compiler, you would edit the input), except if you want to migrate to a different authoring tool (which is an important option that is easily provided by the common XML format).

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sun, 12 May 2019 18:41:08 +0200

On 12.05.2019 18:16, Carsten Bormann wrote:
> ...

First of, +1 to almost everything you said...

> .. >> For RFC5661 and documents derived from it using Adam’s procedure,
that ship has already sailed :-(.
> Yes.  All these RFC references would be updated by the RFC editor into current references.

Actually, I think a processor should optionally insert the DOIs
automatically. The current way of doing things just introduced a
completely unneeded incompatibility.

> ...

FWIW, I disagree (but I realize that I'm probably in a minority). The
problem with Markdown is that the simple things are easy, but everything
else gets messy. I'm looking forward to see how you kram (pun intended)
the V3 features into your tool...

> Right, for a minor update, digging out the v1 tools and finding a platform where they can still run may actually be the best way to proceed.

All you need is a TCL processor. Works fine over here on a newly
installed notebook with Windows 10 and Cygwin.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 12 May 2019 18:57:22 +0200

Hi Julian,

> the V3 features into your tool…

Do I have to?

If an author wants to use a V3 feature that does not lend itself to authoring in markdown, they can always put the XML tags right into their markdown source.

> installed notebook with Windows 10 and Cygwin.

Haven’t tried that in a while…  
Works for me, too (macOS 10.13.6, on a small document, with the usual formatting differences from v2…).  
Good to know.
(Although I have heard people have had problems on other platforms recently.)

Grüße, Carsten

---

From: David Noveck <[email-masked]>
Date: Sun, 12 May 2019 11:11:22 -0400

>> Apparently an .xml file used to prodtce an rfc is not necessarily
acceptable to later versions of xml2rfc.

> Most of your changes have to do with the better validation in the current
tools, namely:

At least more extensive validation.

> — validating the syntax of anchors (no spaces, plus signs)

It is not clear to me, given that anchors are always in quotes, why these
restrictions were added

In any case, they were added with no concern about the fact that many
existing .xml files would be invalidated.  I hope future instances of such
changes can be avoided.

> — validating artwork.

Artwork, by its nature, does not need to be validated.  I really don't want
xml2rfc warnings about the quality of my artwork :-)

However, changes in the processing of character entities within artwork has
given rise to unxpected rejections of existing xml files :-(

> For the latter, doing a wholesale <![CDATA[ … ]]> is always a better
approach than sprinkling &amp;/&lt;

That's good advice which the nfsv4 working group would do well to follow
for future RFC's.

As our RFC's often have lots of XDR, it is all too easy for a case of
amper-lt/amper-gt dysphoria to develop (not in the DSM :-)

> (you almost never need &gt;, by the way).  (See also authoring tools
below.)

If it's *almost* never, my tendency would be to use it always, just in case.

> The anchor syntax — it is just too bad the v1 tool didn’t check that.

Since we can't go back in time and change the v1 tool, a better way to
provide compatibility would be for the existing tool to get a processing
option, allowing it to accept formerly acceptable anchors that have been
forbidden in recent times.

> > I was able to get a an .xml from which a .txt file could be generated,
but despite my best efforts there were differences (more than a few)
between it and rfc5661, which your document would consider to be "spurious"
but appear to be unavoidable. In any case they are not gratuitous changes
and should not interfere with consideration of the document.   The majority
of the diffs arise from the following  issues:
> >       • Despite the fact that the xml for the reference sections of
both xml files are identical and the processing options are identical
(symrefs="no" sortrefs="yes") the reference ids in rfc5661Ready.txt and in
rfc5661 are different so that rfcdiff shows every line containing a
reference as part of a diff.  Apparently, different versions of xml2rfc use
different approaches to sorting references.

> Protip: DO NOT USE numeric references.  Ever.

Good advice, but we need to deal with the fact imappropriate choices were
made in the past and incorporated published RFCs.

> This was stylistically appealing for some tiny documents, but rarely is
appropriate for actual specifications (and certainly not for NFSv4-sized
ones!).

I'm not sure how this choice came to be.   The actual .xml file was
generated by make and I don't think anybody really paid attention to the
processing instructions which were fixed early on and never really
reconsidered.

>>       • There are a fair number of difference that seem to have arisen
because the RFC edtitor made minor corrections directly on the .txt file so
that each such correction (while valid) is reported by rfcdiiff as a
difference.

> but it does require tediously porting back changes to the document
source.

Yes :-(

> >       • The reference sections are exposed to the same sorting issues
as the reference id's.  To the naked eye, and to rfcdiff they look very
different, despite that fact that they contain the same references.

> The sorting issue should be taken care of by not using numeric references

I don't think it will be.   It will eliminate diffs due to different
numeric tags but the layout of the reference sections will be different,
which will show up as a diff.

I hope Adam's document would allow documents with such diffs to be
processed by his procedure.

> (really, for the sake of your readers, please don’t).

For RFC5661 and documents derived from it using Adam's procedure, that ship
has already sailed :-(.

However, when there is a full bis succeeeding RFC5661, not processed
according to Adam's procedure, it should not use numeric references :-)

> Since RFC 5661, we also got DOIs on RFCs, so it is inevitable there are a
lot of diffs.

It is not inevitable as shown by the fact that I didn't run into that
issue.   It's kind of nice to know that there was an issue out there that I
didn't run into :-)

For reasons I  really don't understand, the xml for rfc5661 does not
include rfc reference from external libraries.   It includes them inline,
so a new rfc derived from that xml  file will not include DOIs.   That is
not a problem for Adam's procedure, but it may be for the IESG or the RFC
editor.   I hope that, in processing RFC's using Adam's procedure, people
will overlook the lack of DOIs in the same way that they overlook other
aspects of the document that would prevent a new document of that form from
being published.

> Again, tedious, but not really avoidable.

In cases in which it is not avoidable, the addition of DOI's should not
prevent processing of a document acccording to Adam's procedure.

Why not?

> but that was the way things were done in 2010.

I'm prepared to stick with that, unless there is something better about the
alternatives.

For documents to be processed according to Adam's procedure, the likelihood
of minor diffs arising is such that I don't think a transition is possible.

For a later full bis, I would be replacing major sections of an existing
xml-written document which would make any transition especially difficult.

For completely new documents,my concerns would be to make sure that the
needs of those who might later be called upon to update the document were
taken account of.   Relevant issues:

   - Could we be sure that the new source could be processed at a later
   time, and give rise to the same xml?
   - Would the xml produced by a new tool be easily human-editable?

On Sat, May 11, 2019 at 2:10 AM Carsten Bormann <[email-masked]> wrote:

> Let me extract those parts that pertain to your experiences with the
> evolution of RFCXML and add rfc-interest; please continue discussion on
> rfc-interest (which you may want to subscribe if RFC authoring is part of
> your life).
>
> > Apparently an .xml file used to prodtce an rfc is not necessarily
> acceptable to later versions of xml2rfc.  You can get an idea of the issues
> by looking at rc5661Base.xml and the file I wound up with,
> rfc5661Ready.xml; both are attached.
>
> For the latter, doing a wholesale <![CDATA[ … ]]> is always a better
> approach than sprinkling &amp;/&lt; (you almost never need &gt;, by the
> way).  (See also authoring tools below.)
>
> > I was able to get a an .xml from which a .txt file could be generated,
> but despite my best efforts there were differences (more than a few)
> between it and rfc5661, which your document would consider to be "spurious"
> but appear to be unavoidable. In any case they are not gratuitous changes
> and should not interfere with consideration of the document.   The majority
> of the diffs arise from the following  issues:
> >       • Despite the fact that the xml for the reference sections of both
> xml files are identical and the processing options are identical
> (symrefs="no" sortrefs="yes") the reference ids in rfc5661Ready.txt and in
> rfc5661 are different so that rfcdiff shows every line containing a
> reference as part of a diff.  Apparently, different versions of xml2rfc use
> different approaches to sorting references.
>
> Protip: DO NOT USE numeric references.  Ever.  This was stylistically
> appealing for some tiny documents, but rarely is appropriate for actual
> specifications (and certainly not for NFSv4-sized ones!).
>
> >       • There are a fair number of difference that seem to have arisen
> because the RFC edtitor made minor corrections directly on the .txt file so
> that each such correction (while valid) is reported by rfcdiiff as a
> difference.
>
> Right.  That will be less of a problem in the future, but it does require
> tediously porting back changes to the document source.
>
> >       • The reference sections are exposed to the same sorting issues as
> the reference id's.  To the naked eye, and to rfcdiff they look very
> different, despite that fact that they contain the same references.
>
> The sorting issue should be taken care of by not using numeric references
> Since RFC 5661, we also got DOIs on RFCs, so it is inevitable there are a
> lot of diffs.  Again, tedious, but not really avoidable.
>
> (there are now good markdown and asciidoc choices, as well as an org-mode
> one if that is your thing), but that was the way things were done in 2010.

---

From: David Noveck <[email-masked]>
Date: Mon, 13 May 2019 05:22:25 -0400

> As the references section needs to be updated anyway (for the DOIs), I’m
not sure this is really true.  Or, if it is,
> RFC 5661 maybe isn’t really a candidate for this process, because it may
be impractical to re-generate the exact
> numbering that RFC 5661 used.

I had been assuming that you could turn off sortrefs and construct a
reference section in the exact order that the
v1 tool.   I'll try some experientsto validate that approach.

.> > > Since RFC 5661, we also got DOIs on RFCs, so it is inevitable there
are a lot of diffs.
include rfc reference from external libraries.   It includes them inline,
so a new rfc derived from that xml  file will not include DOIs.

> Yes.  All these RFC references would be updated by the RFC editor into
current references.

That's not a problem for Adam's procedure :-).   The RFC candidate
considred by the IESG would still match
rfc5661.   Then, during RFC editing, the reference section could be revised
to include the DOI's.

>> That is not a problem for Adam's procedure, but it may be for the IESG
or the RFC editor.   I hope that, in processing RFC’s using Adam's
procedure, people will overlook the lack of DOIs in the same way that they
overlook other aspects of the document that would prevent a new document of
that form from being published.

>AFAICT, they can’t, as the RFC editor has committed to providing DOIs.

Please see above:

   - The IESG doesn't have to deal with the DOI diffs, since it won't see
   them.
   - The RFC editor would create DOI diffs but not be bound by Adam's
   procedure in doing so.

Where this might be a problem is in document where the existing xml uses
external reference libraries.   That avoids the need for the RFC to do
anything to provide the DOI's other that updating the libraries.   However
the IESG will see the diffs resulting from the inclusion of DOI's so I
would hope that Adam's document is clear thar they should not be considered
"spurous".

On Sun, May 12, 2019 at 12:57 PM Carsten Bormann <[email-masked]> wrote:

> If an author wants to use a V3 feature that does not lend itself to
> authoring in markdown, they can always put the XML tags right into their
> markdown source.
>
> >> Right, for a minor update, digging out the v1 tools and finding a
> platform where they can still run may actually be the best way to proceed.
> >