Subject: [Xml-sg-cmt] Regression test over existing RFCs

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Sep 2021 11:37:07 -0500

Kesara added a simple test endpoint that just runs the RFCs published as 
XML through xml2rfc asking for text, html, and pdf output.

Note that this first attempt takes almost 6 hours to complete on a fast 
machine.

(env) rjsparks@unformal cli % time make rfcregressiontest > 
~/tmp/rfcregress.txt 2>&1
make rfcregressiontest > ~/tmp/rfcregress.txt 2>&1 20453.07s user 
210.72s system 99% cpu 5:47:43.40 total

The output for the current code produces no errors, but it produces a 
_lot_ of warnings. I'm wondering if those have changed since publication 
per-rfc or if we're throwing warnings that the RPC is deciding to not 
make changes around (which means they are likely noise and need to be 
adjusted).

There is an inflation factor of 3-ish on some of these numbers because 
the current test runs the text, html, and pdf generation separately, but 
the data is still usable.

99 : Found tab on line
33 : Unused reference: There seems to be no reference to
30 : Artwork too wide, reducing indentation from
30 : Postal address part filled in, but not used
30 : Available elements for <whatever> are
30 : Expected a title or title abbreviation of not more than 40 character
17 : Figure too wide, reducing indentation from
4 : Too long line found

Should we change how we are warning on any of those things? Were the 
ones reporting tabs and width violations intentional exceptions?

These are consequences of testing after publication or testing the 
prepped xml - we should mute them for our regression test:

1488 : using existing pn
1353 : more than 3 days away
3: Found an existing Index section, not inserting another one

There are some warnings about nbsp that i don't know how to categorize:
          3887, 3903, 3943, 3983, 4097, 4111, <200b>
          4642, 4680, 4681, 4712, 4732, 4743, <200b>
          5024, 5049, 5054, 5091, 5158, 5216, <200b>
          6083, 6084, 6176, 6347, 6353, 6367, <200b>

Kesara - I think we need these changes:
1) Run xml2rfc generating all 3 outputs at once instead of having three 
separate loops (provide --text --html and --pdf on the same command)
2) Run over the corpus again, unprepping, then making sure xml2rfc 
doesn't error on the unprepped variant
3) For the sake of sanity - filter the known warnings and ok static 
output (like " Created <file>") away

---

From: Alice Russo <[email-masked]>
Date: Tue, 21 Sep 2021 09:48:08 -0700

Hi Robert,

Re:
> 2) Run over the corpus again, unprepping, then making sure xml2rfc doesn't error on the unprepped variant

I wonder if it makes sense to run on the publicly available "not prepped" files? Meaning the files available here: https://www.rfc-editor.org/in-notes/prerelease/

Rationale: The prerelease file was the input to create the RFC.
The --unprep option has not been tested very much, AFAIK. The --unprep output I've reviewed seems to be a partial "unprepping", e.g., each item's derivedCounter is still present in any <ol>.

Although the differences (between not prepped vs. unprepped) should be minor, if there were a decision to make updates to the RFC XML files before they were prepped, my guess is the prerelease/ files would be used rather than by running "--unprep" on those files (which creates a new "intermediate" file).

Thanks,
Alice

---

From: "John R Levine" <[email-masked]>
Date: 21 Sep 2021 12:55:37 -0400

> The --unprep option has not been tested very much, AFAIK. The --unprep output I've reviewed seems to be a partial "unprepping", e.g., each item's derivedCounter is still present in any <ol>.

If it doesn't remove all the prep items, that seems like a bug.

Would anything break if we fixed it?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Sep 2021 12:30:09 -0500

> Although the differences (between not prepped vs. unprepped) should be minor, if there were a decision to make updates to the RFC XML files before they were prepped, my guess is the prerelease/ fil…
Do you think that's someone who is going to make a bis from a published 
document is going to know to do?

---

From: "John R Levine" <[email-masked]>
Date: 21 Sep 2021 13:39:27 -0400

>> Although the differences (between not prepped vs. unprepped) should be 
>> minor, if there were a decision to make updates to the RFC XML files before 
>> they were prepped, my guess is the prerelease/ files would be used rather 
>> than by running "--unprep" on those files (which creates a new 
>> "intermediate" file).

I hope they would if we tell them.  The unprep step is no harder than 
any other invocation of xml2rfc.

I realze that telling people in the IETF and having them pay attention is 
not the same thing.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Sep 2021 12:48:10 -0500

Your context-away-filter may be set too high?

On 9/21/21 12:39 PM, John R Levine wrote:
>>> Although the differences (between not prepped vs. unprepped) should 
>>> be minor, if there were a decision to make updates to the RFC XML 
>>> files before they were prepped, my guess is the prerelease/ files 
>>> would be used rather than by running "--unprep" on those files 
>>> (which creates a new "intermediate" file).
>> Do you think that's someone who is going to make a bis from a 
>> published document is going to know to do?
>
Just to make sure we didn't get lost in ambiguous antecedents, I meant 
"Do you think someone that's going to make a bis would know to ask for 
the files the RPC has tucked away[1] instead of just running unprep".
>
> I realze that telling people in the IETF and having them pay attention 
> is not the same thing.
>
> Regards,

[1] Digging back in the thread, Alice said

> I wonder if it makes sense to run on the publicly available "not prepped" files? Meaning the files available here:https://www.rfc-editor.org/in-notes/prerelease/

---

From: "John R Levine" <[email-masked]>
Date: 21 Sep 2021 14:40:41 -0400

> Your context-away-filter may be set too high?

Partly that, partly I'm confused.

I would hope that if you take an XML file, prep it, and then unprep it, 
the result should be what you started with give or take white space and 
maybe entity recoding.  So it shouldn't matter whether people use the 
unprepped files on the web site or unprep them themselves.

If that doesn't work I think we should at least try to fix it.

R's,
John

>>>> minor, if there were a decision to make updates to the RFC XML files 
>>>> before they were prepped, my guess is the prerelease/ files would be used 
>>>> rather than by running "--unprep" on those files (which creates a new 
>>>> "intermediate" file).
>> I hope they would if we tell them.  The unprep step is no harder than any 
>> other invocation of xml2rfc.
> Just to make sure we didn't get lost in ambiguous antecedents, I meant "Do 
> you think someone that's going to make a bis would know to ask for the files 
> the RPC has tucked away[1] instead of just running unprep".
>> 
>> I wonder if it makes sense to run on the publicly available "not prepped" 
>> files? Meaning the files available 
>> here:https://www.rfc-editor.org/in-notes/prerelease/
>
>

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Sep 2021 14:07:49 -0500

On 9/21/21 1:40 PM, John R Levine wrote:
> I would hope that if you take an XML file, prep it, and then unprep 
> it, the result should be what you started with give or take white 
> space and maybe entity recoding.  So it shouldn't matter whether 
> people use the unprepped files on the web site or unprep them themselves.
>

So, Alice reported earlier in the thread that it doesn't.

Using Jim's rfc-xmldiff tool: 
http://www.nostrum.com/~rjsparks/not-vs-un-prepped/diff.html shows the 
kind of differences.

Alice - I tried to use some other xmldiff-ing tools and found that the 
notprepped versions aren't valid XML (for lack of entity definitions)  - 
would it be easy to fix at least that?

>>>>> Although the differences (between not prepped vs. unprepped) 
>>>>> should be minor, if there were a decision to make updates to the 
>>>>> RFC XML files before they were prepped, my guess is the 
>>>>> prerelease/ files would be used rather than by running "--unprep" 
>>>>> on those files (which creates a new "intermediate" file).
>>> I hope they would if we tell them.  The unprep step is no harder 
>>> than any other invocation of xml2rfc.
>> Just to make sure we didn't get lost in ambiguous antecedents, I 
>> meant "Do you think someone that's going to make a bis would know to 
>> ask for the files the RPC has tucked away[1] instead of just running 
>> unprep".
>>>
>>> I realze that telling people in the IETF and having them pay 
>>> attention is not the same thing.
>>>
>>> Regards,
>>> I wonder if it makes sense to run on the publicly available "not 
>>> prepped" files? Meaning the files available

---

From: "John R Levine" <[email-masked]>
Date: 21 Sep 2021 23:25:25 -0400

> http://www.nostrum.com/~rjsparks/not-vs-un-prepped/diff.html shows the kind 
> of differences.

FYI, I found that just a regular diff -b on the two XML file showed the 
differences pretty clearly.

Regards,

---

From: Alice Russo <[email-masked]>
Date: Wed, 22 Sep 2021 10:56:56 -0700

Robert wrote:
> Alice - I tried to use some other xmldiff-ing tools and found that the notprepped versions aren't valid XML (for lack of entity definitions)  - would it be easy to fix at least that?

If this is about replacing each line as follows, this seemingly could be done programmatically.

Replace
<!DOCTYPE rfc SYSTEM "rfc2629-xhtml.ent"> 
with 
<!DOCTYPE rfc [
 <!ENTITY nbsp  "&#160;">
 <!ENTITY zwsp  "&#8203;">
 <!ENTITY nbhy  "&#8209;">
 <!ENTITY wj   "&#8288;">
]>

Thanks,
Alice

---

From: Robert Sparks <[email-masked]>
Date: Wed, 22 Sep 2021 13:34:55 -0500

Yes, that made it so that pypi.org/project/xmldiff could consume the 
document. Sadly, its output was not helpful.

John do you think its worth programmatically making the substitution 
below in the notprepped versions anyhow?

RjS

---

From: "John R Levine" <[email-masked]>
Date: 22 Sep 2021 14:48:33 -0400

> John do you think its worth programmatically making the substitution below in 
> the notprepped versions anyhow?

I think that anything that removes dependencies on RFC 2629 is a step 
forward, so yes.

Regards,