Subject: [rfc-i] line wrapping in XML

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 11:23:42 +1100

When editing a source document (especially Markdown), many people seem to find it best to NOT line-wrap paragraphs, so that diffs (e.g., on GitHub) are minimal; otherwise re-wrapping causes a lot of noise in diffs during the WG process.

Currently, XML created from markdown using kramdown-rc2629 seems to mirror what the source document contains; if there are line breaks, they will be preserved, but if not, they will not be created.

When an XML document goes to the RFC Editor, it will be line-wrapped (apparently manually). This causes difficulties in creating an XML diff from the source document to the one that's being offered for approval -- something that I feel is necessary, given that XML is now the canonical format.

Currently, I'm using Apple FileMerge to do the diff, because it has the ability to ignore whitespace; however, it still highlights it, so that's not optimal.

Is there something we can do to improve this? I don't think wrapping the source files is a solution, since that reduces utility for document authors, but off the top of my head:

1. Can we identify a freely available tool for correctly line-wrapping XML that people can insert into their toolchain pre-RPC, so that the RPC doesn't need to hand-wrap lines?

2. Can we ask the RPC to stop line-wrapping by default?

3. ???

Cheers,

--
Mark Nottingham   https://www.mnot.net/

_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 30 Oct 2020 00:26:55 +0000

>    1. Can we identify a freely available tool for correctly line-wrapping XML that people can insert into their toolchain pre-RPC, so that the RPC doesn't need to hand-wrap lines?

https://www.html-tidy.org/

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 11:28:01 +1100

Nope; tried, but it removes spaces in odd places that are semantically significant.

> On 30 Oct 2020, at 11:26 am, Salz, Rich <[email-masked]> wrote:
> 
>

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 30 Oct 2020 11:32:53 +1100

You can configure tidy to preserve whitespace in certain elements, so it seems like a viable option.

 <option class="TidyMarkupTeach">
  <name>new-pre-tags</name>
  <type>Tag Names</type>
  <default />
  <example>tagX, tagY, ...</example>
  <description>This option specifies new tags that are to be processed in exactly the same way as HTML's <code>&lt;pre&gt;</code> element. This option takes a space or comma separated list of tag names. <br/>Unless you declare new tags, Tidy will refuse to generate a tidied file if the input includes previously unknown tags. <br/>Note you cannot as yet add new CDATA elements. <br/>This option is ignored in XML mode. </description>
  <seealso>new-blocklevel-tags</seealso>
  <seealso>new-empty-tags</seealso>
  <seealso>new-inline-tags</seealso>
  <seealso>custom-tags</seealso>
    <eqconsole />
 </option>

-- from `tidy -xml-config`

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 30 Oct 2020 11:35:50 +1100

And I should read what I paste.  This says that it ignores the option for XML mode, which isn't great.  I think that you might have to use xml:space="preserve" on those elements that you want to protect, so a wrapper might be needed.  It's a small wrapper though.

On Fri, Oct 30, 2020, at 11:32, Martin Thomson wrote:
>   <description>This option specifies new tags that are to be processed 
> in exactly the same way as HTML's <code>&lt;pre&gt;</code> element. 
> This option takes a space or comma separated list of tag names. 
> <br/>Unless you declare new tags, Tidy will refuse to generate a tidied 
> file if the input includes previously unknown tags. <br/>Note you 
> cannot as yet add new CDATA elements. <br/>This option is ignored in 
> XML mode. </description>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 29 Oct 2020 20:52:49 -0400

Mark Nottingham <[email-masked]> wrote:
    > When editing a source document (especially Markdown), many people seem
    > to find it best to NOT line-wrap paragraphs, so that diffs (e.g., on
    > GitHub) are minimal; otherwise re-wrapping causes a lot of noise in
    > diffs during the WG process.

Yup.
I like once line per sentence though.
I used to make my XML and my markdown all pretty, but I stopped once I taught
my editor to make it pretty as I type.

    > When an XML document goes to the RFC Editor, it will be line-wrapped
    > (apparently manually). This causes difficulties in creating an XML diff
    > from the source document to the one that's being offered for approval
    > -- something that I feel is necessary, given that XML is now the
    > canonical format.

When you say that they get wrapped, does it just the wrap long lines to
multiple lines, or does it put lines together which are in different
sentences? To take the above paragraph, do we get:

    > When an XML document goes to the RFC Editor, it will be line-wrapped
    > (apparently manually). This causes difficulties in creating an XML diff
    > from the source document to the one that's being offered for approval
    > -- something that I feel is necessary, given that XML is now the
    > canonical format.

or:

    > When an XML document goes to the RFC Editor, it will be line-wrapped
    > (apparently manually).
    > This causes difficulties in creating an XML diff from the source
    > document to the one that's being offered for approval -- something that
    > I feel is necessary, given that XML is now the canonical format.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Michael StJohns <[email-masked]>
Date: Thu, 29 Oct 2020 20:54:52 -0400

> 1. Can we identify a freely available tool for correctly line-wrapping XML that people can insert into their toolchain pre-RPC, so that the RPC doesn't need to hand-wrap lines?

Emacs SGML  or nXML mode - C-x h M-x sgml-pretty-print

Mike

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 11:56:51 +1100

The problem is that this happens in blocks like this:

<t>This document introduces a set of common data structures for use in definitions of new HTTP field values to address these problems. In particular, it defines a generic, abstract model for them, along with a concrete serialization for expressing that model in HTTP <xref target="RFC7230" format="default"/> header and trailer fields.</t>

which becomes:

<t>This document introduces a set of common data structures for use in
definitions of new HTTP field values to address these problems. In
particular, it defines a generic, abstract model for them, along with a
concrete serialization for expressing that model in HTTP 
<xref target="RFC7230" format="default" />header and trailer fields.</t>

Notice the removed space after the <xref> element. I don't think xml:space="preserve" is the solution here.

This has been reported to Tidy many times over the years:
 https://sourceforge.net/p/tidy/bugs/663/
 https://sourceforge.net/p/tidy/bugs/740/
 https://github.com/htacg/tidy-html5/issues/818

Cheers,

> On 30 Oct 2020, at 11:35 am, Martin Thomson <[email-masked]> wrote:
>

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 30 Oct 2020 11:56:41 +1100

I just tested tidy and it's almost good enough, but it has one major problem that I've found.

FWIW, this is what I ran:
tidy -xml -o tls.xml -w 80 -i draft-ietf-quic-tls.xml

The problem is that there are places in the document like this:

> This document describes how QUIC <xref target="QUIC-TRANSPORT" format="default" /> is

And tidy removes the space after the <xref/>, which is quite unfortunate.

On Fri, Oct 30, 2020, at 11:35, Martin Thomson wrote:
> And I should read what I paste.  This says that it ignores the option 
> for XML mode, which isn't great.  I think that you might have to use 
> xml:space="preserve" on those elements that you want to protect, so a 
> wrapper might be needed.  It's a small wrapper though.
>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 11:57:24 +1100

Sorry, nothing can convince me to become an Emacs user.

Cheers,

> On 30 Oct 2020, at 11:54 am, Michael StJohns <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Oct 2020 01:58:23 +0100

> I like once line per sentence though.

(Yep.  I learned this as “classical keyboarding”.  We discussed this over in https://github.com/cabo/kramdown-rfc2629/issues/80 …)

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 12:00:31 +1100

> On 30 Oct 2020, at 11:52 am, Michael Richardson <[email-masked]> wrote:
> 
> 

Exactly.

>> I feel is necessary, given that XML is now the canonical format.

E.g., 

      <t>Specifying the syntax of new HTTP header (and trailer) fields is an onerous task; even with the guidance in <xref target="RFC7231" section="8.3.1" format="default"/>, there are many decisions -- and pitfalls -- for a prospective HTTP field author.</t>
      <t>Once a field is defined, bespoke parsers and serializers often need to be written, because each field value has a slightly different handling of what looks like common syntax.</t>

becomes

      <t>Specifying the syntax of new HTTP header (and trailer) fields is an
      onerous task; even with the guidance in <xref target="RFC7231" section="8.3.1"/>, there are many decisions -- and
      pitfalls -- for a prospective HTTP field author.</t>
      <t>Once a field is defined, bespoke parsers and serializers often need
      to be written, because each field value has a slightly different handling
      of what looks like common syntax.</t>

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 12:04:09 +1100

> On 30 Oct 2020, at 11:58 am, Carsten Bormann <[email-masked]> wrote:
> 

*confused* My understanding of what you wrote there was that it was still wrapped:

> In particular if you can resist the temptation to reformat even sentences (unless the reach the margin of the TTY, in which case you just break that line).

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Oct 2020 02:14:16 +0100

> *confused* My understanding of what you wrote there was that it was still wrapped:

Yes, that is the classical way (you MUST wrap at some point on punched cards(*)).

Here the challenge is to resist simply refilling the sentence after each change (until that becomes an obvious improvement).
So you wrap, but don’t refill.

But in auto-flowing environments like this email, I don’t normally use wrapping (unless I copy text from an editor), and you might catch me writing another sentence on a single line.
But then we don’t diff mail...

Grüße, Carsten

(*) Actually, one's prose gets mightily better if each sentence does fit on a punched card :-)

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2020 21:14:50 -0400

So run it in batch mode from the command line. 

> On Oct 29, 2020, at 20:57, Mark Nottingham <[email-masked]> wrote:
>

---

From: Michael StJohns <[email-masked]>
Date: Thu, 29 Oct 2020 21:14:56 -0400

It's good then that you don't have to (become an emacs user that is):

I haven't tested this but I think the command line version is:

      emacs -batch <filename>.xml -f mark-whole-buffer -f 
sgml-pretty-print -f save-buffer

Mike

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2020 21:16:09 -0400

That works if you don’t mind overwriting the original. 

> On Oct 29, 2020, at 21:15, Michael StJohns <[email-masked]> wrote:
> 
>      emacs -batch <filename>.xml -f mark-whole-buffer -f sgml-pretty-print -f save-buffer
> 
> Mike
> 
> 
>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 12:18:13 +1100

Thanks! Is it possible to control the wrap column?

> On 30 Oct 2020, at 12:14 pm, Michael StJohns <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Oct 2020 02:24:08 +0100

> Thanks! Is it possible to control the wrap column?

You can set fill-column, but I do get some weird results.
Also note this snippet from the source:

  ;; TODO:
  ;; - insert newline between some start-tag and text.
  ;; - don't insert newline in front of some end-tags.

Especially the last TODO is a bit jarring.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 30 Oct 2020 12:25:25 +1100

> Especially the last TODO is a bit jarring.

Yes, I'm seeing a lot of those, which is not great.

---

From: Michael StJohns <[email-masked]>
Date: Thu, 29 Oct 2020 21:39:19 -0400

> Yes, I'm seeing a lot of those, which is not great.
Like I said, I haven't tested any of this, but you can set the value of 
fill-column on the command line I think as --eval "(setq fill-column 
72)", or on a per document basis by including a formatting comment on 
the first line (and assuming this is preserved by the RPC, probably the 
best way to get a consistent document):

<!-- -*- mode: nXML; fill-column: 72 -*- -->

or alternately at the end of the file:

<!-- Local Variables: -->
<!-- fill-column: 72 -->
<!-- End: -->

With respect to the TODO stuff - one possibility is to split off a copy 
of the source and modify it to meet IETF needs - obviously renaming the 
function.

Later, Mike

---

From: Michael StJohns <[email-masked]>
Date: Thu, 29 Oct 2020 22:01:33 -0400

Again - without testing this I think its:  replace the -f save-buffer 
with  --eval '(write-file "newfile.xml" t)'

Mike

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 30 Oct 2020 11:14:09 -0000

Sorry to be jumping into the topic late...

Didn't someone write an RFC about this? With an example script for wrapping, unwrapping?

RFC 8792.

Best,
Adrian

-----Original Message-----
From: rfc-interest <[email-masked]> On Behalf Of Michael StJohns
Sent: 30 October 2020 02:02
To: Ted Lemon <[email-masked]>
Cc: [email-masked]
Subject: Re: [rfc-i] line wrapping in XML

Mike

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 31 Oct 2020 10:40:18 +1100

That's for examples in RFC content. This is about the line wrapping that the RPC does to the XML that now gets published as the canonical version of the RFC itself; they do so manually, which causes unpleasant diffs with what's submitted.

I'm starting to think that if we can't find a widely usable way to line-wrap XML before submission without errors (it's a shame about Tidy), we should consider asking the RPC to stop line-wrapping.

Cheers,

> On 30 Oct 2020, at 10:14 pm, Adrian Farrel <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 31 Oct 2020 21:41:30 +0100

On 2020-10-31, at 00:40, Mark Nottingham <[email-masked]> wrote:
> unpleasant diffs

The obvious fix is to normalize both sides to be compared and do a diff on the normalized versions.

An imperfect normalizer such as tidy may be good enough for this.
(Assuming that there also will be some proofreading and not just diff checking.)

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 1 Nov 2020 13:21:19 +1100

> On 1 Nov 2020, at 7:41 am, Carsten Bormann <[email-masked]> wrote:
> 

I don't think that will work; if we use Tidy pre-submission, it's going to introduce errors that have to be manually corrected at some point. if we use Tidy to create a version of the XML to facilitate comparison, it's of little utility, because the RPC will very likely break lines at different places (since they do it by hand, AIUI).

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 1 Nov 2020 06:21:35 +0100

> I don't think that will work; if we use Tidy pre-submission, it's going to introduce errors that have to be manually corrected at some point. if we use Tidy to create a version of the XML to facili…

What I was trying to say was that the comparison operation (and that only) should involve normalization of both inputs to be compared.  You don’t use the normalization outputs for anything else.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 1 Nov 2020 16:22:58 +1100

> On 1 Nov 2020, at 4:21 pm, Carsten Bormann <[email-masked]> wrote:
> 

I understand. I'm just uncomfortable comparing to anything other than what actually is being approved.

Cheers,