Subject: [rfc-i] entities and unicode

From: Miek Gieben <[email-masked]>
Date: Fri, 3 Dec 2021 11:17:48 +0100

Hello all,

In https://www.rfc-editor.org/materials/FAQ-xml2rfcv3.html it says I need to wrap unicode
characters in <u> tags (which is already a bit confusing:
https://github.com/rfc-format/draft-iab-xml2rfc-v3-bis/issues/205).

Due to some other bug, I was testing (html) entities and if I put:

     <t>this is some dashes &#x2011;</t>

In the XML, xml2rfc --text renders the "-" (but then the proper unicode one) in the text
document... which now puzzles me.

Is <u> really needed? Or are entities not allowed? Or something else that I'm not seeing?

Thanks for any insight.

/Miek

--
Miek Gieben
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Dec 2021 11:37:05 +0100

> document... which now puzzles me.

Can’t reproduce (xml2rfc 3.11.1).

The v2 support in xml2rfc turns the en-dash into a single hyphen/minus and the em-dash into a double hyphen/minus.  Umlauts turn into language specific substitutes, e.g, Keränen turns into Keraenen (great for German, ouch for Finnish).

But if I put these characters into a v3 source, they are all converted into a single hyphen/minus during rendering.
(Umlauts turn into beautiful &#228; in the plaintext and HTML [the latter with &amp; in the source]; Ker&#228;nen — I’d rather have xml2rfc turn into green smoke here.)

> Is <u> really needed? Or are entities not allowed? Or something else that I'm not seeing?

There is also <contact>, which the grammar unfortunately breaks in most contexts, but does work in plain paragraphs.

https://mailarchive.ietf.org/arch/msg/rfc-markdown/7spMIPZdz6S8_NbZZ2WcSB-ScPY

Grüße, Carsten

---

From: Miek Gieben <[email-masked]>
Date: Fri, 3 Dec 2021 11:53:55 +0100

[ Quoting <[email-masked]> in "Re: [rfc-i] entities and unicode..." ]
>Can’t reproduce (xml2rfc 3.11.1).

oh yep, I see, this was ascii-dash, your umlaut example shows it's not unicode, thanks.

>(Umlauts turn into beautiful &#228; in the plaintext and HTML [the latter with &amp; in the source]; Ker&#228;nen — I’d rather have xml2rfc turn into green smoke here.)

ugh, that's a bit suprising, i.e. it works for some code points, but not others. I agree
something should be changed here.

>https://mailarchive.ietf.org/arch/msg/rfc-markdown/7spMIPZdz6S8_NbZZ2WcSB-ScPY

ack. FWIW, having a <u format="lit"> would help as you can then use unicode in the text
more easily (without the (CJK UNIFIED IDEOGRAPH-6C34, U+6C34) stuff after it (example
taken from the xml2rfc faq)

/Miek

---

From: "John Levine" <[email-masked]>
Date: 3 Dec 2021 16:27:08 -0500

>taken from the xml2rfc faq)

The Unicode support in v3 was deliberately quite limited since we weren't sure how well it would
work and how good rendering support on people's devices would be.

It turns out that it renders just fine so we should reconsider allowing Unicode in most places
without special marking, give or take some version of "don't be ridiculous".

R's,
John

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Dec 2021 22:33:06 +0100

> without special marking, give or take some version of "don't be ridiculous".

Yes, and an important observation is that “don’t be ridiculous” is a style preference (to be enforced by the RFC editor function), not something that xml2rfc can enforce mechanically for us.

(Of course, a future version of idnits can implement warnings to support many of the style preferences.  E.g., do we want typographical quotes(*)?  Lots of fun for the RSWG-to-be :-)

Grüße, Carsten

(*) Yes.

---

From: "John R Levine" <[email-masked]>
Date: 3 Dec 2021 17:30:06 -0500

> Yes, and an important observation is that “don’t be ridiculous” is a style preference (to be enforced by the RFC editor function), not something that xml2rfc can enforce mechanically for us.

For ridiculous I was more thinking of combinations of code points that 
don't make typographic sense.  I agree that while idnits might point out 
obvious problems it's not something to enforce mechanically.

> (Of course, a future version of idnits can implement warnings to support many of the style preferences.  E.g., do we want typographical quotes(*)?  Lots of fun for the RSWG-to-be :-)

I really wish we could get rid of <tt> since it has no semantics.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Julian Reschke <[email-masked]>
Date: Sat, 4 Dec 2021 12:44:23 +0100

> without special marking, give or take some version of "don't be ridiculous".

That is slightly incorrect.

V3 (as defined in RFC 7991) does not limit the set of code points at
all. It was a decision of Henrik for xml2rfc for limit the characters,
and to introduce <u> as mandatory element for non-ASCII characters in prose.

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Sat, 4 Dec 2021 08:08:22 -0600

On 12/4/21 5:44 AM, Julian Reschke wrote:
> all. It was a decision of Henrik 

No - Heather made the decision to limit the codepoints - that's captured 
in the style guide and even in an RFC.

Henrik created the <u> to allow non-ASCII characters in specific 
circumstances working with the RPC  - he didn't make this stuff up by 
himself.

The RFC Editor (at least the time) owned the decision for if/when we 
could go to full utf-8.

(fwiw, allowing responsible use of utf-8 is something that I think we 
should try to move to sooner than later.)

> for xml2rfc for limit the characters,

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 Dec 2021 15:32:41 +0100

On 2021-12-04, at 12:44, Julian Reschke <[email-masked]> wrote:
> all.

Well, the original plan for v3 was written down in RFC 7991 to 7998, which includes RFC 7997, published December 2016.  
The latter embodies the conservative stance to “The Use of Non-ASCII Characters in RFCs” that was prevailing at the time.

Paraphrasing Moltke, no plan survives contact with reality.
So v3 of course no longer is that what was planned 5 years ago.  
For all practical purposes, what is v3 today is defined in:

https://xml2rfc.tools.ietf.org/xml2rfc-doc.html

(which of course is informed by RFC 7991 to 7998 - these are just not that useful any more as detailed reference documents).

> It was a decision of Henrik for xml2rfc for limit the characters,

There are some specific decisions Henrik thankfully made, one of which was to implement support for RFC 7997 beyond the original grammar (<contact, <u, interpreting the asciiXxx= attributes as Latin-extended instead of ASCII, etc.).  Events have prevented these necessary features from being rounded out with growing experience.

However, after 5 years, we now need to make a conscious decision to transcend RFC 7997 based on the experience we have with the v3 transition.  
Practically, this will need to wait for RFCedp to complete (any day now :-), some recruiting along the new organization for the RFC series, and some consensus forming.
This is frustrating, as, while we are waiting, RFCs are being published all the time that could be done in a better way, but I understand what’s in the way of progress.

Grüße, Carsten