Subject: Re: [rfc-i] Unicode in xml2rfc v3

From: Lars Eggert <[email-masked]>
Date: Thu, 17 Dec 2020 08:57:45 +0200

Hi,

On 2020-12-16, at 20:48, John Levine <[email-masked]> wrote:
> In article <[email-masked]> you write:
>> It's ridiculous that I can't just write α when I mean α.
> 
> I agree with you but I don't yet see how we get from here to there or exactly
> where there is.

I think it was Julian who proposed to lift the restriction on Unicode to only be allowed in <contact> and instead rely on the community (at the I-D stage) and the IESG/ISE/RPC for when I-Ds become RFCs to check for "abuse" of Unicode (which I struggle to see happening in practice.)

Lars
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Dec 2020 09:15:32 +0100

> I think it was Julian who proposed to lift the restriction on Unicode to only be allowed in <contact> and instead rely on the community (at the I-D stage) and the IESG/ISE/RPC for when I-Ds become…

That is technically the easiest approach, and what the design team
actually had in mind (not that <u> was introduced after RFC 7991 was
published).

If we want to retain the requirement to use <u> for non-ASCII, another
approach would be to relax <u> in a way that it does not insert anything
automatically.

Best regards, Julian

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 17 Dec 2020 20:19:50 +1100

> approach would be to relax <u> in a way that it does not insert anything

In terms of signaling intent to use Unicode, as opposed to accidental use (think things like smart quotes), <u> has some virtues.  This would be a perfectly acceptable outcome to me.  A new formatting option for <u> (a default ideally) would be my suggestion.

---

From: "John R Levine" <[email-masked]>
Date: 17 Dec 2020 11:51:17 -0500

> I think it was Julian who proposed to lift the restriction on Unicode to only be allowed in <contact> and instead rely on the community (at the I-D stage) and the IESG/ISE/RPC for when I-Ds become…

In general I agree but I would like some guardrails about codepoints and 
combinations that are unlikely or obscure and likely not to render 
reliably.

And, of course, there are emoji.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Lars Eggert <[email-masked]>
Date: Thu, 17 Dec 2020 18:58:30 +0200

On 2020-12-17, at 18:51, John R Levine <[email-masked]> wrote:
> In general I agree but I would like some guardrails about codepoints and combinations that are unlikely or obscure and likely not to render reliably.
> 

How likely is it that someone would try to actually use any of these, and if they did, someone wouldn't notice at some point and call it out?

In other words, is it really worth doing this? Or are those guardrails so simple that it's straightforward to do this?

Thanks,
Lars

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 17 Dec 2020 17:58:51 +0100

On 2020-12-17, at 10:19, Martin Thomson <[email-masked]> wrote:
> 
> accidental use (think things like smart quotes)

While it is possible to use typographically acceptable quotes by accident, enabling their conscious use by authors who care is a worthy consideration.

More important are dashes and minus signs; the lack of these is really jarring in the current RFCs.  Discarding the information which is which in the XML and replacing everything by the ASCII minus-hyphen character that is then rendered as a hyphen does not help.

Grüße, Carsten

---

From: "John R Levine" <[email-masked]>
Date: 17 Dec 2020 12:01:07 -0500

> How likely is it that someone would try to actually use any of these, and if they did, someone wouldn't notice at some point and call it out?

Deliberately, pretty low, by accident due to overhelpful editing tools, 
not all that low.

> In other words, is it really worth doing this? Or are those guardrails so simple that it's straightforward to do this?

Unicode has script rules we can point to.  We don't have to invent 
anything.

Regards,

---

From: tom petch <[email-masked]>
Date: Thu, 17 Dec 2020 17:10:29 +0000

> In terms of signaling intent to use Unicode, as opposed to accidental use (think things like smart quotes), <u> has some virtues.  This would be a perfectly acceptable outcome to me.  A new formatt…

Quotes are a pain in some IETF work.  There are a number of IETF WG who 
produce minutes of meetings where all quote characters appear as six 
hexadecimal digits.  Not unintelligible but a pain.  Since it keeps on 
happening, I assume that this is deliberate and given the opportunity 
will use these strings in other places too (as I recall, minutes are 
required to be in ASCII so any guidance for this context is likely to be 
ignored).

I can translate those characters into Unicode but what a waste of time.

Tom Petch

> .

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 17 Dec 2020 18:25:58 +0100

> On 2020-12-17, at 18:10, tom petch <[email-masked]> wrote:
> 
> Quotes are a pain in some IETF work.  There are a number of IETF WG who produce minutes of meetings where all quote characters appear as six hexadecimal digits.  

Pointers please.

Mistakes like these will never survive the RFC editor so I don’t think they add any additional consideration.

Grüße, Carsten

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 Dec 2020 09:00:27 +1300

> I think it was Julian who proposed to lift the restriction on Unicode to only be allowed in <contact> and instead rely on the community (at the I-D stage) and the IESG/ISE/RPC for when I-Ds become…

Agreed. I think the original decision to be very restrictive was due to general concerns about moving away from .txt as the primary format. How π, still less emojis, will be rendered in ASCII remains an issue, but just as RFC1119 worked out fine, I think we should get over the fact that some (or most) future RFCs simply won't work in .txt.

    Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Dec 2020 21:08:06 +0100

> Agreed. I think the original decision to be very restrictive was due to general concerns about moving away from .txt as the primary format. How π, still less emojis, will be rendered in ASCII remai…

We wanted to be restrictive, but we did not intend to be a restrictive
as xml2rfc (in v3 mode) turned out to be. Keep in mind that <u> is not
in RFC 7991.

And, FWIW, π shouldn't be a problem anywhere. Emojis might.

Best regards, Julian

---

From: Jay Daley <[email-masked]>
Date: Fri, 18 Dec 2020 09:10:01 +1300

> On 18/12/2020, at 9:00 AM, Brian E Carpenter <[email-masked]> wrote:
> 

Can someone explain to me why our .txt files have to be ASCII?  I doubt I have a single text tool left that can’t process UTF-8.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Dec 2020 21:26:39 +0100

Am 17.12.2020 um 21:10 schrieb Jay Daley:
> ...
> Can someone explain to me why our .txt files have to be ASCII?  I doubt
> I have a single text tool left that can’t process UTF-8.
> ...

They don't have to, and they haven't (in modern times) since
<https://www.rfc-editor.org/rfc/rfc8187.txt> (over three years ago).

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 Dec 2020 10:28:33 +1300

RFC 7997 opened the door, but not very wide. In earlier days ASCII was enforced.

Regards
   Brian Carpenter

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Dec 2020 00:57:24 +0100

> RFC 7997 opened the door, but not very wide. In earlier days ASCII was enforced.

A very typical mistake.

When some direction of travel is controversial, there is a tendency to make “compromises”.  This can have the effect that all the costs of taking that direction accrue (because they are not actually influenced by the compromise), but the benefits are artificially limited.

That is exactly what happened here.

We should have embraced the change that emanated from RFC 2277 and the contemporary groundswells in the industry, and two decades later (almost a decade after the first hesitant steps were made with the RFC format), the current status looks comical.

Emojis in RFCs don’t have to be ruled out by some mechanism (just as writing RFCs that are entirely about farting doesn't have to be); having a bit of (much more flexible) policy about this is fine (although we somehow never needed that for the second example).

By the way, authoring tools can still help prevent accidents; smart quotes were mentioned, as probably should be BOMs, various forms of zero-width (invisible) characters etc.  This is particularly relevant for source-code and code-spans, which can be mangled badly when such accidents happen.  None of this is new with going beyond ASCII, as various problems with backslashes in nroff times should remind us of, or the HT interpretation issue that led to the RFC 7386/7396 disaster.

Grüße, Carsten

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 17 Dec 2020 16:29:52 -0800

On Thu, Dec 17, 2020 at 3:57 PM Carsten Bormann <[email-masked]> wrote:

> Emojis in RFCs don’t have to be ruled out by some mechanism (just as
> writing RFCs that are entirely about farting doesn't have to be); having a
> bit of (much more flexible) policy about this is fine (although we somehow
> never needed that for the second example).
>

FWIW, while I don't generally think we ought to have our documents written
in emojis, I do see some potentially valid reasons for them to appear, for
instance if one was writing a document about the use of emojis.

-Ekr

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 Dec 2020 14:56:50 +1300

On 18-Dec-20 13:29, Eric Rescorla wrote:
> 
> 
> On Thu, Dec 17, 2020 at 3:57 PM Carsten Bormann <[email-masked] <mailto:[email-masked]>> wrote:
> 
> FWIW, while I don't generally think we ought to have our documents written in emojis, I do see some potentially valid reasons for them to appear, for instance if one was writing a document about the use of emojis.

Correct. Or we might want to use Cyrillic in a document about deçeptive DNS domain names. But then, regardless of what we want to prohibit via the style guide, we'd need to be explicit that .txt files are UTF-8. I could be mistaken, but I don't think we've said that in any current RFC.

     Brian

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 18 Dec 2020 13:12:18 +1100

On Fri, Dec 18, 2020, at 12:56, Brian E Carpenter wrote:
> I could be mistaken, but I don't think we've said that in any current RFC.

https://tools.ietf.org/html/rfc7994#section-2 seems pretty clear to me.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 Dec 2020 15:28:25 +1300

> https://tools.ietf.org/html/rfc7994#section-2 seems pretty clear to me.

Ah, right. It's the "limited and controlled fashion" that we are debating, then.

    Brian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Dec 2020 07:45:15 +0100

> When some direction of travel is controversial, there is a tendency to make “compromises”.  This can have the effect that all the costs of taking that direction accrue (because they are not actuall…

On the other hand, the idea was to publish "snapshots" of the drafts we
were developing, and to revise them as we gain experience. That part of
the plan has failed completely (we're now over four years after approval).

Best regards, Julian

---

From: tom petch <[email-masked]>
Date: Fri, 18 Dec 2020 12:51:18 +0000

>> Quotes are a pain in some IETF work.  There are a number of IETF WG who produce minutes of meetings where all quote characters appear as six hexadecimal digits.

LSR, RTGWG

Tom Petch

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Dec 2020 14:01:57 +0100

Am 18.12.2020 um 13:51 schrieb tom petch:
>>>> In terms of signaling intent to use Unicode, as opposed to
>>>> accidental use (think things like smart quotes), <u> has some
>>>> virtues.  This would be a perfectly acceptable outcome to me.  A new
>>>> formatting option for <u> (a default ideally) would be my suggestion.
>>>
>>> Quotes are a pain in some IETF work.  There are a number of IETF WG
>>> who produce minutes of meetings where all quote characters appear as
>>> six hexadecimal digits.
>>
>> Pointers please.
>
>
> LSR, RTGWG

Ok, looked at <https://datatracker.ietf.org/doc/minutes-109-rtgwg/> and
didn't see anything.

Could you be more specific?

Best regards, Julian

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 18 Dec 2020 12:13:32 -0500

On 12/17/20 7:29 PM, Eric Rescorla wrote:

> FWIW, while I don't generally think we ought to have our documents 
> written in emojis, I do see some potentially valid reasons for them to 
> appear, for instance if one was writing a document about the use of emojis.

Agree.

Similarly, I don't think we should totally ban graphics that contain 
color or greyscale. There are good reasons to restrict that sort of 
usage to make documents accessible to more people. But surely there are 
exceptions. For instance a document that is explicitly concerned with 
color and greyscale.

	Thanks,
	Paul

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Thu, 17 Dec 2020 13:27:42 -0800

> Can someone explain to me why our .txt files have to be ASCII?  I doubt I have a single text tool left that can’t process UTF-8.

Because non-ASCII characters cannot improve a well-written RFC -- or a well-written program for that matter.  Graphics and diagrams in place of formal descriptions (in ASCII) are also signs of the sloppiness of the authors -- or at least of the little faith they have on their implementers.

-- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: https://marc.petit-huguenin.org
Profile: https://www.linkedin.com/in/petithug
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Dec 2020 18:24:16 +0100

On 2020-12-17, at 22:27, Marc Petit-Huguenin <[email-masked]> wrote:
> 
> Because non-ASCII characters cannot improve a well-written RFC

Not true.

(I’ll leave that as unsupported as the statement above, but I think there is enough evidence in this thread already.)

Grüße, Carsten

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 18 Dec 2020 15:27:24 -0500

Actually, gien the differences in rendition, documents (if we ever have 
such) about color or gray scale may be exactly when you do NOT want to 
use color and grayscale.  Slight mis-rendition could easily result in 
quite confusing documents.

Our initial agreement did include that we would revisit this.  And get 
publication expertise on the known problems in this space would be very 
helpful.  (We are not the first folks to wrestle with these issues.)

Yours,
Joel

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 18 Dec 2020 14:38:07 -0800

On Fri, Dec 18, 2020 at 9:16 AM Marc Petit-Huguenin <[email-masked]>
wrote:

> >>> I think it was Julian who proposed to lift the restriction on Unicode
> to only be allowed in <contact> and instead rely on the community (at the
> I-D stage) and the IESG/ISE/RPC for when I-Ds become RFCs to check for
> "abuse" of Unicode (which I struggle to see happening in practice.)
> >>
> >> Agreed. I think the original decision to be very restrictive was due to
> general concerns about moving away from .txt as the primary format. How π,
> still less emojis, will be rendered in ASCII remains an issue, but just as
> RFC1119 worked out fine, I think we should get over the fact that some (or
> most) future RFCs simply won't work in .txt.
> >
> Because non-ASCII characters cannot improve a well-written RFC -- or a
> well-written program for that matter.

I don't think I agree with this. For instance, while yes it's possible to
formally specify the code points for emoji in hex or whatever, it's
certainly easier to read the text if you can see them, I think.

-Ekr

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Fri, 18 Dec 2020 15:56:13 -0800

> certainly easier to read the text if you can see them, I think.

CLDR short names, prefixed in a similar way than Unicode codepoints (e.g. <N+ROMAN NUMERAL NINE> for <U+2178>) could still be translated to their graphical equivalent in HTML and PDF, but will not make me want to gouge my eyes when implementing from the text format.

In a way, this is to programmers what the alt attribute is for the blind and visually impaired.

(Yes, I believe that the printed paginated text version of a standard is the proper way to implement a standard.  And carrying non-hypertext files in HTTP is in bad taste.  And milk should be poured first in the tea cup)

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 19 Dec 2020 07:41:14 +0100

> CLDR short names, prefixed in a similar way than Unicode codepoints (e.g. <N+ROMAN NUMERAL NINE> for <U+2178>) could still be translated to their graphical equivalent in HTML and PDF, but will not…

That’s what <u> would be good for.

But if I have, say, a formal notation that uses ⅸ as a normal part of writing, just as people talking about congestion control are likely to use α and β, the text must not be littered with explanations any more than it would be when using seriously weird characters such as *, @, or #.

That would also enable RFCs to exhibit typographic normalcy e.g. in the use of dashes and minus signs, which need to be distinct from hyphens in the non-monospace fonts used now for RFCs in their HTML and PDF forms.

Grüße, Carsten