Subject: Require guidance on Unicode in IETF formats

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 10 Jan 2007 21:30:56 +0100

We require some guidance from our Area Directors about the use of
Unicode in an IETF format. On the mailing list [email-masked], a
discussion was raised on wether we should accept only ASCII in the
language we define (our work is to define a format, not a protocol) or
the full Unicode character set.
(http://www1.ietf.org/mail-archive/web/cosmogol/current/msg00007.html
and follow-ups.)

Some people claimed that Unicode support was more or less mandatory at
the IETF and that a format without it had no chance of being
adopted. Besides, internationalization is a very good thing, anyway,
for the world-wide Internet.

Some people feared that mandating Unicode would complicate the grammar
and would drastically reduce the number of tools available to write
parsers for this format. They think that Cosmogol, being intended
mostly for RFC or other ultra-technical usages do not have the same
requirments as a general protocol like HTTP or NNTP.

We identified the following RFC as possibly relevant:

RFC 2277 / BCP 18 IETF Policy on Character Sets and Languages

RFC 2223 Instructions to RFC Authors

RFC 3536 Terminology Used in Internationalization in the IETF

But none seems to bring a clear answer. Is Unicode support a MUST, a
SHOULD or a MAY in a new protocol?

How many *new* IETF formats are in Unicode? (Apart from those based
only on XML, like Atom in RFC 4287.) Old formats like ABNF do not
count because they derive from an older format.

_______________________________________________
Cosmogol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/cosmogol

---

From: Lisa Dusseault <[email-masked]>
Date: Wed, 10 Jan 2007 13:25:09 -0800

I am not sure I have any special wisdom or can point to any policy  
requirements on this one.   Our requirements for IETF standards to be  
i18n usually focus on what gets shown to the application user, and  
less importantly the administrative user, and not what the protocol  
implementor or debugger (or log file or wire trace) sees.  Thus error  
messages need not be Unicode and need not be translated, as long as  
it's reasonable that a client implementation could look up an ASCII  
error message or the associated code and figure out what to display  
in the user's language.  Method names and header/parameter names in  
protocols can definitely be ASCII, and not translated.

So who's the *user* of a State Machine Description Language (SMDL)?   
If I can use a SMDL to describe a bunch of states for a protocol that  
is properly i18n for the final end-user of the *protocol*, that seems  
like the minimum that we can base on general IETF policy  
requirements.  If there are further requirements, such as comments  
being i18n, those are community requirements rather than direct  
consequence of IETF/IESG policy.

I will point out since it was brought up earlier, that ABNF is  
expressed in ASCII but it *can* specify protocol syntax in UTF8 or  
another encoding.    Do we need ABNF to be able to declare rule names  
with non-ASCII characters, or to allow non-ASCII characters in  
comments?  Would we bother rewriting ABNF to make that possible?

Specific use cases may be helpful here. One use case could be "A  
German speaker communicating with German-speaking coworkers about  
their code-base needs to be able to name a state something like  
'Exclusiv Verändert' ". If everybody agrees to support that use case  
then the SMDL needs to be able to support non-ASCII (or at least  
obviously encoded) state names.   Alternatively the consensus could  
be that use case isn't necessary, either because in practice  
programmers use state machines and they're used to ASCII labels, or  
because of a decision to limit the scope of the SMDL to IETF RFCs  
where labels are even more consistently in English.

Lisa

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 11 Jan 2007 15:12:32 +0000

> where labels are even more consistently in English.

My interpretation of this is "it's up to us".

I can see two use cases that are of interest in this context. The first is
Lisa's one: should a German speaker be able to name a state or an action
something in German? If "yes", then we need at least native UTF-8. If "no",
then we can stick with ASCII.

The second one is dependent on the first: if our hypothetical German has
written a state machine in German, does she need to be able to transform
its description into ASCII, transmit it to someone else, and have him
reconstruct the original? [1] Or can we assume that she has an 8-bit clean
emailer to send it?

My personal vote: the first use case should be addressed, the second is one
we can ignore.

[1] This was the use case for ISO C that led to trigraphs, for those who
know what they are. And no, I am *not* suggesting them.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 11 Jan 2007 17:09:52 +0100

On Thu, Jan 11, 2007 at 03:12:32PM +0000,
 Clive D.W. Feather <[email-masked]> wrote 
 a message of 42 lines which said:

> should a German speaker be able to name a state or an action
> something in German? If "yes", then we need at least native UTF-8.

I technically disagree. If "yes", then we need the Unicode character
set. It can be in native UTF-8 or as an escape sequence. (See W3C's
distinction between the character model of the Web - always Unicode -
and the many encodings they can use.)

The good things about escape sequences:

* they can be sent / received with 7-bits tools,

* they can be processed with any parsing tool,

* everyone else does it, even XML.

The bad things:

* they are ugly for the writer (the casual reader may see them after a
translation to HTML or SVG or similar so it may not be ugly for him).

* they give a second-class status to non-ASCII characters, which is
bad.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 11 Jan 2007 16:27:23 +0000

Stephane Bortzmeyer said:
> set. It can be in native UTF-8 or as an escape sequence.

True. Perhaps I should have added "simply" or "easily" to the use case.

> The bad things:
> * they are ugly for the writer

This is what I see as the problem with only having escape sequences.