Subject: [EAI] proposal to permit unquoted UTF8 display_name

From: Soobok Lee <[email-masked]>
Date: Sun, 4 Feb 2007 13:39:59 +0900

[utf8headers] already permit this quoted form by redefining
qtext to have utf-8 characters:
  1)	To: "cafe'" <[email-masked]>,

But, it does not permit this unquoted one:
  2)	To: cafe' <[email-masked]>,
  3)	To: IMA <[email-masked]>,

Please read the rfc2822/[utf8headers] excerpts below.

If we are happy only with 2) and need not 1), 
 we can ignore this issue.

But, if we need to permit 1) in order be consistent with permitting 2),
we may need to redefine "phrase" and its subcomponents.

<quote from rfc2822>

name-addr       =       [display-name] angle-addr

display-name    =       phrase

phrase          =       1*word / obs-phrase

word            =       atom / quoted-string

obs-phrase      =       word *(word / "." / CFWS)

atom            =       [CFWS] 1*atext [CFWS]

dot-atom        =       [CFWS] dot-atom-text [CFWS]

dot-atom-text   =       1*atext *("." 1*atext)

atext           =       ALPHA / DIGIT / ; Any character except controls,
                        "!" / "#" /     ;  SP, and specials.
                        "$" / "%" /     ;  Used for atoms
                        "&" / "'" /
                        "*" / "+" /
                        "-" / "/" /
                        "=" / "?" /
                        "^" / "_" /
                        "`" / "{" /
                        "|" / "}" /
                        "~"
</quote>

[utf8headers] does not redefine "atext" and "atom", by the reason
described in itself (see below), and instead defined separate"utf8-atext"
and "utf8-atom". And, its parent "phrase" is not redefined to
contain "utf8-atom".

<quote from [utf8headers]>

6.2.  Syntax extend from RFC 2822

   The following rules are intended to extend the corresponding rules in
   RFC 2822 to allow UTF8 characters.

   ctext   =  NO-WS-CTL /     ; all of <text> except
              %d33-39 /       ; SP, HTAB, "(", ")"          
              %d42-91 /       ; and "\"
              %d93-126 /
              UTF8-xtra-char
   
   qtext   =       NO-WS-CTL /     ; all of <text> except
              %d33 /               ; The rest of the US-ASCII
              %d35-91 /        ; characters not including "\" 
              %d93-126 /       ; or the quote character
              UTF8-xtra-char
                           
   text    =  %d1-9 /         ; all UTF-8 characters except
              %d11-12 /       ; US-ASCII NUL, CR and LF
              %d14-127 /
              UTF8-xtra-char 
   
   utext   =  NO-WS-CTL /     ; Non white space controls
              %d33-126 /      ; The rest of US-ASCII
              UTF8-xtra-char
   
   This means that all the RFC 2822 constructs that build upon these
   will permit UTF-8 characters, including comments and quoted strings.
   Besides, in order to allow UTF8 characters in <addr-spec> we have to
   change the syntax of <atext>.  However, it will also lead <mesg-id>
   to allow UTF8 characters, which is not allowed due to the limitation
   describe in Section 6.4.  So <utf8-atext> is added to meet this
   requirement.
   
   utf8-atext   =  ALPHA / DIGIT /
                   "!" / "#" /     ; Any character except
                   "$" / "%" /     ; controls, SP, and specials.
                   "&" / "'" /     ; Used for atoms
                   "*" / "+" /
                   "-" / "/" /
                   "=" / "?" /
                   "^" / "_" /
                   "`" / "{" /
                   "|" / "}" /
                   "~" /
                   UTF8-xtra-char

   utf8-atom     = [CFWS] 1*utf8-atext [CFWS]

   utf8-dot-atom = [CFWS] utf8-dot-atom-text [CFWS]

   utf8-dot-atom-text = 1*utf8-atext *("." 1*utf8-atext)

</quote>

We need to redefine "phrase" and define new "utf8-word".

<quote : I propose to add to [utf8headers] >

phrase          =       1*utf8-word / obs-phrase 

utf8-word       =       utf8-atom / quoted-string

obs-phrase      =       utf8-word *(utf8-word / "." / CFWS)

</quote> 

Soobok

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Soobok Lee <[email-masked]>
Date: Sun, 4 Feb 2007 13:43:15 +0900

>   3)	To: IMA <[email-masked]>,

[NOTE]   3) is already permitted in RFC2822,
I  put this together with 2) to ease comparison and 
clarify the need for redefiniton of "phrase".

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Sun, 4 Feb 2007 13:56:01 +0900

On Sun, Feb 04, 2007 at 01:39:59PM +0900, Soobok Lee wrote:
>   3)	To: IMA <[email-masked]> ( valid under rfc2822/utf8headers),
> 
> If we are happy only with 1) and need not 2), 

[CORRECTION]  But, if we need to permit 2) in order to be consistent with 
  permitting 3) and 2), we may need to redefine "phrase" and its subcomponents.

---

From: John C Klensin <[email-masked]>
Date: Sun, 04 Feb 2007 09:48:17 -0500

> clarify the need for redefiniton of "phrase".

I obviously cannot speak for others, I would feel no pain over
requiring quotes if "phrase" contains any non-ASCII characters
(in addition to requiring them if it contains ASCII Specials).
Not only would that choice be better for compatibility with
legacy implementations, but, as the example above illustrates,
not requiring that the string be quoted would require that we
establish rules about things that look like quotes.  The latter
would take us down the rathole over which IDNA has had a
monopoly and we really don't want to go there.

That comment would apply whether the example in (3) was intended
to be
    c a f U+0065 U+2019     or
    c a f U+00E9
(note that 
    c a f U+0027
requires quotes today)

      john

---

From: Soobok Lee <[email-masked]>
Date: Mon, 5 Feb 2007 17:50:30 +0900

On Sun, Feb 04, 2007 at 09:48:17AM -0500, John C Klensin wrote:
> legacy implementations, 

What I observed and described in the previous thread was this:

 1) unquoted_display_name_in_utf8   <ascii@address> (invalid under rfc2822)

 will be treated/parsed as if it were 

 2) "unquoted_display_name_in_utf8" <ascii@address> (valid under utf8headers)

by most existing email address parsers (from mutt1.4 to sendmail 8.13.x)
and probably so by future EAI-capable email address parsers.

Even though  1) is invalid under RFC2822, it has been respected 
by implementors so far. I guess such convention is prevailing and
will be so even in the future.  I think we have had the pressure from
implementors to acknowledge 1) as de-facto (industry) standard and
incorporate it into RFCs.

So, I propose here :  

 How about lifting up the rfc2822' sanction against 1)
 and making 1) as legitimate one from [utf8headers]  ?

> but, as the example above illustrates,
> establish rules about things that look like quotes.  

Beside quotes-like characters, we have other problematic
character candidates in display_name like : 
  space-like,angle-bracket-like,@-like,:-like ones and many more. 

So, even if we leave current "requirement" for double quotes for 
non-ascii UTF8 display_name   intact in [utf8headers], 
that *** won't be sufficient *** for preventing such confusions.

Instead, we can permit unquoted forms explicitly and 
recommend MUA implementors that:
 unquoted non-ascii UTF8 display_name should be presented
 end users as quoted form by transforming 1) into 2)
 before displaying it, of course, with cautions about
 above-mentioned other candidates for confusions.

> That comment would apply whether the example in (3) was intended

Both!

> requires quotes today)

U+0027 is a single quote(') and is a "atom" element character
along with backquote(`) (see below). So no requirement for double quotes.
We need double quotation requirement for these ascii chars:  

         ,@()[]<>\;:" etc. 

So, "c a f :" should be double-quoted, while "c a f '" is not.

You may have numeric typo errors in "c a f U+0027" above.

<quote from rfc2822>
display-name    =       phrase

atext           =       ALPHA / DIGIT / ; Any character except controls,
                        "!" / "#" /     ;  SP, and specials.
                        "$" / "%" /     ;  Used for atoms
                        "&" / "'" /  <-----------------
                        "*" / "+" /
                        "-" / "/" /
                        "=" / "?" /
                        "^" / "_" /
                        "`" / "{" /
                        "|" / "}" /
                        "~"
</quote>

Even after adding "utf8-xtra-char" to the end of "atext"  (making utf8-atext), 
double quotations requirement above for(,@()[]<>\;:) is still intact.
When utf8 display_name contains only  extened Latin,Hiragana or Hangul characters ,
we should not require double quotes, for the same reason we do not require "" 
in  "To: IMA <[email-masked]>".

If display_name contains non-ascii UTF8-encoded (quote|comma|@|space)-like 
homograph characters, MUA should take cautions in displaying them before 
end users, whether is quoted or not. This recommendation is valid even for
local part of non-ascii email address.

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Mon, 5 Feb 2007 18:38:30 +0900

>                         "$" / "%" /     ;  Used for atoms

In other words, even after [utf8headers] permits unquoted utf8-atext string in
display_name,

 "cafe':" should be double-quoted, while "cafe'!" need not. ( e' == u+00e9)
  because  ":" is not permitted even in "utf8-atext" as well as in "atext".

Soobok

---

From: John C Klensin <[email-masked]>
Date: Mon, 05 Feb 2007 07:15:36 -0500

--On Monday, 05 February, 2007 17:50 +0900 Soobok Lee
<[email-masked]> wrote:

> by most existing email address parsers (from mutt1.4 to
> sendmail 8.13.x) and probably so by future EAI-capable email
> address parsers.
> 
> Even though  1) is invalid under RFC2822, it has been
> respected  by implementors so far. I guess such convention is
> prevailing and will be so even in the future.  I think we have
> had the pressure from implementors to acknowledge 1) as
> de-facto (industry) standard and incorporate it into RFCs.
> 
> So, I propose here :  
> 

(1) We cannot change RFC2822.  This is an operational and
implementation matter, not a procedural one.

(2) RFC2822 --and our rules-- are not about what happens in an
MUA, but about what goes out over the wire.  I believe that you
are making tests with what MUAs will accept and then confusing
the two.  Sending strings that RFC 2822 requires be quoted onto
the wire without quotes is much less common than you seem to
assume.  Sending those strings unquoted is certainly not a
"de-facto industry standard".

There is at least one pair of MUAs that are widely enough used
to claim "de-facto industry standard" status all by themselves
that, at least in most versions, quote _all_ display name
strings, whether they contain special characters or not, and
insist on always having a display name, even if, to do so, they
must transform
   To:   user@domain
into
   To: "user@domain" <user@domain>

and...

> So, even if we leave current "requirement" for double quotes
> for  non-ascii UTF8 display_name   intact in [utf8headers], 

That is why, despite the small inconvenience, I suggested just
requiring quotes for all such strings that contain non-ASCII
characters.  Although a parser would have no problems, trying to
figure out while characters look enough like others to be
problematic is an exercise that we can best avoid.   Of course,
an MUA might accept whatever it liked without quotes and quote
it before putting it on the wire, just as is the case today.

>  above-mentioned other candidates for confusions.

This is not a matter within the charter of this WG.

      john

p.s. Similar-looking characters are very rarely homographs.

---

From: Soobok Lee <[email-masked]>
Date: Tue, 6 Feb 2007 00:12:26 +0900

> "de-facto industry standard".

I accept.

>    To: "user@domain" <user@domain>

I see. If this WG has consensus that 'To: "display_name" <user@domain>' is 
the most desirable & normal form which we should recommend in headers,
we can "require" non-ascii display_name "MUST" be double-quoted always.

> problematic is an exercise that we can best avoid.  

I think requiring quotes does not cause any significant inconvenience.

> Of course,an MUA might accept whatever it liked without quotes and quote

How about putting this recommendation into [utf8headers] explicitly ?

> This is not a matter within the charter of this WG.

Okay.  

Soobok