Subject: [I18ndir] HTML, email addresses, etc

From: John C Klensin <[email-masked]>
Date: Sun, 07 Jun 2020 23:46:31 -0400

Hi.

The following comments by John Levine and myself might be
relevant to people on this list.  

https://github.com/whatwg/html/issues/4562#issuecomment-640070477

https://github.com/whatwg/html/issues/4562#issuecomment-640290584

I don't have an opinion (yet) about whether any IETF action is
needed or appropriate.   Two things occur to me as possible:

(1) We could try to clarify the SMTPUTF8 specs by noting the
difficulties associated with trying to determine whether a
purported email address is valid or not, especially in the light
of fairly clear language in RFC 5821 that only the system (or
set of systems) actually hosting that address can know and the
greater diversity of troubles people can get themselves into
with a nearly-unlimited range of Unicode code points rather than
just ASCII.

(2) We could actually modify those specs to prohibit, or
strongly recommend against, allocating email addresses whose
local-parts contain anything but Unicode code points associated
with graphics, or even letters, digits, and a small selection of
symbols (possibly bringing the local-part recommendations close
to a PRECIS profile).   Or perhaps a different set of
restrictions. Might prevent a certain amount of mischief
associated with excess cleverness or strange (to me) ideas about
freedom of expression although I not sure it would have any
significant value against deliberate attacks.

Or, of course, we could say "someone else's problem" and just
move on.

Much, much, too late now, but I've started to wonder if we could
not have made SMTPUTF8 much more useful in practice if the SMTP
server was able to advertise the scripts that it supported.
Haven't thought through how to make that work, but giving the
client a clue about what would be considered "complete nonsense"
rather than just "invalid address" might have been helpful.

I have not posted this to the EAI ([email-masked]) list because I
wanted to see if people here felt that it was worth doing
something before I opened the discussion up.

thanks
   john

---

From: "Marc Blanchet" <[email-masked]>
Date: Mon, 08 Jun 2020 07:51:43 -0400

On 7 Jun 2020, at 23:46, John C Klensin wrote:

> to a PRECIS profile).

this is what I was going to say. IMHO, any protocol identifier (such as 
EAI) using something more than ASCII must define a PRECIS profile. This 
is the only chance to get it working. plain UTF8 for an identifier is 
just plain wrong.

Regards, Marc.

---

From: "John Levine" <[email-masked]>
Date: 8 Jun 2020 10:54:52 -0400

In article <[email-masked]> you write:
>is the only chance to get it working. Plain UTF8 for an identifier is 
>just plain wrong.

That is true but I'm wondering how many profiles we need. As jck
pointed out elsewhere, an address with a Tamil mailbox and a Chinese
domain name is unlikely to work.

---

From: "Marc Blanchet" <[email-masked]>
Date: Mon, 08 Jun 2020 11:05:11 -0400

> domain name is unlikely to work.

one profile. profile will get rid of all irrelevant codepoints, specify 
the normalization form, etc… Profile won’t go into specific scripts.

Marc.

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jun 2020 01:58:58 -0400

--On Monday, June 8, 2020 11:05 -0400 Marc Blanchet
<[email-masked]> wrote:

>>> is the only chance to get it working. Plain UTF8 for an
>>> identifier is just plain wrong.
>> 
>> That is true but I'm wondering how many profiles we need.  As
>> jck pointed out elsewhere, an address with a Tamil mailbox
>> and a Chinese domain name is unlikely to work.
> 
> one profile. profile will get rid of all irrelevant
> codepoints, specify the normalization form, etc… Profile
> won't go into specific scripts.

Combining a comment on your earlier note to this one:

One difficulty here is that, absent the ability to erase the
differences among languages and scripts and --at least-- making
sure everyone can conveniently read and enter every code point
and sequence of code points, we have only a choice between
learning as we go along and adjusting as needed versus not
learning and repeating old mistakes.  IDNA2003 (and UTR#46) were
designed, more or less, around the assumptions that most Unicode
graphics should be allowed, that normative tables were a good
idea, and that case folding, NFKC, and ignoring code points that
were "ignorable" would protect us from any of the associated
issues without causing serious problems.    As described in RFC
4690 (which would have been longer if we knew then what we know
now, particularly about issues normalization does not address),
that didn't work out very well: names that people legitimately
wanted to use were excluded, there was a great deal of potential
for confusion of a type that would impact security, the
maintenance requirements were too high, etc..  So we went back
and did IDNA 2008 and, in particular, tried to adapt the LDH
rule of traditional host names to Unicode (as well as addressing
some other issues).  That adaptation make it, nearly intact,
into PRECIS.

It might be worth remembering that before any of the work that
started with Stringprep and IDNA started, rather strong
arguments were made that domain names were protocol identifiers,
that they should stay in a limited subset of ASCII, and that
non-ASCII naming should be dealt with in other ways, probably at
another layer, and mapped as needed into the DNS.  Part of that
point of view was that non-ASCII labels in the DNS would just
lead to endless trouble about character equivalences, visual
confusion, difficulties caused by different principles (not just
different code points) among writing systems, a need for types
of aliasing the DNS model doesn't support, difficulties with the
DNS "feature" that octets with the high bit off were ASCII and
could exploit ASCII characteristics while other octets were just
octets, and so on.  I wouldn't go so far as to claim that those
taking that position have been proven right but it has now been
more than 17 years since the Stringprep and IDNA2003 specs were
published and at least 20 since the demands for IDNs got loud
and, lo, we are [still] dealing with those predicted issues.  To
some extent, we did IDNA at least as much because it was clear
that the alternative would have been many registries and
implementations making up and deploying their own, incompatible
and non-interoperable, ideas about how to have IDNs rather than
out of conviction that IDNs were a good idea.  And it was a
brilliant (IMO) idea for avoiding tremendous transitional
disruptions at the application level.

Then we came to a demand for non-ASCII email mailbox names.  The
situation was a bit similar with some people arguing that they
would do nothing but cause trouble and fragment the mail system,
with personal name phrases being a more than adequate option,
especially in a world in which many popular MUAs didn't display
actual addresses unless coerced.  And, as with IDNs around 2001
and 2002, it was clear that, if the IETF didn't take action to
standardize something, we would end up with nasty
interoperability problems.  So what became the EAI WG was
created as a mix of people who really, really, wanted
internationalized addresses with people who had misgivings but
were concerned about the interoperability issues.

Drawing on experience with email local parts going back to RFCs
821 and 822 and the knowledge that there was a long history of
email being used to communicate with embedded devices (some
decades old), transmission of commands is subject lines,
per-message or per-recipient backward-pointing addresses, signed
local parts, and other strangenesses including addresses that
were deliberately extremely difficult to type, the WG
extrapolated from the 821 rules and allowed  virtually any
Unicode code point in the local part, noting that, unlike the
DNS situation, local parts were firmly under the control of the
delivery system's management and that there were rules in the
basic protocol specification prohibiting anything else from
trying to interpret addresses or transform them into other
forms. The discussion is possibly worth having again, but I'm
not convinced that the very permissive rule that had WG
consensus was either unwise or that it should be changed, so the
situation is not like the PRECIS one.  

However, RFC 5321 already notes that [ASCII] local parts are
case-sensitive but that any mail system that establishes
"mailbox" and "mAIlbOx" as pointing to separate mailboxes and
especially separate user accounts is looking for trouble.  If
and when we get around to revising 5321/5322 (and maybe sooner),
I'm going to argue that the quoting mechanisms for assorted
characters that cannot be used without quoting have never really
worked consistently well (often due to operating system
conventions and procedures not under the control of the mail
system), that people don't understand them, and that the same
"understand that, if you do this, you are looking for trouble"
warning might reasonably be applied to creation of such mailbox
names.   Suggesting (or even specifying) a PRECIS profile for
what is reasonably safe would seem reasonable; pushing to
require it, maybe not so much so.

But then we get back to John's comment and my problem example
and why PRECIS is useful, but not sufficient.  There are a
number of combinations -- of local parts and domains and within
local parts-- that I don't expect we will ever see in the wild
because those who operate servers would not dream of allowing
them except as a concession to people whose desire to be cute or
to express themselves go far enough to be a hazard to others.
It is reasonable to assume that, if such addresses show up on
the wire, they are associated with spam or other malicious or
destructive behavior.  And it is reasonable to advise server
operators that allowing those combinations is generally a bad
idea, both to provide support to them when someone tries to
object and to make the Internet a better and more reliable
place.   Generally, PRECIS (like IDNA) deals with code points
and not strings and rules like "no mixed scripts" are far too
blunt as instruments for the cases of interest.   Instead,
proper treatment of those cases actually does need to deal with
individual sets of scripts and maybe even languages (my earlier
notes said "writing system" rather than "script" for a reason.

That means that, for mailbox names, the idea of requiring a
PRECIS profile is both overly prescriptive and inadequate.  As a
source of advice: probably quite appropriate.

Just my opinion but one based on a few years of experience with
email and strong memories of the EAI WG discussions.

best,
    john

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 9 Jun 2020 17:57:51 +0900

> domain name is unlikely to work.

Yes, but an address with a Tamil mailbox and a Devanagari domain name 
might indeed work. Similarly for an address with a Mongolian mailbox and 
a Chinese domain name.

Regards,   Martin.

---

From: "John R Levine" <[email-masked]>
Date: 9 Jun 2020 09:19:12 -0400

> Yes, but an address with a Tamil mailbox and a Devanagari domain name might 
> indeed work. Similarly for an address with a Mongolian mailbox and a Chinese 
> domain name.

Indeed, that's the problem.  It's really hard to describe the combinations 
of scripts that make sense.  I think there are a lot of characters that 
are unlikely ever to appear in normal mailboxes, half sized math 
punctuation and the like, but beyond that it's a mess.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jun 2020 09:25:47 -0400

--On Tuesday, June 9, 2020 17:57 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>>> this is what I was going to say. IMHO, any protocol
>>> identifier (such as EAI) using something more than ASCII
>>> must define a PRECIS profile. This is the only chance to get
>>> it working. Plain UTF8 for an identifier is just plain wrong.
>> 
> Yes, but an address with a Tamil mailbox and a Devanagari
> domain name might indeed work. Similarly for an address with a
> Mongolian mailbox and a Chinese domain name.

Absolutely.   And those are good examples of why a simplistic
rule banning mixed scripts would not be helpful.

    john

---

From: Nico Williams <[email-masked]>
Date: Wed, 10 Jun 2020 16:18:36 -0500

> https://github.com/whatwg/html/issues/4562#issuecomment-640070477

I'm not sure that we need a new input type for EAI addresses.  If the
user types in non-ASCII, then presumably *they* know that the domain can
handle EAI.

Another way to put it is that if we did add a new input type, should
developers build websites where the user must pick whether to enter an
EAI or not?  Or should developers just always use the new input type?

(Answer to the second question: No! First the page must detect whether
the browser supports the new input type!)

If we add a new input type, then after a while that will be all that is
used.  Ergo it's not needed.

The mere fact that the user typed in an internationalized mailbox name
is all the evidence we need that it is likely that the domain supports
EAI.

You could say that because *your* system doesn't support EAI for
outbound email, you won't accept it as input.  _This_ could be a new
option on the input.

> with a nearly-unlimited range of Unicode code points rather than

Ulitmately, the true right way to validate a mailbox name is to ask its
MTA to validate it.

> significant value against deliberate attacks.

I feel like saying no to (2).  Requiring that mailbox names be printable
is OK, but I wouldn't forbid, say, emoji.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 10 Jun 2020 17:53:43 -0500

On Mon, Jun 08, 2020 at 07:51:43AM -0400, Marc Blanchet wrote:
> this is what I was going to say. IMHO, any protocol identifier (such as EAI)
> using something more than ASCII must define a PRECIS profile. This is the
> only chance to get it working. plain UTF8 for an identifier is just plain
> wrong.

I'm inclined to be as permissive as possible.

Ever play video games or used social media?  Have you noticed that many
people have usernames with -gasp- whitespace in them?  SASL wanted to
prohibit that -- a recipe for those applications not using SASL[*], a
recipe for failure.

Let's be permissive.  I'm OK with disallowing non-ASCII punctuation,
quotation marks, and misc things like non-breaking spaces (which should
be squeezed out) and so on, but I don't think we should try too hard to
make lists of banned characters.

Nico

[*] Well, OK, they don't and weren't going to use SASL, most likely.
    But the point stands.  These are policies that should be
    site-specific, not Internet-wide.

---

From: Nico Williams <[email-masked]>
Date: Wed, 10 Jun 2020 17:58:44 -0500

> rule banning mixed scripts would not be helpful.

As if people might not have names with mixed scripts.  What's next,
mixed ethnicities?!  Especially for for *input* fields, some here would
not let me have the address I _want_?  In this day and age?

There is no confusables issue here: these are addresses entered by the
user, not displayed to the user (and if you display any, chances are
it's the user's to the same user).

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Jun 2020 00:23:36 -0400

--On Wednesday, June 10, 2020 17:58 -0500 Nico Williams
<[email-masked]> wrote:

> As if people might not have names with mixed scripts.  What's
> next, mixed ethnicities?!  Especially for for *input* fields,
> some here would not let me have the address I _want_?  In this
> day and age?

And that is what I said -- that a rule banning mixed scripts
would not be helpful-- so I am not sure what you are responding
to with apparent vehemence. 

> There is no confusables issue here: these are addresses
> entered by the user, not displayed to the user (and if you
> display any, chances are it's the user's to the same user).

I think we almost agree.  Almost.  Let me try to break this down
a bit, focusing on where we might disagree.

(1) I think I still favor having a separate type, whether called
"i18email" or something else, rather than just saying "from now
on, 'email' means just about anything you can write in UTF-8
that has an '@' in the middle" or something close to it.  There
are two reasons, one is that I've seen, too many times,
situations in which a bad-end or intermediate system that is not
written well or robustly is handed UTF-8 (or anything much
different from ASCII) when ASCII is expected and basically
self-destructs, often leading to either (a) something completely
unpredictable and unintelligible (to the user) in the database
or passed on to other systems or (b) an error message with the
semantics and information content of "fail" or "you lose".
The advantage of using a different keyword, even if one believes
it is slight, is that it gives us a chance to specify what the
associated field might mean and to let the page designer take
explicit responsibility rather than having one definition last
week and another definition next week.

(2) As to whether we should give them a rule or regular
expression to associate with the new type that goes much beyond 
   bunch-of-octets-for-local-part@bunch-of-octets-for-domain-part
I rather doubt it.  Experience indicates that, if people try to
perform most sophisticated checks, they will end up with false
negatives and thereby block  things that the back-end systems or
delivery-end mail servers would accept.  I might even argue
against blocking foo@bar@example.!!!#$%^& because, as far as can
be known generally, it might make perfect sense to a local mail
system even if not on the Internet.  

However, if one is going to adopt that model, then the rule has
to be "don't mess with it".  What is accepted goes directly to
the bad-end or mail system: no compatibility equivalents (not
even wide and narrow width forms), no normalization at all, no
case-folding or lower-casing, no dropping of joiners,
non-joiners, or other code points that your note indicated were
unimportant.  Once one goes down that slippery slope, it is not
clear there is any rational stopping point.

(3) What advice to give people who are defining email addresses
for their mail servers (or deciding what to allow users to
create) is a different matter.  We know that trying to make case
distinctions and to either reject incoming messages where the
case isn't quite right or to map forms that differ only by case
causes problems.  Prohibiting that is very different from saying
"likely to case problems, especially confused and unhappy users,
so don't do it unless you need to, understand the implications,
and are willing to explain why" (a more elaborate version of
what 5321 says).  Saying "following a reasonable PRECIS profile
is probably a good idea and is likely to avoid surprises -- if
you need to do something else or allow an exception, be sure you
know what you are doing" seems reasonable to me.  Different from
the requirement I think Marc was suggesting, but still
reasonable guidance.    

The similarity between advice like either of those examples and
the "you need to know what you are doing but here are some
general hints about things to avoid unless you really, really,
know what you are doing" advice of IDNA2008 (and the
clarifications in 5891bis) should be clear to everyone reading
this.

My thought about what to do with a clarification or guidance to
the SMTPTUF8 rules for local parts was similar.  Strongly
suggest avoiding anything but letters, digits, and symbols or
punctuation already in active use 821/2821/5821 email (noting
that some of that punctuation would probably require some
qualifications to PRECIS rules).  Suggest that using a PRECIS
profile, where possible, is a much better idea than working out
per-protocol or per-mail-server rules and conventions.   And
stress that any of those code-point and rule-based mechanisms
are still a superset of what almost any sensible mail system
would allow, but the optimal subset for a given system is going
to need to depend on local needs and may require rules or
conventions about more than code points.

And, finally, whether emoji belong in local-parts or not is
something I'll happily encourage you to discuss on the PRECIS
list as long as you and others on that list remember at least
three things:  (i) the names by which Unicode calls them are not
necessarily the names by which they are called or read out loud,
especially in languages very different from English; (ii) the
graphemes associated with a given emoji code point are not
standardized and for some of them vary widely; (iii) combining
sequences are often problematic in identifiers [1].  Combining
sequences of emoji --either with emoji-specific qualifiers or
with ZWJ-- are not nearly as well structured and predictable as
ones with letters [2].  I'd probably think of more if it weren't
past midnight here.  So, if I were giving advice to a delivery
mail system administrator, it would be "don't unless you are
sure you know your users, their likely correspondents, and have
a plan about any support that might be necessary".   YMMD, of
course.

best,
   john

[1] This is, of course,  one reason IDNA imposes an NFC
requirement and, if I recall, PRECIS does too, at least for
profiles that might be relevant.

[2] As an exercise, think about the emoji sequences for
"vomiting cowboy", "vomited up a cowboy", and "vomited on the
cowboy"

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 11 Jun 2020 16:07:24 +0900

On 11/06/2020 13:23, John C Klensin wrote:

> I rather doubt it.

Please let's make that *at least*
   bunch-of-characters-for-local-part@bunch-of-characters-for-domain-part

Regards,  Martin.

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 11 Jun 2020 16:25:56 +0900

> user types in non-ASCII, then presumably *they* know that the domain can

They know that the receiving domain can handle EAI. But before the mail 
is received, it actually has to be sent. That happens from an MTA 
associated (in some way) with the Web page where the the user types in 
the address.

> EAI or not?

No. The distinction doesn't apply to a single Web site.
[Except for your point about browser differences and for cases where a 
Web site has different parts where mail gets routed differently. E.g. 
imagine a trading web site that sends mails to sellers via mail system A 
and mail to buyers via mail system B.]

But it applies to the Web as a whole. Most current web sites send their 
mail via an MTA that is not (yet) set up for EAI. And many users may 
have both an ASCII-only and an EAI email address. It helps a lot if 
users get told by the browser that the Web site they are on (more 
specifically, the associated outgoing MTA) can't handle their EAI address.

> Or should developers just always use the new input type?
> 

That's exactly what this option is about.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Jun 2020 06:45:00 -0400

--On Thursday, June 11, 2020 16:25 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>> I'm not sure that we need a new input type for EAI addresses.
>> If the user types in non-ASCII, then presumably *they* know
>> that the domain can handle EAI.
> 
> They know that the receiving domain can handle EAI. But before
> the mail is received, it actually has to be sent. That happens
> from an MTA associated (in some way) with the Web page where
> the the user types in the address.

It may be a fine distinction, but while they presumably know the
delivery server (receiving domain) can handle the address
(otherwise, they would not have that address), they cannot know
whether there is a path from what submission server is
presumably accessed from that web page to the delivery server.
Equally or more important, we need to remember that, because of
the use of email addresses as identifiers, there may be systems
that don't accept identifier strings that are perfectly
acceptable when actually used in email.   

We have a very common example of that which doesn't even involve
i18n issues.  Assume you have an email server or service that is
perfectly ok with [email-masked] (an address style that
is perfectly acceptable under RFC 5321/5322) and that allows
such addresses.  Whether it treats it as "same user, different
folder from [email-masked]" (the subaddress case), a
completely different mailbox (the "'+' is just another character
case), or discards the "+" and treats it internally as
"[email-masked]" (leads inevitably to different mailboxes)
is up to it -- no one else has any control.  But, when some web
page that is being used for data entry by the user decides that
"+" does not belong in local parts and hence, when the user
types [email-masked] says "invalid address" and will not
let the user proceed without providing an address that it likes,
it just does not make any difference what the delivery server
will accept.

>> Another way to put it is that if we did add a new input type,
>> should developers build websites where the user must pick
>> whether to enter an EAI or not?
> 
> [Except for your point about browser differences and for cases
> where a Web site has different parts where mail gets routed
> differently. E.g. imagine a trading web site that sends mails
> to sellers via mail system A and mail to buyers via mail
> system B.]
> 
> But it applies to the Web as a whole. Most current web sites
> send their mail via an MTA that is not (yet) set up for EAI.
> And many users may have both an ASCII-only and an EAI email
> address. It helps a lot if users get told by the browser that
> the Web site they are on (more specifically, the associated
> outgoing MTA) can't handle their EAI address.

Yes.

>> (Answer to the second question: No! First the page must
>> detect whether the browser supports the new input type!)
>> 
>> If we add a new input type, then after a while that will be
>> all that is used.  Ergo it's not needed.

Unfortunately, when whatever is there (the older type or
keyword) is widely deployed and well-entrenched, "a while" can
be rather long.  For examples, see the number of SMTP-senders
who are still using HELO or, if you prefer a more painful
example, IPv6.

>> The mere fact that the user typed in an internationalized
>> mailbox name is all the evidence we need that it is likely
>> that the domain supports EAI.
>> 
>> You could say that because *your* system doesn't support EAI
>> for outbound email, you won't accept it as input.  _This_
>> could be a new option on the input.
> 

Exactly.

And, to address your other comment:

>>     bunch-of-octets-for-local-part@bunch-of-octets-for-domain
>>     -part I rather doubt it.
> 

I assume that, in the above, "character" means "Unicode code
point in UTF-8".  Fine with me.  I used "octets" only because,
following a variation on what I think was the argument Nico was
making, I didn't want to impose a requirement for validation on
the browser (or other HTML interpreter).

best,
   john

---

From: Nico Williams <[email-masked]>
Date: Thu, 11 Jun 2020 07:59:27 -0500

> That's exactly what this option is about.

The default needs to be permissive then.

---

From: Nico Williams <[email-masked]>
Date: Thu, 11 Jun 2020 19:11:12 -0500

> The default needs to be permissive then.

Let me expand a bit.  The backend can do its own validation and must not
rely on the frontend.  Too-restrictive frontend default validation will
simply cause failure to support EAI in the common case because
developers will not even know they have to enable it.

Nico
--

---

From: Martin J. Dürst <[email-masked]>
Date: Sat, 13 Jun 2020 15:20:31 +0900

> point in UTF-8".

Yes, with the caveat that it might not actually be in UTF-8 because e.g. 
the browser internally uses UTF-16.

Regards,   Martin.

> Fine with me.  I used "octets" only because,

---

From: John C Klensin <[email-masked]>
Date: Sat, 13 Jun 2020 02:39:38 -0400

Yes.  Thanks.  I keep thinking about what goes over the wire,
but you are absolutely correct.

   john

--On Saturday, June 13, 2020 15:20 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>>>>      bunch-of-octets-for-local-part@bunch-of-octets-for-dom
>>>>      ain -part I rather doubt it.
>>> 
>>> bunch-of-characters-for-local-part@bunch-of-characters-for-d
>>> om ain-part
>> 
> Yes, with the caveat that it might not actually be in UTF-8
> because e.g. the browser internally uses UTF-16.
> 
> Regards,   Martin.
> 
>> following a variation on what I think was the argument Nico
>> was making, I didn't want to impose a requirement for
>> validation on the browser (or other HTML interpreter).
>> 
>> best,
>>     john