Subject: [ietf-smtp] How wrong is this EAI implementation

From: "John R Levine" <[email-masked]>
Date: 20 Jun 2020 14:54:17 -0400

I am poking at the widely used Roundcube web mail package.

One of my test addresses is [email-masked]

When I enter it into Roundcube as a username, it turns the domain into 
a-labels, [email-masked] and uses that as the 
address in the From: line, and to authenticate to IMAP and 
submission.  I can understand where that came from, but it still seems 
wrong.

Addresses that I put on the To: line seem unmolested.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Sat, 20 Jun 2020 17:03:29 -0400

John,

IIR (I have not gone back and reread the spec), conversion of
the domain name to A-labels is strongly discouraged at any point
prior to the MSA or MTA actually looking up the domain name so a
connection can be opened.  For message headers (including not
just the "From:" header field but trace fields, etc.), it is
even more strongly discouraged: the intent was "restricted UTF-8
all the way" and "making users look at A-labels is just looking
for trouble".  However, the intent with IDNA (and, I think, the
push for IRIs) was much the same, keeping internal forms and
users separated and the latter working with native-character
forms to the extent possible.  However, the specs recognized
that exceptional circumstances could arise and therefore stop
short of complete prohibitions.

I haven't thought it through but, somehow, converting domains in
backward-pointing addresses (what do they do to/with
"Reply-to:"?) seems more, rather than less, offensive than
forward-pointing ones.

So, again relying on memory, the behavior you describe is in bad
taste and almost certainly represents bad judgment, but may not
actually be non-conforming.  I'll leave the question of how far
on that spectrum an implementation needs to lie to be considered
"broken".

  john

--On Saturday, June 20, 2020 14:54 -0400 John R Levine
<[email-masked]> wrote:

> When I enter it into Roundcube as a username, it turns the
> domain into a-labels,
> [email-masked] and uses that as
> the address in the From: line, and to authenticate to IMAP and
> submission.  I can understand where that came from, but it
> still seems wrong.
>

---

From: "John R Levine" <[email-masked]>
Date: 20 Jun 2020 18:23:26 -0400

> "Reply-to:"?) seems more, rather than less, offensive than

They turn Reply-To into a-labels too.  Bah.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sat, 20 Jun 2020 19:33:12 -0400

--On Saturday, June 20, 2020 18:23 -0400 John R Levine
<[email-masked]> wrote:

>> I haven't thought it through but, somehow, converting domains
>> in backward-pointing addresses (what do they do to/with

At least consistent.

Like you, I can guess why they are doing this (especially given
consistency).  That doesn't make it less bad taste, more in the
spirit of the spec, or more use-friendly.  But, coming back to
your original question, I think this is permitted and questions
about "wrong" or "broken" are matters of personal opinion.

   john

---

From: Keith Moore <[email-masked]>
Date: Sat, 20 Jun 2020 20:11:49 -0400

On 6/20/20 7:33 PM, John C Klensin wrote:

> But, coming back to

About this behavior or EAI in general?

---

From: John C Klensin <[email-masked]>
Date: Sun, 21 Jun 2020 00:36:41 -0400

--On Saturday, 20 June, 2020 20:11 -0400 Keith Moore
<[email-masked]> wrote:

>> your original question, I think this is permitted and
>> questions about "wrong" or "broken" are matters of personal
>> opinion.
> 

Try reading my first response to John Levine today.

The SMTPUTF8 set of protocols, produced by the EAI WG, recommend
against some things that then allow because the WG knew it
couldn't predict every possible case.  In that regard, the WG's
thinking and conclusions are not much different from the
"operational necessity" language that was put into 5321.   In a
more perfect world, without odd circumstances and with a
competent collection of Protocol Police with wide-reaching
enforcement powers, we would not have done things that way.
There is also a case to be made that the whole idea of non-ASCII
addresses was a bad idea (and non-ASCII domain names probably
were too).  In both cases, the designers were quite aware of
those arguments (and some might even agree) but people were
going to do those things and the question was ultimately not "is
this a good idea" but "do you believe in an interoperable
Internet or not".

Not sure that answers your question.  Even if it does, YMMV.

   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 21 Jun 2020 06:52:52 +0000

I note that there is no explicit requirement to treat incoming a-labels 
as equivalent with u-labels anywhere in the EAI RFCs. Which means that 
you can convert labels before sending, but you cannot rely on the 
receiver to interpret the result as desired.

Bit too subtle, 
really.

Arnt

---

From: "John Levine" <[email-masked]>
Date: 21 Jun 2020 10:53:42 -0400

>receiver to interpret the result as desired.

I don't think that 5321 requires that [email-masked] and [email-masked]
be treated the same, either.

Beyond some point, you can't force people to be reasonable.

In the particular case I mentioned, my MTA is Courier which treats
A-labels and U-labels equivalently in mail addresses, but not in
identifiers for authentication.  It was easy enough to add both versions
to the identifer table, but I think I'm going to send a bug report to
roundcube.

---

From: Ned Freed <[email-masked]>
Date: Sun, 21 Jun 2020 10:11:48 -0700 (PDT)

> > receiver to interpret the result as desired.

The EAI documents repeatedly refer to there being two different "forms" for
domains, u-label and a-label. The only reasonable interpreration of such
statements is that the they are equivalent. RFC 6531 even goes so far as to
say that u-labels form should be used when the SMTPUTF8 extension is available,
a-label form when it isn't.

Additionally, the point that always seems to be elided in these discussions is
that MTAs do NOT have the luxury of treating addresses as opaque strings.
Elimination of duplicate addresses, alias lookup, and canonicalization  of
local address forms are all things that MTAs do routinely, and all of them
require at a minimum the ability to compare addresses and get correct results.
This combined with the "mutiple forms" notion in the documents leaves very
little wiggle room for implementers, explicit requirements or not.

And yes, it's a bit of a PITA to code.

> be treated the same, either.

Section 2.4 explicitly says domains are case-insensitive. Not a lot of wiggle
room here either.

> Beyond some point, you can't force people to be reasonable.

No, but I think you can make a case that any implementation that fails to treat
the forms as equialent is incompliant.

> to the identifer table, but I think I'm going to send a bug report to

Yet another case where comparisons are unavoidable.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 23 Jun 2020 15:09:25 +0200

> Beyond some point, you can't force people to be reasonable.

Agree entirely. And this is why I think the EAI specs are too subtle on 
this point.

I do have an opinion about what that point is, and what the documents ought 
to say. But my real issue is that it's possible to review source code 
against every MUST and SHOULD in 653x and miss that there's something to 
consider.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Tue, 23 Jun 2020 19:16:52 -0400

--On Tuesday, June 23, 2020 15:09 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

>> I don't think that 5321 requires that [email-masked] and
>> [email-masked] be treated the same, either.
>> 
> Agree entirely. And this is why I think the EAI specs are too
> subtle on this point.
> 
> I do have an opinion about what that point is, and what the
> documents ought to say. But my real issue is that it's
> possible to review source code against every MUST and SHOULD
> in 653x and miss that there's something to consider.

In my capacity as the former co-chair of the former WG a few
observations while speaking only for myself:

* Like so many other i18n efforts in the IETF in recent years,
the WG had mostly run out of energy by the time it got those
documents out.  I don't know if there would be energy and the
needed focus to start (and ultimately finish) clarifying
documents but other events of the last year or two leave me very
doubtful.

* There were several important contributors to EAI WG-produced
specs who remain on the [email-masked] list (although even some of
them have move on to other work professionally) but who are not
on this one.   Absent a move by the relevant ADs (presumably
after discussion on both lists) to consolidate them, if one
wants to have a serious discussion about SMTPUTF8 ("EAI")
issues, those issues should probably be addressed there (instead
or in addition).

* Just as I think that the "core" RFC 5321/5322 specs and
possibly some closely-related work are overdue for an
applicability statement that discusses what is common,
reasonable, and/or likely to work well in practice independent
of what is formally permitted, a similar document would almost
certainly be useful about assorted i18n extensions to email.
I'd think that, not only should 6530-6533 and 6855-6858 be
addressed (including whatever experience we have had that would
suggest that one or both of 6857 and 6858 should be revised or
deprecated), but that such an effort would want to make clear
statements about the relevance of PRECIS to email address
local-parts and to discuss and make recommendations about when
to use non-ASCII addresses and relevant headers and when to
stick to RFC 2231-style encoded words.  If anyone thinks it is
the right time to do the work and produce that document, and
that the IETF would have the energy and will to process it in a
timely way, I look forward to an I-D with great anticipation.

And, fwiw, I certainly agree that one could do a code review
against normative statements and miss that there are issues to
consider.

best,
   john