Subject: [EAI] IDN domains and EAI

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 7 May 2014 12:59:24 +0200

Hi,

I've run into two problems that need settling. I'll raise them in order. 
The first one: Is this a legal SMTP command?

   mail from:<[email-masked]>

I see nothing to forbid it, but I tend to think that it ought to be 
forbidden. Allowing xn-- forms gives us no added value AFAICT and 
complicates life for the poor downtrodden implementers among us.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 May 2014 07:07:45 -0400

--On Wednesday, 07 May, 2014 12:59 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> Hi,
> 
> I've run into two problems that need settling. I'll raise them
> in order. The first one: Is this a legal SMTP command?
> 
> I see nothing to forbid it, but I tend to think that it ought
> to be forbidden. Allowing xn-- forms gives us no added value
> AFAICT and complicates life for the poor downtrodden
> implementers among us.

We discussed it at length.   While I suppose we could have said
"bad idea" more strongly, the interoperability implications of
forbidding an address that is valid for RFC 5321 SMTP would be
pretty serious.  Remember too that having things like that be
valid was in important design consideration for IDNA.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 7 May 2014 14:18:47 +0200

You discussed it at length before I started paying attention. OK.

Well, nothing says [email-masked] has to be considered equal to 
[email-masked], so I'll just ignore that, which I think will make them 
different in practice.

Arnt

---

From: Joseph Yee <[email-masked]>
Date: Wed, 7 May 2014 10:00:57 -0400

Short version:
The SMTP command is legal, but not encouraged (strongly). Especially this
is Mail From.

Long version:
This is more of questions.  Why in A-label? Are you just trying to
determine whether to fail the transmission at that point? Or thinking of
utilize it for certain situations?

Best,
Joseph

P.S. And your second question? ;)

On Wed, May 7, 2014 at 8:18 AM, Arnt Gulbrandsen
<[email-masked]>wrote:

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

---

From: [email-masked]
Date: Wed, 07 May 2014 07:04:05 -0700 (PDT)

> complicates life for the poor downtrodden implementers among us.

The implementation I'm working on supports it, and treats it as
equivalent to the corresponding 8bit.

				Ned

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 10:18:52 -0400

On Wed, May 07, 2014 at 02:18:47PM +0200, Arnt Gulbrandsen wrote:

> [email-masked],

"Equal to" in what sense?  Assuming that the xn--blah and błäh are
corresponding A-label and U-label, then they have to be "considered
equal" in some sense, since exactly one of those is going to be in the
DNS.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 7 May 2014 16:23:15 +0200

> equivalent to the corresponding 8bit.

Could you elaborate a bit on why you chose that precise behaviour?

Arnt

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 May 2014 10:24:28 -0400

--On Wednesday, 07 May, 2014 14:18 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> Well, nothing says [email-masked] has to be considered
> equal to [email-masked], so I'll just ignore that, which I
> think will make them different in practice.

Indeed, I would expect some problems if people try to compare
full addresses for equality.   Just as 
   [email-masked]   and 
   [email-masked]
are considered to be different even if the associated DNS
entries are
   example.net. 0 MX smtp.example.net.
   example.com. 0 MX smtp.example.com.
I'd assume that, depending on how and where they test,
implementations might construe 
   [email-masked]
   [email-masked]

as different, depending on how and where they test.   That would
violate the comparison rules of IDNA, but is tricky if one
thought one were doing an SMTP comparison rather than an IDNA
comparison.  In particular, many SMTP MTA implementations
require local configuration of all of the domain names for which
the server will accept mail.  Depending on how such a server is
upgraded to EAI, and where in its code that information is used,
one might want (or need) to configure both the A-label and
U-label forms of the domain name.

JianKang, please make a note that a comment about this should be
included when 6531 is updated unless people are convinced that
note should be in 6530, in which case Yangwoo and I should make
that note.

I've noted this on the Wiki page at
http://tools.ietf.org/wg/eai/trac/wiki That page is probably a
good place to leave notes to remind ourselves about issues to be
considered for future revisions, but please do not expect me to
maintain it.

All: please do not introduce new materials or discussion into
that Wiki page without raising them on the mailing list and,
when you do so, identify the source (as I have done in the new
entry).   If nothing else, it should increase the odds that
someone will pay attention to your point or question.

thanks,
    john

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 May 2014 10:34:25 -0400

--On Wednesday, 07 May, 2014 10:00 -0400 Joseph Yee
<[email-masked]> wrote:

>...
> Long version:
> This is more of questions.  Why in A-label? Are you just
> trying to determine whether to fail the transmission at that
> point? Or thinking of utilize it for certain situations?

FWIW, until EAI is really widely deployed, if I'm sending to,
or, more important, from, a host that has a non-ASCII domain
name but the relevant local-part is entirely ASCII, the
conservative course of action is to use A-labels and then not
request EAI services at all in the MAIL or RCTP commands,
thereby avoiding the risk of rejection or final delivery
problems because of EAI non-support.  From there, it is a tiny
step to A-label form addresses creeping into the EAI
environment, especially if messages are resent.  That may
provide at least one reason why these things may show up even
though we (and the "keep in native character form and convert to
A-labels only at the DNS interface" argument essentially made in
RFC 6055) would prefer that they not.

    john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 7 May 2014 16:38:26 +0200

I wrote:
> Well, nothing says [email-masked] has to be considered equal 
> to [email-masked], so I'll just ignore that, which I think will 
> make them different in practice.

I should perhaps elaborate on why I chose that.

1. xn-- is little-used today, so the argument to support it is weak.

2. I fear that if "mail from:<[email-masked]>" is legal, then it's 
reasonable to assume that "mail from:<[email-masked]> smtputf8" is 
legal too, and then "From: Asdf Jkl <[email-masked]>" is going to 
happen, and all of a sudden MUAs and mail body parsers need to deal with 
IDNA. Not very frequently. Similar to the infrequently used 
Content-Disposition/filename parsing hackery many of them contain. But I'd 
prefer to block that somewhere.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 May 2014 10:59:59 -0400

> 1. xn-- is little-used today, so the argument to support it is

I don't know if that is true.  Someone from a place where IDNs
are in heavy use to support scripts with little in common with
Latin may want to comment.  There is no question that making end
users look at the things is a bad idea if there are any
alternatives (again, especially for non-Latin strings).

> 2. I fear that if "mail from:<[email-masked]>" is legal,
> then it's reasonable to assume that "mail
> from:<[email-masked]> smtputf8" is legal too, and then
> "From: Asdf Jkl <[email-masked]>" is going to happen, and
> all of a sudden MUAs and mail body parsers need to deal with
> Content-Disposition/filename parsing hackery many of them
> contain. But I'd prefer to block that somewhere.

Personal opinion: What Ned is doing is probably right, certainly
for transport and probably for EAI-aware local handling.  It
would be rational for a final delivery server that "knows" it is
injecting the message into a fully-i18n-capable environment (or
that the message needs EAI anyway, e.g., for non-ASCII local
parts) to upgrade any A-label forms it sees to U-labels (an
advantage of the fully-reversible mappings in IDNA2008).    

However, if some mail body scanner wanted to include a rule that
said "only an idiot or an idiot-spammer would send around
A-labels, so the message should be treated as suspect", that
certainly would not violate any protocols.

    john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 7 May 2014 17:36:17 +0200

On Wednesday, May 7, 2014 4:59:59 PM CEST, John C Klensin wrote:
> for transport and probably for EAI-aware local handling.

Perhaps. But I fear the costs: Letting xn-- into bodies, letting phpmailer 
users get around a lack of EAI by using xn--, encouraging mail format 
upgrade by final delivery software.

Arnt

---

From: "Jiankang Yao" <[email-masked]>
Date: Thu, 8 May 2014 11:18:42 +0800

based on RFC6531, RFC5321 and RFC5890, "mail from:<[email-masked]>" should be legal.

another question is that
  If client smtp send the command "mail from:<[email-masked]>", is it possible that server smtp regard it and use it as "mail from:<[email-masked]>"?

if that is ok, the implementors can convert the a-label into u-label instead of forbidding it.

Jiankang Yao

From: Arnt Gulbrandsen
Date: 2014-05-07 18:59
To: ima
Subject: [EAI] IDN domains and EAI
Hi,

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 9 May 2014 09:25:16 +0200

I'm sorry I missed the question.

On Wednesday, May 7, 2014 4:00:57 PM CEST, Joseph Yee wrote:
> This is more of questions.  Why in A-label? Are you just trying 
> to determine whether to fail the transmission at that point? Or 
> thinking of utilize it for certain situations?

I'm adding EAI support to an MTA, methodically working through the code, 
and at one point in the code the question arose. I don't have any 
particular desire, so I chose to do what was easiest (nothing else seems to 
justify added effort).

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 9 May 2014 09:30:40 +0200

On Wednesday, May 7, 2014 12:59:24 PM CEST, Arnt Gulbrandsen wrote:
> in order.

The second (of three, now): It would be good to have something like 
example.com. I found one piece of documentation that used example.com as 
example domain, but I'm not sure whether "example" was meant to be an 
a-label or u-label.

I don't see anything reserved. Has any of you made a suitable domain?

Arnt

---

From: John C Klensin <[email-masked]>
Date: Fri, 09 May 2014 06:30:42 -0400

One added observation for those who are inclined to make changes
to domains from A-label form to U-label form after initial
submission.  Remember that a non-EAI-aware implementation is
almost certain to use A-labels if it can handle a non-ASCII
address at all (or the user is likely to slip non-ASCII domains
past it by using A-labels).  My recollection is that, when we
tried to get the SPF, DKIM, and DMARC folks to pay attention to
EAI addresses and IDNs, they weren't very interested.

So, with the understanding that I haven't researched this and
have no idea what the answers are, I suggest that anyone
contemplating changing the form of an address carefully review
those specs, figure out if it would invalidate the signatures,
and, if so, whether that is significant.

After the DMARC mailing list debacle, it would be good to not be
surprised.

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 09 May 2014 07:07:08 -0400

p.s. If anyone does the SPF/ DKIM/ DMARC analysis, please post
here -- I imagine several of us are interested.

  john

---

From: Franck Martin <[email-masked]>
Date: Fri, 9 May 2014 12:15:50 +0000

> EAI addresses and IDNs, they weren't very interested.
My recollection is slightly different than yours. 

I got a spec changed in last call to avoid the group syntax in From: header becomes a new weak standard.

---

From: John C Klensin <[email-masked]>
Date: Fri, 09 May 2014 09:49:34 -0400

(application ADs copied and subject changed --should have done
the latter sooner)

--On Friday, 09 May, 2014 12:15 +0000 Franck Martin
<[email-masked]> wrote:

>> One added observation for those who are inclined to make
>> changes to domains from A-label form to U-label form after
>> initial submission.  Remember that a non-EAI-aware
>> implementation is almost certain to use A-labels if it can
>> handle a non-ASCII address at all (or the user is likely to
>> slip non-ASCII domains past it by using A-labels).  My
>> recollection is that, when we tried to get the SPF, DKIM, and
>> DMARC folks to pay attention to EAI addresses and IDNs, they
>> weren't very interested.
>> 
> I got a spec changed in last call to avoid the group syntax in
> From: header becomes a new weak standard.

Franck,

What I meant was that we informally approached the leadership of
the relevant WGs about making sure that the EAI extensions,
including non-ASCII headers and the like, were fully supported
by SPF and DKIM (and, by extension although it is not IETF work,
DMARC).  We did not get any useful response.  IIR, in at least
one case, we were told that it would be future work, if ever.

Changes affected in Last Call, rather than as an earlier part of
WG process and design, are completely consistent with that
history.

More important in this context, I don't understand what "to
avoid the group syntax in From: header becomes a new weak
standard" means in practical terms and to internationalization
of email addresses and headers.  I also don't know what a "weak
standard" is -- as far as I know, the IETF only has two
categories of standards and neither of them is called "weak".

So which spec did you get changed, to what, why, and what are
the practical implications to EAI and/or SPF/DKIM deployment?

FWIW, I've been asked in several contexts lately (and Joseph has
too) why the EAI work is not deploying more quickly and what can
be done to remove barriers and encourage deployment.  It would
be truly unfortunate to have to add to those reports a statement
that the IETF has adopted standards for message authorization
and spam mitigation and that parties deploying email now have a
choice that makes the price of internationalization fewer
weapons against spam.

best,
   john

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 9 May 2014 15:58:45 +0200 (CEST)

> choice that makes the price of internationalization fewer

... it was time ago when the IETF "adopted" (or shall we say "nearly 
adopted", or "decided to publish anyhow to avoid more damage") standards 
like SPF and DKIM... and at that time we did not consider EAI in the 
already complex (and not fully satisfactory) scenarios that SFP and DKIM 
were designing. If we also considered at that time EAI implications, we 
should have had even more reasons not (I said NOT) to go with SPF and 
DKIM... too late, the dogs are out :-(

Maybe it is time to consider an informational report about the status of 
SPF and DKIM (and similar) techniques deplyoment, and their pro/cons in 
the global messaging scenario... and to take after that some decisions, 
too.

a bit "off topic", but John gave the occasion to mention this.

all the best!

>   john

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Franck Martin <[email-masked]>
Date: Fri, 9 May 2014 15:31:15 +0000

I'm talking about RFC6854.

It seems the industry is moving towards domain reputation, so any document that removes the need for a domain identifier is, I think, a step in the wrong direction.

On the from header, it is about one of the rare element exposed to the end user. While you cannot teach all users to recognize phish and be successful, it is not a reason to remove the identifiers for the users that can.

I'm sometimes surprised very few people check that the domain in the From exists... We have started to rely so much on fuzzy logic, rather than hard test, to block spam/malware.

The applicability statement in RFC6854 limit the scope, to say, if there is nothing else you can do, it may be ok while we transition.

For SPF/DKIM/DMARC they rely on the domain name, so conversion from UTF8 to puny code and back is possible, so it is just a question of implementation, more than standard. The local part is irrelevant for these protocols. Am I missing something here?

If an ASCII implementation see a domain name in UTF8, it may not be able to convert it to puny code, for lack of library implementation, and may decide to reject the message as a whole. This is a risk to face. The fix will depend on where is the pain. It may not be pleasant. We are not used to run the latest mail software as we are for web software.

I suspect there will be no issue when email is not forwarded, when it is forwarded then there will be breakage that will need to be resolved.

>From my poor understanding of all these RFC, this resolution may be left to the end user, sometimes with bounce messages which UI developers think they are not easy to the recipient of such message (we should do a better experience here, few ideas but for later).

Overall, when I ask for EAI support, nobody is able to figure out how it will work exactly for the end users. So everyone I speak to is waiting for an implementation.

I think you may need to do a case scenario document with screenshots, and may be an interop event. ;)

Printed on recycled paper!

> On May 9, 2014, at 9:49, "John C Klensin" <[email-masked]> wrote:
>

---

From: Barry Leiba <[email-masked]>
Date: Fri, 9 May 2014 12:10:15 -0400

> like SPF and DKIM... and at that time we did not consider EAI in the already
> complex (and not fully satisfactory) scenarios that SFP and DKIM were
> designing. If we also considered at that time EAI implications, we should
> have had even more reasons not (I said NOT) to go with SPF and DKIM... too
> late, the dogs are out :-(

This is simply not true with DKIM.  The DKIM working group *did*
discuss EAI, and decided two things:

1. We didn't see issues that caused EAI to interfere with DKIM, nor
the other way 'round.

2. We decided that we were OK, and that the EAI WG should have a
second look and document issues there, as their documents were
significantly behind the DKIM ones.

The EAI group lost a lot of its energy and probably didn't really look
at DKIM, but I think that's OK: I remain convinced that EAI and DKIM
do not affect each other (apart from the documented signature breakage
if things are "downgraded").

I don't think SPF has any relation to EAI either.

I'm having a lot of trouble understanding the point in Franck's
message, but one thing I get out of it is that Franck thinks that
something about the RFC822 From header affects SPF and DKIM.  It does
not.  There is nothing in either of those specs that has anything to
do with the From header other than its being mandatory that "From" be
included in the "h=" tag in DKIM.

It's true that DMARC makes use of the From header, and that there
*are* effects for DMARC.  That's OK: DMARC is coming along later, and
it's the responsibility of the DMARC spec to both address and document
these issues.

Barry

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 9 May 2014 18:17:03 +0200 (CEST)

> I don't think SPF has any relation to EAI either.

I mean I do not recall we explicitly discussed EIA possible issues when 
deciding what to do with DKIM and SPF... we discussed other things, like 
users mobility, etc.

> included in the "h=" tag in DKIM.

apart the downgrading case... no.

---

From: SM <[email-masked]>
Date: Fri, 09 May 2014 09:34:43 -0700

Hi Claudio,

{removed Apps ADs from Cc]

At 06:58 09-05-2014, Claudio Allocchio wrote:
>adopted", or "decided to publish anyhow to avoid more damage") 
>standards like SPF and DKIM... and at that time we did not consider 
>EAI in the already complex (and not fully satisfactory) scenarios 
>that SFP and DKIM were designing. If we also considered at that time 
>EAI implications, we should have had even more reasons not (I said 
>NOT) to go with SPF and DKIM... too late, the dogs are out :-(

There was some discussion about EAI in the SPFBIS working group (see 
threads at 
https://www.ietf.org/mail-archive/web/spfbis/current/msg02231.html 
and 
http://www.ietf.org/mail-archive/web/spfbis/current/msg01794.html). 
Things could have been better if people with an interest in EAI 
raised issues and discussed about those issues [1][2].

Regards,
-sm

1. If I recall correct John Klensin raised some issues.
2. It's difficult to be the lonely participant arguing against a 
reluctant working group.

---

From: [email-masked]
Date: Fri, 09 May 2014 14:55:12 -0700 (PDT)

> 1. We didn't see issues that caused EAI to interfere with DKIM, nor

Really? The ABNF in RFC 6376 is limited to 7bit, which makes accomodation of
utf-8 domains in the d= field (and possibly other places) rather difficult. The
only mention of accomodating utf-8 domains is this:

      INFORMATIVE IMPLEMENTATION NOTE: Although the "plain text" defined
      below (as "tag-value") only includes 7-bit characters, an
      implementation that wished to anticipate future standards would be
      advised not to preclude the use of UTF-8-encoded ([RFC3629]) text
      in tag=value lists.

That's pretty weak tea IMO.

And it's also insufficient. SPF, DKIM, and DMARC all depend on looking up
various domains in the DNS. Since the DNS is limited to a-labels, there has to
be a translation step - that domain in the d= field has to be converted from a
U-label to an A-label. If there's a mention of that anywhere in the RFC I'm not
seeing it.

Now, I suppose you could say that all this is obvious and doesn't need to be
spelled out. But not so fast: You could also design it where the A-label form
has to appear in d= and any translation (from, say, the From: header field a la
DMARC) has to happen before that.

Those two approaches are clearly not compatible, and I can make a case for
either of them being used.

> I don't think SPF has any relation to EAI either.

See above. Additionally, forwarders needing to deal with SPF are compelled to
use SRS or something similar. An EAI-encoded message with UTF-8 in the MAIL
FROM has to be accomodated, and note that SRS may cause DSNs to travel via an
alternate path which needs to support EAI for things to work.

But AFAICT we have no way to talk about any of this, having no-bid any document
describing the SRS solution space.

> it's the responsibility of the DMARC spec to both address and document

I confess I don't really get the point of Franck's message either, but I think
it's a symptom of a larger problem: An overall erosion of concern for continued
interoperability of the myriad components that make up the present-day email
service.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Fri, 9 May 2014 23:28:48 -0400

> U-label to an A-label. If there's a mention of that anywhere in the RFC I'm not

RFC 6376, Section 3.5:

   d= The SDID claiming responsibility for an introduction of a message
      into the mail stream (plain-text; REQUIRED).  Hence, the SDID
      value is used to form the query for the public key.  The SDID MUST
      correspond to a valid DNS name under which the DKIM key record is
      published.  The conventions and semantics used by a Signer to
      create and use a specific SDID are outside the scope of this
      specification, as is any use of those conventions and semantics.
      When presented with a signature that does not meet these
      requirements, Verifiers MUST consider the signature invalid.

      Internationalized domain names MUST be encoded as A-labels, as
      described in Section 2.3 of [RFC5890].

That last sentence is the operative bit.

Barry

---

From: Franck Martin <[email-masked]>
Date: Sat, 10 May 2014 18:26:13 +0000

I have 2 points, for clarification, ;)

1) this has nothing to do with SPF, DKIM, DMARC. RFC6854 states this is fine to receive an email with an anonymous author.
From: “anonymous”;
or worst:
From: “Paypal security”;

With no email address or domain, to verify who is really the author (yes it can be forged, but also it can be DKIM signed)
In other words, it does not matter who is the author of the email. 
I don’t think we want to go this path, just for security reasons. 
The applicability statement, limits the use of this syntax, I would have preferred this RFC does not even exists. This is weakening the security of email not increasing it.

2) as a few pointed here, for SPF, DKIM, and even DMARC. The domain part of an email address can easily be converted to A-label, and this is what the MTA has to do before passing it to SPF or DKIM, or DMARC. I think these protocols will be constrained to A-labels and this is fine.

---

From: Franck Martin <[email-masked]>
Date: Sat, 10 May 2014 18:26:14 +0000

---

From: [email-masked]
Date: Sat, 10 May 2014 11:44:38 -0700 (PDT)

> That last sentence is the operative bit.

Missed that, sorry. That eliminates the ambiguity at least.

But there's still a potential issue; it's just pushed off to other
specifications. Requiring an A-label in the d= field means anything comparing a
domain name against the field has to convert to A-label form. And that has
to be specified each and every time.

The DMARC specification actually discusses this and says when and how to do it,
using the term "conversion". I think it's OK in this regard.

However, while RFC 7208 (SPF) mentions this, the language used is ambiguous. It
says:

4.3.  Initial Processing

   If the <domain> is malformed (e.g., label longer than 63 characters,
   zero-length label not at the end, etc.) or is not a multi-label
   domain name, or if the DNS lookup returns "Name Error" (RCODE 3, also
   known as "NXDOMAIN" [RFC2308]), check_host() immediately returns the
   result "none".  DNS RCODEs are defined in [RFC1035].  Properly formed
   domains are fully qualified domains as defined in [RFC1983].  That
   is, in the DNS they are implicitly qualified relative to the root
   (see Section 3.1 of [RFC1034]).  Internationalized domain names MUST
   be encoded as A-labels, as described in Section 2.3 of [RFC5890].

Note the last sentence. Does that mean "convert any U-labels to A-labels" or
does it mean "reject anything that isn't already encoded in A-label form".

Normally I'd say the former, but given that this appears in a paragraph talking
about stuff that's not allowed. It's also the language other specifications use
for restrictions; different language is used when calling for a conversion.

The good news is this can be handed as an erratum.

				Ned