Subject: [ietf-smtp] How is EAI mail implemented ?

From: "John R Levine" <[email-masked]>
Date: 15 Jun 2021 13:32:34 -0400

This is sort of a followup to the discussion abut domain names in Received 
lines in EAI messages.

RFC 6531 describes a model where EAI mail is conceptually parallel to 
ASCII mail -- clients tag incoming EAI messages with SMTPUTF8 keywords, 
those EAI messages can only be relayed to servers that offer SMTPUTF8, and 
so forth.  Having looked at a bunch of EAI mail software, nobody 
actually implements it that way.

Since all computers handle 8-bit bytes, mail software generally handles 
8-bit data without doing anything special.  You have to change the code 
that does DNS lookups to turn U-labels into A-labels, but that's about it.

On my system I turn addresses into A-labels on the way in, and back into 
U-labels on the way out (only if there is a UTF-8 local part) and handle 
all the local routing and deliveries with A-labels.  But a lot of MTAs 
don't even do that, they just allow any IDN in the domain, and if you want 
the U-label and A-label versions of an address to deliver to the same 
place, you have to configure them both.

Nobody I've seen tags messages as EAI in their internal queues.  On 
outgoing mail, they check on the fly to see if it's an EAI message: 
non-ASCII characters in the envelope or message headers (Exim doesn't even 
look at the headers, and says that's not a bug.)  A lot of MTAs add the 
SMTPUTF8 MAIL FROM tag to all outgoing mail to servers that offer 
SMTPUTF8, because why not.  I think they all notice if an EAI message is 
sent to a non-EAI server, and a few in that case do odd things like 
turning a UTF-8 local part into a MIME encoded word in the envelope.

This approach is a lot easier to code than trying to tag all the queued 
messages, and it can deliver more mail if, e.g., an incoming message has 
an ASCII bounce address and UTF-8 recipients but is relayed to an ASCII 
recipient, the relay doesn't need EAI.

When looking at IMAP and POP servers, again, since computers all handle 
8-bit data, you get most of the way there for free.  I haven't found any 
IMAP servers with UTF8=ACCEPT or POP with UTF8 that really works, but I've 
found plenty with LOGIN and AUTHENTICATE commands that take UTF-8 
strings, and IMAP searches with the complex old character encoding work 
remarkably well.  It often seems even to find strings in unencoded UTF-8 
headers which I wasn't expecting, perhaps again something that works by 
mistake.

None of this means the RFCs have to change but it might be time for an 
applicability statement or something about how EAI is likely to coexist 
with ASCII mail for a long time.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Ned Freed <[email-masked]>
Date: Tue, 15 Jun 2021 10:43:49 -0700 (PDT)

> all the local routing and deliveries with A-labels.

It sounds to me like this doesn't allow for UTF-8 domains in aliases. If so,
that's not very friendly.

In any case, this isn't how we do it. We try very hard not to modify addresses,
and instead do various forms of "canonicalization" when looking them up or
comparing them. This is undoubtedly ugly, and to someone like me who also writes
code for AVR microcontrollers it seems pretty inefficient, but as a practical
matter these transformations are down in the noise for modern CPUs.

> But a lot of MTAs

In our case configuring either one is sufficient. Of course this leaves open the
question of comparing local parts. We haven't really solved this one yet.

> Nobody I've seen tags messages as EAI in their internal queues. 

We do. In fact our EAI tagging tells us what aspects of EAI are
in use: MAIL FROM address, RCPT TO address, main headers, body.
It turns out to be handy to know this up front.

> On
> look at the headers, and says that's not a bug.)

How strictly we follow the RFCs is settable in our case. If someone
wants strict behavior they can have it, if someone is content
to send messages with SMTPUTF8 headers to a non-EAI server they can
have that too.

> A lot of MTAs add the
> SMTPUTF8, because why not.

Because it's possible that regardless of what you have observed, that server may
then refuse to relay an EAI-tageed but actually non-EAI message to a non-EAI
server?

> I think they all notice if an EAI message is

Yuck. Frankly, it would be better to just send UTF-8 in this case than to come
up with a private encoding for an address that likely belongs to the server
you're sending to.

> recipient, the relay doesn't need EAI.

IME the tagging part was trivial. The canonicalization was considerably harder.
(And writing the tests for all of it was a real PITA.) However, the really
difficult part is balancing standards compliance and all this turning into a
major support call generator.

> headers which I wasn't expecting, perhaps again something that works by

I don't think it's a mistake, exactly. More like the optimum handling
for invalid messages turns out to align with the proper handling
for SMTPUTF8 messages.

> with ASCII mail for a long time.

I don't have enough feedback from actual use to be able to say anything
definitive, but my guess is the document EAI needs is something that the IETF
would not be willing/able to write: One that says what parts of the standard
should be followed, what parts should be outright violated, and when.

				Ned

---

From: Claus Assmann <[email-masked]>
Date: Tue, 15 Jun 2021 20:26:26 +0200

> Nobody I've seen tags messages as EAI in their internal queues.

sendmail does:
writing: queueup()
	/* send various flag bits through */
...
	if (e->e_smtputf8)
		*p++ = 'e';

reading: readqf()
...
	  case 'e':	/* message requires EAI */
		e->e_smtputf8 = true;

-- 
Address is valid for this mailing list only, please do not reply
to it direcly, but to the list.

---

From: Jeremy Harris <[email-masked]>
Date: Tue, 15 Jun 2021 19:36:50 +0100

> 	  case 'e':	/* message requires EAI */

and Exim does:

writing: spool_write_header()
...
  if (message_smtputf8)
   {
   fprintf(fp, "-smtputf8\n");

reading: spool_read_header()
...
     else if (Ustrncmp(p, "mtputf8", 7) == 0)
       message_smtputf8 = TRUE;

-- 
Cheers,
   Jeremy

---

From: "John R Levine" <[email-masked]>
Date: 15 Jun 2021 14:59:16 -0400

> that's not very friendly.

My system is qmail underneath and local domains are all mapped to 
sub-addresses before delivery.  If you put an address in a .qmail file, 
the envelope is normalized so U-labels work.

>> But a lot of MTAs don't even do that, they just allow any IDN in the 
>> domain, and if you want the U-label and A-label versions of an address 
>> to deliver to the same place, you have to configure them both.

> In our case configuring either one is sufficient. Of course this leaves 
> open the question of comparing local parts. We haven't really solved 
> this one yet.

I think we agree that even though the spec is unclear, routing U-label 
and A-label addresses differently is a bug.  I haven't seen anyone try to 
normalize local parts beyond perhaps NFC.

>> A lot of MTAs add the SMTPUTF8 MAIL FROM tag to all outgoing mail to 
>> servers that offer SMTPUTF8, because why not.

> Because it's possible that regardless of what you have observed, that 
> server may then refuse to relay an EAI-tageed but actually non-EAI 
> message to a non-EAI server?

Hey, don't shoot. I'm just the messenger.  I do minimal EAI, only tag it 
if it needs it.

> Yuck. Frankly, it would be better to just send UTF-8 in this case than 
> to come up with a private encoding for an address that likely belongs to 
> the server you're sending to.

This was their own MAIL FROM which they are presumably prepared to 
unscramble.  They didn't appear to rewite other people's addresses.

> IME the tagging part was trivial. The canonicalization was considerably 

Depends on the way your mail system is set up, in my case it was the other 
way around.

> for invalid messages turns out to align with the proper handling

Not so much mistake as code that happens to work because byte strings are 
byte strings.

> should be followed, what parts should be outright violated, and when.

It'd be interesting to try an applicability statement.  We could shade the 
advice as "here's what we have observed to work."  It's no more of a 
stretch than a BCP that describes guesses that have never been 
implemented, and we have lots of those.

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 15 Jun 2021 17:22:41 -0400

>> Nobody I've seen tags messages as EAI in their internal queues.

Huh.  Why does it set the UTF8 flag on MAIL FROM when sending ASCII messages?

From: Test Sender <[email-masked]>
To: Test Recipient <[email-masked]>
Date: Wed, 09 Jun 2021 16:19:34 +0000
Subject: EAI simple test message 5ea435533e5b10541788

Regards,

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 15 Jun 2021 18:43:33 -0400

> Huh.  Why does it set the UTF8 flag on MAIL FROM when sending ASCII messages?

A few months ago someone @icann.org contacted me regarding "UASG030 EAI  
Testing Results for Courier". After swapping a few messages, the upshot was  
that, apparently, the testing apparatus that was employed in that endeavor  
expected SMTPUTF8 senders specify the UTF8 flag on anything that's sent to a  
receiver that advertises SMTPUTF8, even if it's pure ASCII.

I'm the kind that doesn't keep his INBOX forever. After everything was said  
and done that conversation didn't seem like something that needed to be  
archived, so I don't have the complete exchange any more, but that was the  
capsule summary of it. I couldn't quite see much sense in this myself,  
honestly, but neither did I see much of a reason to die on that specific  
hill, so I just made a few tweaks and moved on.

So, if the majority opinion here is that UTF8 does not need to be set on  
ASCII-only mail, then maybe somewhere would want to discuss this with that  
other group of people, over there.

---

From: "John Levine" <[email-masked]>
Date: 15 Jun 2021 19:51:27 -0400

>receiver that advertises SMTPUTF8, even if it's pure ASCII.

The person running the tests these days is me, and that's the exact
opposite of what the tests are looking for, that it *doesn't* send the
UTF8 flag on ASCII mail. I didn't run the tests on Courier, but I
think the people who did checked correctly.

By the way, last time I checked, when Courier adds a Received line it says
UTF8ESMTP rather than UTF8SMTP.  Did you ever get around to fixing that?  It
looks like a one or two line fix.  Exim had the same bug.

R's,
John

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 15 Jun 2021 21:49:25 -0400

> think the people who did checked correctly.

It's possible that I interpreted the results incorrectly. It was mostly a  
somewhat cryptic data dump, whose contents was not immediately obvious. It's  
also possible I'm misremembering the whole thing, since I don't see the code  
change that I thought was made.

> looks like a one or two line fix.  Exim had the same bug.

Yup, I got around to it.

---

From: Claus Assmann <[email-masked]>
Date: Wed, 16 Jun 2021 07:41:57 +0200

> Huh.  Why does it set the UTF8 flag on MAIL FROM when sending ASCII messages?

I could look at it if you specify how to reproduce that (without
having to install lots of things for some test suite) -- did you
use the -U flag by chance?
You can also report problems to the sendmail support address.

---

From: "John Levine" <[email-masked]>
Date: 16 Jun 2021 16:14:31 -0000

>use the -U flag by chance?

It was locally injected with "sendmail [email-masked] [email-masked]"

I thought it might be the -U flag but I checked and that test doesn't set ut.

Jun 16 16:09:06 xn--5nq21jyu9d1ta sendmail[85985]: 15GG96Ga085985: Authentication-Warning: localhost.my.domain: eaitest set sender to [email-masked] using -f
Jun 16 16:09:06 xn--5nq21jyu9d1ta sendmail[85985]: 15GG96Ga085985: from=[email-masked], size=323, class=0, nrcpts=1, msgid=<[email-masked]>, relay=eaitest@localhost
Jun 16 16:09:07 xn--5nq21jyu9d1ta sendmail[85985]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Jun 16 16:09:07 xn--5nq21jyu9d1ta sm-mta[85986]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1.3, verify=NO, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Jun 16 16:09:07 xn--5nq21jyu9d1ta sm-mta[85986]: 15GG96OH085986: from=<[email-masked]>, size=856, class=0, nrcpts=1, msgid=<[email-masked]>, proto=UTF8SMTPS, daemon=IPv4, relay=localhost [127.0.0.1]
Jun 16 16:09:07 xn--5nq21jyu9d1ta sendmail[85985]: 15GG96Ga085985: to=[email-masked], delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=30323, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (15GG96OH085986 Message accepted for delivery)
Jun 16 16:09:07 xn--5nq21jyu9d1ta sm-mta[85988]: STARTTLS=client, relay=mail1.iecc.com., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Jun 16 16:09:08 xn--5nq21jyu9d1ta sm-mta[85988]: 15GG96OH085986: to=<[email-masked]>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=30856, relay=mail1.iecc.com. [IPv6:2001:470:1f07:1126:33:5370:616d:6d31], dsn=2.6.0, stat=Sent (Accepted message qp 24085 bytes 1975)

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Claus Assmann <[email-masked]>
Date: Wed, 16 Jun 2021 18:56:12 +0200

> It was locally injected with "sendmail [email-masked] [email-masked]"

I can't reproduce that - the sink does not receive an SMTPUTF8 argument:

EHLO seven.dev-lab.TLD
250-local.host ESMTP Hi there        
250-SMTPUTF8  
250-8BITMIME  
250 HELP  
MAIL From:<[email-masked]>  
250 Ok  
RCPT To:<[email-masked]>  
250 Ok  
DATA  

If SMTPUTF8 is not given during submission, then it will only be
added if one of the envelope addresses is rewritten to a non-ASCII
format during processing, e.g., using aliases -- it's easy to
see that in the code:
fgrep 'smtputf8 = true' *.[ch]
If you can provide all the required information to reproduce the
problem, please send it to the sendmail support address.

---

From: Claus Assmann <[email-masked]>
Date: Wed, 16 Jun 2021 20:45:58 +0200

> It was locally injected with "sendmail [email-masked] [email-masked]"

Do you still have that "shim" (mentioned in a usenet posting)?
  In my tests I worked around it with a one line shim that always adds
  -U to the sendmail command and my guess is that everyone will do that.
Hopefully not -- because currently sendmail doesn't "downgrade",
i.e., it does not check whether SMTPUTF8 could be removed
(maybe this might be implemented in a future version).

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 16 Jun 2021 16:44:52 -0400

> Nobody I've seen tags messages as EAI in their internal queues.

Postfix does.

> I think they all notice if an EAI message is sent to a non-EAI server,
> and a few in that case do odd things like turning a UTF-8 local part
> into a MIME encoded word in the envelope.

Postfix does not (erroneously) MIME encode structured header parts.

-- 
    Viktor.

---

From: "John Levine" <[email-masked]>
Date: 16 Jun 2021 17:17:17 -0400

>Do you still have that "shim" (mentioned in a usenet posting)?

Good thinking, but I got rid of the shim, typed a test message with the Mail command, same result:

=====
Date: Wed, 16 Jun 2021 21:12:43 GMT
From: John L <[email-masked]>
To: [email-masked]
Subject: a boring little message

bye
=====

Perhaps the A-labels in the domain name are confusing it?

In any event, I think you will find that expecting the users to tell
you when a message needs EAI handling rather than figuring it out when
it does the submission cleanup will cause endless pain. Neither
postfix nor exim have anything like the -U flag.

R's,
John

---

From: Claus Assmann <[email-masked]>
Date: Thu, 17 Jun 2021 07:29:32 +0200

> Good thinking, but I got rid of the shim, typed a test message with the Mail command, same result:

Obviously something is different on your system compared to my basic
test setup - and without all the necessary info I guess nobody will
be able to reproduce this.

Maybe you could do some debugging:
sendmail -v ...
to see what's going on
and set the DeliveryMode of the sendmail daemon to 'q'
so you can check what's in the queue file?

> Perhaps the A-labels in the domain name are confusing it?

No. See the fine code (as pointed out before):
it looks for non-ASCII chars.