Subject: Re: [EAI] non-EAI messages

From: Shawn Steele <[email-masked]>
Date: Fri, 31 Dec 2010 18:22:01 +0000

> The reason it's a terrible idea is that only some subset of the messages a given MTA handles will be EAI messages.

I'm not sure how this is interesting?  

If the server thought it was EAI, but it only used the ASCII range, the exact same thing would (should) happen as if it wasn't an EAI message.

So, IMO, having the client say "I speak EAI too" once it hears UTF8SMTP would finish the handshaking.

OTOH: If I'm an EAI aware client, and I get a response from an EAI aware server that requires UTF-8, I'd really like that to happen, even if my original message & header were all ASCII.
 
-Shawn

---

From: Troy Starr <[email-masked]>
Date: Fri, 31 Dec 2010 13:01:55 -0800

Hi Shawn -

> I'm not sure how this is interesting?

Assuming that all of the future MTAs in the message delivery chain support UTF8SMTPbis, then I agree, it's not interesting.  But what happens if the previous MUA/MSA/MTA applied a UTF8SMTPbis flag to the entire session rather than on a per-message basis, then the current MTA wants to deliver one of those messages to an MTA which doesn't support UTF8SMTPbis?  I see the following possibilities:

1. The message envelope and/or headers really does contain UTF-8 characters.
   (a) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, bounces the e-mail back to the sender with an NDR.
   (b) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, parses the envelope and message headers looking for non-ASCII characters.  It finds at least one non-ASCII character, so it bounces the e-mail back to the sender with an NDR.
   (c) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, parses the envelope and message headers looking for non-ASCII characters.  It has a bug that leads it to believe that there are no non-ASCII characters, so it assumes that the UTF8SMTPbis flag was an ignorable error and attempts to send the message anyway.  At this point, who knows what will happen to the message since we've just violated the RFC 5321 SMTP specification.  Perhaps the receiving MTA will catch the violation and reject the submission.  Perhaps the receiving MTA will attempt to deliver the message, but it will be corrupted.  Perhaps the message will silently go into a black hole, never to be seen again.  In any event, it is undefined, unpredictable, and unreliable.
   (d) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, attempts to send the message anyway.  At this point, who knows what will happen to the message since we've just violated the RFC 5321 SMTP specification.  Perhaps the receiving MTA will catch the violation and reject the submission.  Perhaps the receiving MTA will attempt to deliver the message, but it will be corrupted.  Perhaps the message will silently go into a black hole, never to be seen again.  In any event, it is undefined, unpredictable, and unreliable.

2. The message envelope and/or headers only contains ASCII characters.
   (a) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, bounces the e-mail back to the sender with an NDR.
   (b) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, parses the envelope and message headers looking for non-ASCII characters.  It doesn't find any, so it assumes that the UTF8SMTPbis flag was an ignorable error and sends the message anyway.  The message arrives successfully because there are no non-ASCII characters present.
   (c) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, parses the envelope and message headers looking for non-ASCII characters.  It has a bug that leads it to believe there are non-ASCII characters, so it bounces the e-mail back to the sender with an NDR.
   (d) The MTA, seeing that the MTA it's trying to deliver the message to doesn't support UTF8SMTPbis, attempts to send the message anyway.  The message arrives successfully because there are no non-ASCII characters present.

If we were to quickly summarize pros/cons of each, we would get:

(a) Trust UTF8SMTPbis flag, no parsing
Pro: It is predictable and incurs virtually no cost to the MTA.
Con: It will not attempt any "downgrade" to deliver the message which may have been deliverable if it made an extra effort.

(b) and (c) Ignore UTF8SMTPbis flag, parse
Pro: It makes an extra effort to "downgrade" and deliver the message.
Con: It imposes a message processing cost on the MTA, which may be an implementation hurdle, and may also be a significant processing cost for high-volume MTAs.  It may also be unpredictable if the parser is buggy.  Finally, it imposes the MTA's judgment of whether the message should be considered a UTF8SMTPbis message over the original sender's judgment.

(d) Ignore UTF8SMTPbis flag
Pro: It might happen to work some of the time.
Con: Is not predictable and will often violate SMTP RFCs.

I think we can agree that (d) is a poor choice and should be rejected as an option.  (a) is the simplest and least costly route since it relies on the first link in the chain to properly flag a UTF8SMTPbis message as such.  If that message does not contain non-ASCII characters, then it's important for the first link in that chain to not flag the message as a UTF8SMTPbis message.  (b) and (c) are the tricky ones.  All of the cons I've listed make me very cautious about recommending (b) as an option, let alone assuming that it will be the typically implemented behavior.

As such, (a) seems to be the wisest choice IMO.

> OTOH: If I'm an EAI aware client, and I get a response from an EAI aware server that requires UTF-8, I'd really like that to happen, even if my original message & header were all ASCII.

Maybe I've parsed your sentence poorly, and if so I apologize...  I don't think there's such a thing as an EAI aware server that notifies the client that it requires UTF-8 (by which I assume you mean will only accept messages flagged as UTF8SMTPbis).  This RFC allows EAI aware servers to advertise the option to clients to send messages as UTF8SMTPbis messages, but as an optional extension to SMTP, it's still perfectly valid to send a non-UTF8SMTPbis message.  I haven't seen any part of the RFC that allows the server to advertise to the client that it only accepts messages flagged as UTF8SMTPbis.

In addition, I don't see what advantage you get flagging a pure-ASCII header message with the UTF8SMTPbis flag.  Indeed, I only see disadvantages for the reasons I've stated above.  What are the advantages you see in flagging messages as UTF8SMTPbis even if the envelope and headers are only ASCII?

- Troy Starr