Subject: [ietf-smtp] Should we update an RFC if people refuse to implement parts of it ?

From: "John R. Levine" <[email-masked]>
Date: 24 May 2021 19:05:53 -0400

I've been doing some EAI tests.  RFC 6531 section 3.7.3 says that domain 
names in trace headers SHOULD all be U-labels.  In practice, everyone uses 
A-labels for reasons that are not absurd.  The FROM clause has the name 
from the EHLO command which has to be sent as A-labels, and it's quite 
plausible to have a message where the message itself is entirely ASCII, 
sent to a UTF-8 address, which then could be forwarded to an ASCII address 
except that it has those U-labels in the trace header which would have to 
be downgraded.  I've talked to a couple of MTA maintainers who have told 
me forget it, that's silly, not doing that.

In a situation like this, would it make sense in a future update to the 
RFC to adjust the advice to the reality?

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Mon, 24 May 2021 22:31:59 -0400

It would certainly be appropriate to either revise the spec or,
better from my point of view, create a short applicability
statement that explains the reasoning behind the SHOULD and the
circumstances under which it would be sensible to ignore it.

The thing that sometimes gets lost in "let's make the spec
conform to what implementations are doing" discussions in this
area is that there is an assumption behind the whole collection
of SMTPUTF8 specs, namely that the world really wanted non-ASCII
addressing and header field values.   If that were true and
things had gone as quickly as the advocates predicted, then your
argument would not hold because the odds of the downstream
system in the scenario you describe not having support for the
extension would be low. If we are merely being slower than
expected getting there, then advising people to do something
that is sub-optimal for other reasons is a bad idea because it
is likely to just slow things down further.

Now, if, on the other hand, the reality were that no one is
going to fully implement SMTPUTF8 other than mail providers in a
few countries that intend to use it internally, then we should
probably be reconsidering the whole model --not just this
detail-- lest we end up with two mail systems and needing
gateways, possibly even gateways with serious downgrade and/or
encapsulation facilities between them.

   Just my personal opinion, of course.
    john

--On Monday, May 24, 2021 19:05 -0400 "John R. Levine"
<[email-masked]> wrote:

> I've been doing some EAI tests.  RFC 6531 section 3.7.3 says
> that domain names in trace headers SHOULD all be U-labels.  In
> practice, everyone uses A-labels for reasons that are not
> absurd.  The FROM clause has the name from the EHLO command
> which has to be sent as A-labels, and it's quite plausible to
> have a message where the message itself is entirely ASCII,
> sent to a UTF-8 address, which then could be forwarded to an
> ASCII address except that it has those U-labels in the trace
> header which would have to be downgraded.  I've talked to a
> couple of MTA maintainers who have told me forget it, that's
> silly, not doing that.
> 
> In a situation like this, would it make sense in a future
> update to the RFC to adjust the advice to the reality?

---

From: "John Levine" <[email-masked]>
Date: 25 May 2021 14:29:45 -0400

It appears that John C Klensin  <[email-masked]> said:
>circumstances under which it would be sensible to ignore it. ...

>addressing and header field values. ...

This may seem like splitting hairs but there is a difference between
header fields that users see and fields that they don't.  Message-IDs
are still generally ASCII in EAI messages, and I don't see any
benefit from making the domain names in trace headers U-labels rather
than A-labels.

I'm not getting any pushback on Return-Path which really does need
to be UTF-8 is it's an EAI address.  I think the few MTAs that put
a FOR clause in the Received header put EAI addresses there, too.

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 25 May 2021 15:32:52 -0400

> me forget it, that's silly, not doing that.

Disclaimer: I'm one of the MTA maintainers who found the RFC advice in
question to be unworthy of implementation.

> RFC to adjust the advice to the reality?

Without daring to generalise to broader questions of whether/when the
spec or the implementations have the final say, on the particulars of
this specific case I think it is easy to conclude that it best to amend
the specification.

    * The specification in question prescribes the format of some fields
      of trace headers that most users will not see.

    * The specifications causes the format used in the trace header to
      differ from the actual data exchanged on the wire.  These are the
      client-to-server EHLO name and the server-to-client greeting name,
      also repeated in the first line of the EHLO response.

      Because both are exchanged prior to negotiation of SMTPUTF8, they
      always in A-label form.

    * The trace headers should match data recorded in system logs and
      are mostly for the benefit of the postmaster.  There's more value
      in making them readable to any/all the postmasters along the
      delivery path, than to those versed in the native tongue of the
      message sender and ultimate recipients.

    * It is possible that the message headers are all ASCII, and the
      only presence of UTF8 is in the envelope, which may get rewritted
      or narrowed in transit.  By the time the message is delivered to
      some mailboxes, the only non-ASCII content that might remain could
      be the trace headers in question (if they follow the RFC).

      Given that not all IMAP clients, MUAs, ... support EAI, the
      conservative thing to do (a la Postel) is to not needlessly
      inject UTF8 content into the message headers.

Therefore, I find the specific RFC text to be misguided, and not
only inessential to interoperability/utility, but actually working
against both.  I am therefore inclined to ignore this part of the
RFC with prejudice.  If there's an opportunity to revise (remove)
the text, that'd be great.

-- 
    Viktor.

---

From: John C Klensin <[email-masked]>
Date: Tue, 25 May 2021 15:34:13 -0400

--On Tuesday, May 25, 2021 14:29 -0400 John Levine
<[email-masked]> wrote:

>> It would certainly be appropriate to either revise the spec
>> or, better from my point of view, create a short applicability
>> statement that explains the reasoning behind the SHOULD and
>> the circumstances under which it would be sensible to ignore
>> it. ...
> 
>> area is that there is an assumption behind the whole
>> collection of SMTPUTF8 specs, namely that the world really
>> wanted non-ASCII addressing and header field values. ...
> 
> This may seem like splitting hairs but there is a difference
> between header fields that users see and fields that they
> don't.  Message-IDs are still generally ASCII in EAI messages,
> and I don't see any benefit from making the domain names in
> trace headers U-labels rather than A-labels.
> 
> I'm not getting any pushback on Return-Path which really does
> need to be UTF-8 is it's an EAI address.  I think the few MTAs
> that put a FOR clause in the Received header put EAI addresses
> there, too.

I recognize the distinction while also realizing that, as you
know, things often leak.  My recollection (if Jiankang is
following this, his memory is probably better than mine) is that
the WG explicitly discussed the issue and concluded that
U-labels were a better idea than A-labels.  

One way or the other, let me split a different hair.  As I read
RFC 6531, all the penultimate paragraph of Section 3.7.3 says is
that the "server SHOULD use the U-label form".   It does not
discuss circumstances under which it is reasonable for an
implementation to do otherwise.  If the implementations you are
talking about have rational reasons for doing otherwise, then
they are still conforming, even if not exhibiting preferred
behavior.   Does that mean the spec should be revised?  I don't
think so unless (1) we feel a need to be explicit about when it
is ok to do something different than the SHOULD specifies, (2)
we gave decided it is not the preferred behavior after all, or
(3) we have concluded that the assumption that substantially all
MTAs will migrate to SMTPUTF8 support is incorrect.  

So, barring the third option above and given that U-labels and
A-labels are reversibly convertible without information loss, I
just do not see a big deal 
here, certainly not one that would justify reopening the
document(s).

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 25 May 2021 16:06:49 -0400

Victor, John, and probably others,

(explicitly adding Jiankang and the ART ADs in case they not
been following this list closely.)

In addition to believing the text is probably right as it now
appears, I should be explicit about why I'm pushing back on
doing anything about this other that reassuming implementers who
use A-labels that no one is likely to call the Protocol Police.  

The other reasons, and the more important ones, are completely
pragmatic.  First, given the heavy dependencies of the SMTPUTF8
documents on RFCs 5321 and 5322 and that the latter are being
revised, I think it would not be helpful to revises the SMTPUTF8
documents until after we have finished with the revisions of the
base documents.  As I assume others have noticed, those
revisions are not proceeding quickly... and, wherever the fault
lies, is it not with Pete or myself. Second, even if there is
sufficient energy to do serious work on email (and the rate of
progress in EMAILCORE could be interpreted as evidence to the
contrary), there is ample evidence (painful to me personally)
that there is little or none of the combined knowledge,
expertise, and energy to do serious i18n work.  Revision of
these specs requires knowledge and interest in both (and, for
this particular case, specific knowledge and experience with
some of the subtle tradeoffs in IDNA2008).  And finally, unless
I have misread the signals from the ADs, we are not likely to
see many (if any at all) no-WG, AD-sponsored standards track
documents about which there is any controversy.  In the case,
the controversies are already clear from this fairly short
thread.  So, unless someone is really to propose a WG and make a
serious suggestion about where the energy is going to come from
--especially when EMAILCORE is still going on-- well...

I would add to this disclaimer that my hope and expectation when
the EAI WG wound down was that fully conforming implementations
would deploy rapidly and extensively enough that, by the time we
got around to revising 5321 and 5322, we could just fold most of
those specs in, maybe talking about "minimal" and "recommended"
(i.e., fully internationalized) implementations.  Silly me.

best,
   john

--On Tuesday, May 25, 2021 15:32 -0400 Viktor Dukhovni
<[email-masked]> wrote:

>> that domain  names in trace headers SHOULD all be U-labels.
>> In practice, everyone uses  A-labels for reasons that are not
>> which has to be sent as A-labels, and it's quite  plausible
>> to have a message where the message itself is entirely ASCII, 
> Disclaimer: I'm one of the MTA maintainers who found the RFC
> advice in question to be unworthy of implementation.
> 
> Without daring to generalise to broader questions of
> whether/when the spec or the implementations have the final
> say, on the particulars of this specific case I think it is
> easy to conclude that it best to amend the specification.
> 
>     * The specification in question prescribes the format of
> some fields       of trace headers that most users will not
> see.
> 
>     * The specifications causes the format used in the trace
> header to       differ from the actual data exchanged on the
> wire.  These are the       client-to-server EHLO name and the
> server-to-client greeting name,       also repeated in the
> first line of the EHLO response.
> 
>       Because both are exchanged prior to negotiation of
> SMTPUTF8, they       always in A-label form.
> 
>     * The trace headers should match data recorded in system
> logs and       are mostly for the benefit of the postmaster.
> There's more value       in making them readable to any/all
> the postmasters along the       delivery path, than to those
> versed in the native tongue of the       message sender and
> ultimate recipients.
> 
>     * It is possible that the message headers are all ASCII,
> and the       only presence of UTF8 is in the envelope, which
> may get rewritted       or narrowed in transit.  By the time
> the message is delivered to       some mailboxes, the only
> non-ASCII content that might remain could       be the trace
> headers in question (if they follow the RFC).
> 
>       Given that not all IMAP clients, MUAs, ... support EAI,
> the       conservative thing to do (a la Postel) is to not
> needlessly       inject UTF8 content into the message headers.
> 
> Therefore, I find the specific RFC text to be misguided, and
> not only inessential to interoperability/utility, but actually
> working against both.  I am therefore inclined to ignore this
> part of the RFC with prejudice.  If there's an opportunity to
> revise (remove) the text, that'd be great.

---

From: "John Levine" <[email-masked]>
Date: 25 May 2021 22:45:34 -0400

It appears that John C Klensin  <[email-masked]> said:
>base documents. ....

Oh, absolutely.  If it seemed like I wanted to revise 6531 right now,
I don't.  It was more about is it worth making a list of things we
might want to revise later.

---

From: "Jiankang Yao" <[email-masked]>
Date: Thu, 27 May 2021 00:59:53 +0800

From: John C Klensin
Date: 2021-05-26 03:34
To: John Levine; ietf-smtp
Subject: Re: [ietf-smtp] Should we update an RFC if people refuse to implement parts of it ?

> U-labels were a better idea than A-labels.  

Yes, I think so. The EAI WG discussed this issue.
Section 3.7.3. Trace Information  encourages to use UTF-8 form.
One reason I think is that trace information will be put into header for human reading. UTF-8 is better for human reading.
A-label is ugly for human reading, but ok for machine reading.

Best Regards
Jiankang Yao

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 26 May 2021 17:59:49 -0400

On Thu, May 27, 2021 at 12:59:53AM +0800, Jiankang Yao wrote:

> > I recognize the distinction while also realizing that, as you know,
> > things often leak.  My recollection (if Jiankang is following this,
> > his memory is probably better than mine) is that the WG explicitly
> > discussed the issue and concluded that U-labels were a better idea
> > than A-labels.  
> 
> Yes, I think so. The EAI WG discussed this issue.  Section 3.7.3.
> Trace Information  encourages to use UTF-8 form.  One reason I think
> is that trace information will be put into header for human reading.

But, and this is crucial, the human reading the trace information is
rarely either the sender or the ultimate recipient of the message, who
are generally presented with a subset of the headers fields ("To", "Cc",
"Date", "Subject" ...).  Examination of trace headers is far more likely
to a task for a mail system administrator.  They're used in abuse
reports and the like, and a uniform representation is more important
than familiarity to the community of readers of some given language.

Mail system administrators don't necessarily read the same scripts as
their end users.  Mail system administrators would reasonably expect
the EHLO name recorded in trace headers to match the name received in
the actual EHLO command on the wire (and recorded in their email logs).

> UTF-8 is better for human reading.  A-label is ugly for human reading,
> but ok for machine reading.

Sure, if the text is Russian, some Latin-based alphabet or at a stretch
Greek, I can more easily distinguish one U-label string from another
than an A-label form like "xn--b1adqpd3ao5c.org", ... and yet I'd much
rather see A-labels in trace headers than Arabic or Chinese.  The text
in 3.7.3 is not something I'm inclined to implement.

Specifying the use of U-labels in the "from" and "by" clauses rather
looks like a bad judgement call, rough consensus or not.  Until the
Protocol Police show up, I'm sticking with A-labels. :-(

-- 
    Viktor.

---

From: Ned Freed <[email-masked]>
Date: Sun, 30 May 2021 15:49:25 -0700 (PDT)

> than familiarity to the community of readers of some given language.

And what the admin usually wants to do is either a comparison or check the
domain with the DNS in some way. So an A-label can be more convenient.

And in the unlikely event an admin needs to translate the A-label to a
U-label, there are an abundance of tools that I can use to do it.

> in 3.7.3 is not something I'm inclined to implement.

Actually, it depends on the A-labels. Because of the compression involved
A-labels often emphasize small differences that may be difficult to see in a
crap monospaced font for a non-Latin script, even one you're familiar with.

> Protocol Police show up, I'm sticking with A-labels. :-(

Me too.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Sun, 30 May 2021 21:19:09 -0400

Gentlemen,

An observation and a question...

--On Sunday, May 30, 2021 15:49 -0700 Ned Freed
<[email-masked]> wrote:

>> > > I recognize the distinction while also realizing that, as
>> > > you know, things often leak.  My recollection (if
>> > > Jiankang is following this, his memory is probably better
>> > > than mine) is that the WG explicitly discussed the issue
>> > > and concluded that U-labels were a better idea than
>> > > A-labels.
>> > 
>> > Yes, I think so. The EAI WG discussed this issue.  Section
>> > 3.7.3. Trace Information  encourages to use UTF-8 form.
>> > One reason I think is that trace information will be put
>> > into header for human reading.
> 
>> But, and this is crucial, the human reading the trace
>> information is rarely either the sender or the ultimate
>> recipient of the message, who are generally presented with a
>> subset of the headers fields ("To", "Cc", "Date", "Subject"
>> ...).  Examination of trace headers is far more likely to a
>> task for a mail system administrator.  They're used in abuse
>> reports and the like, and a uniform representation is more
>> important than familiarity to the community of readers of
>> some given language.
> 
> And what the admin usually wants to do is either a comparison
> or check the domain with the DNS in some way. So an A-label
> can be more convenient.
> 
> And in the unlikely event an admin needs to translate the
> A-label to a U-label, there are an abundance of tools that I
> can use to do it.

I may live in a different world, or at least with different
users, than the two of you, but I quite often see users pointed
to trace fields in order to make estimates of the validity of a
message.  And that brings me close to the reasoning I think the
WG used more generally: that it was better to stick to "native
character" forms (in this case, U-labels), especially when there
was a possibility of a full mailbox name with a UTF-8 local part
and a domain part containing IDN labels.   That is obviously not
the case for clauses of "Received:" fields other than "for",
but, if the WG extrapolated from that principle to this
particular case (and I don't remember how much detailed
attention the particular case got.  The other argument for
U-labels is that, precisely as Ned points out, if an admin needs
to translate an A-label, there are many tools and admins are
likely to know where to find them.  By contrast, if a user is
presented with an A-label, the reaction is at least as likely to
be "WT<rude word>" as "I have a tool handy that does that".

A different piece of the same story is that, if I had much more
confidence that authors of MUAs and other mail access tools
would think carefully about these subtle issues and get them
right, I'd say that, given the dual relationship between
A-labels and U-labels, it makes no difference which of the two
are used on the wire -- it is all a presentation issue.  On the
other hand, my experience of the last decade or so has given me
no such confidence.

>...
>> Sure, if the text is Russian, some Latin-based alphabet or at
>> a stretch Greek, I can more easily distinguish one U-label
>> string from another than an A-label form like
>> "xn--b1adqpd3ao5c.org", ... and yet I'd much rather see
>> A-labels in trace headers than Arabic or Chinese.  The text

And you might feel differently if, instead of reading primarily
Greek, Latin, and Cyrillic, your primary familiarity was with
Chinese or Devanagari or Mongolian or Thai or Arabic or anything
else that is very different from those three very closely
related scripts.   And your ability to distinguish U-labels in
the three scripts you cite may be severely impaired if someone
is deliberately trying to deceive given the number of characters
that are reused among the three scripts and, in many type
styles, almost indistinguishable.  

And, yes, the conclusion from the above is that sometimes
A-labels are better and sometimes U-labels are better and that
sometimes it depends on the reader.

> Actually, it depends on the A-labels. Because of the
> compression involved A-labels often emphasize small
> differences that may be difficult to see in a crap monospaced
> font for a non-Latin script, even one you're familiar with.

Exactly.

>> Specifying the use of U-labels in the "from" and "by" clauses
>> rather looks like a bad judgement call, rough consensus or
>> not.  Until the Protocol Police show up, I'm sticking with
>> A-labels. :-(
> 
> Me too.

IMO, wfm.  The text says "SHOULD", the two are easily
convertible by anyone who knows where to find the tools (and
that will far more often be admins than end-users even those I'm
concerned about the latter too) and, for "from" at least, simply
copying the EHLO field (which necessarily uses A-labels for any
IDN label fields) seems to me to be a fairly strong argument.
But that isn't the question in the subject line that started
this thread: should we open the document and change it, perhaps
to say "SHOULD...A-label".   Well, I'm not convinced that is it
worth the trouble, especially given all of the other issues we
are having getting SMTPUTF8 universally understood and deployed.

On the other hand, and here comes the question: There are
proposals floating around that would define new header fields
that would reflect, include, or depend on, different sorts of
forward-pointing and reverse-pointing addresses.  Should we be
taking the position that none of those should move forward
unless they explicitly address what should be done when those
fields are not traditional ASCII ones?

best,
  john