Subject: [ietf-smtp] Is this a valid Message-ID header ?

From: "John Levine" <[email-masked]>
Date: 4 Mar 2020 23:34:13 -0500

I read IETF mailing lists on the IETF's IMAP server* and of late I've been
seeing some awfully funky stuff.

This is an actual Message-ID header in one of the messages in dnsop.
Is that valid?  Even though the MIME decodes to a an ASCII message ID
in the <string@domain> format, I think the answer is no.  That's not
what RFC 5322 sec 3.6 allows.

Message-ID: =?utf-8?q?=3CBN7PR11MB25474DF04998FF3AA9E0B80BC9E40=40BN7PR11MB2?= =?utf-8?q?547=2Enamprd11=2Eprod=2Eoutlook=2Ecom=3E?=

R's,
John

PS: Same question about References with mime-encoded text.

* - actually I copy the messages to my local news server so I can use
threading and killfiles.  The copy program was having trouble with
the message-ids, which NNTP needs to do duplicate detection.

---

From: Paul Smith <[email-masked]>
Date: Thu, 5 Mar 2020 09:13:24 +0000

> Message-ID: =?utf-8?q?=3CBN7PR11MB25474DF04998FF3AA9E0B80BC9E40=40BN7PR11MB2?= =?utf-8?q?547=2Enamprd11=2Eprod=2Eoutlook=2Ecom=3E?=

I would say, no, it's not valid.

The MIME encoding of headers is meant to sit on top of RFC 5322. So, 
format rules in RFC 5322 still have to apply BEFORE decoding the headers.

So,

From: =?utf-8?q?B=C3=B3b Smith?= <[email-masked]>

is valid, because that matches the RFC 5322 rules even though, without 
the MIME decoding, it's slightly gibberish. The format is still valid.

But

From: =?utf-8?q?B=C3=B3b Smith =[email-masked]=3E?=

is not valid, because it doesn't match the rules, even though the MIME 
decoded headers will be identical.

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Thu, 05 Mar 2020 22:27:40 -0500

> what RFC 5322 sec 3.6 allows.

I'll see your Message-ID, and raise you a... not even sure *what* to call it. :)

>From my collection of emails that give exmh indigestion:

% grep UTF-8 183
From: =?UTF-8?B?dGlwLWJvdCBmb3IgRGF2aWQgQnJvd24gPHRpcGJvdEB6eXRvci5jb20+?=@zytor.com
To: =?UTF-8?B?bGludXgtdGlwLWNvbW1pdHNAdmdlci5rZXJuZWwub3Jn?=@zytor.com
Subject: =?UTF-8?B?W3RpcDptbS9yZWFkb25seV0gQVJNL3Zkc286IE1hcmsgdGhlIHZEU08gY29kZSA=?=  =?UTF-8?B?cmVhZC1vbmx5IGFmdGVyIGluaXQ=?=
Git-Commit-ID: =?UTF-8?B?MTFiZjliODY1ODk4OTYxY2VlNjBhNDFjNDgzYzlmMjdlYzc2ZTEyZQ==?=
X-Mailer: =?UTF-8?B?dGlwLWdpdC1sb2ctZGFlbW9u?=
Robot-ID: =?UTF-8?B?PHRpcC1ib3QuZ2l0Lmtlcm5lbC5vcmc+?=
Robot-Unsubscribe: =?UTF-8?B?Q29udGFjdCA8bWFpbHRvOmhwYUBrZXJuZWwub3JnPiB0byBnZXQgYmxhY2tsaXM=?=  =?UTF-8?B?dGVkIGZyb20gdGhlc2UgZW1haWxz?=
MIME-Version: =?UTF-8?B?MS4w?=
Content-Transfer-Encoding: =?UTF-8?B?OGJpdA==?=
Content-Type: =?UTF-8?B?dGV4dC9wbGFpbjsgY2hhcnNldD1VVEYtOA==?=
Content-Disposition: =?UTF-8?B?aW5saW5l?=

Amazingly enough, this managed to collect 10 Received: headers, land in my
mailbox, and exmh *mostly* figured it out:

From: tip-bot for David Brown <[email-masked]>@zytor.com
To: [email-masked]@zytor.com
Subject: [tip:mm/readonly] ARM/vdso: Mark the vDSO code   read-only after init
Git-commit-id: 11bf9b865898961cee60a41c483c9f27ec76e12e
X-mailer: tip-git-log-daemon
Robot-id: <tip-bot.git.kernel.org>
Robot-unsubscribe: Contact <mailto:[email-masked]> to get blacklis  ted from these emails
Mime-version: 1.0
Content-transfer-encoding: 8bit
Content-type: text/plain; charset=UTF-8
Content-disposition: inline

But then finally gave up because it tried to use the undecoded content-type,
gave up, and tossed up a "save/print/don't decode MIME" dialog.

And yes, tip-bot got fixed quite quickly thereafter... :)

---

From: Mark Andrews <[email-masked]>
Date: Fri, 6 Mar 2020 14:36:18 +1100

Conversion of all ASCII headers to utf-8 mime encoded should be banned. Apart
from testing encoding/decoding it serves no useful purpose and if you want
to perform such testing you can always add a non ASCII character.  MUA’s that
do this are deliberately breaking interoperability with old MUAs.

Mark

> On 6 Mar 2020, at 14:27, Valdis Klētnieks <[email-masked]> wrote:
> 
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 5 Mar 2020 23:40:01 -0500

> do this are deliberately breaking interoperability with old MUAs.

The spec is pretty clear where you can have MIME entities and where you 
can't.  You can't have them in Message-ID's, References, or the actual 
address in To, From, Cc, etc.

R's,
John

>> And yes, tip-bot got fixed quite quickly thereafter... :)

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Thu, 05 Mar 2020 23:46:52 -0500

> do this are deliberately breaking interoperability with old MUAs.

You apparently missed the part where the broken bot was quickly fixed...

(And now that I think about it, exmh was actually meta-buggy in that it
accepted a utf-8 encoded Mime-version: header rather than treat it as
a flat non-Mime mail...)

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Mar 2020 00:31:19 -0500

--On Thursday, March 5, 2020 23:40 -0500 John R Levine
<[email-masked]> wrote:

>> Conversion of all ASCII headers to utf-8 mime encoded should
>> be banned. Apart from testing encoding/decoding it serves no
>> useful purpose and if you want to perform such testing you
>> can always add a non ASCII character.  MUA's that do this
>> are deliberately breaking interoperability with old MUAs.
> 
> The spec is pretty clear where you can have MIME entities and
> where you can't.  You can't have them in Message-ID's,
> References, or the actual address in To, From, Cc, etc.

In the interest of clarity, what you are referring to as "MIME
entities" are encoded words as specified in RFC 2231.  Right?

And because encoded words are not permitted in Message-IDs or
the other fields you mention, those fields must conform to the
specified syntax without decoding.   If they don't, it is pretty
much up to the receiving system what to do about it.  I suppose
that could including treating the strings as encoded works and
see where decoding takes one, but that would be so far beyond
the standard that questions about the decoded (UTF-8 or
otherwise) field looks like are really not relevant here: such a
string is either valid or invalid without decoding and, as far
as the standard is concerned, that is the end of it.

    john

---

From: "John R Levine" <[email-masked]>
Date: 6 Mar 2020 11:34:50 -0500

> entities" are encoded words as specified in RFC 2231.  Right?

Right.

> much up to the receiving system what to do about it. ...

That was my understanding, too.  I'm finding these MIME message-ID's in 
the IETF's IMAP server, so now I'm wondering if they were in the original 
message.  Here's one that showed up today.  I think they're all from 
Microsoft.

https://mailarchive.ietf.org/arch/msg/last-call/xTEWTOyy4HOX-wyvFVaOicn2P-I/#

Regards,

---

From: Ned Freed <[email-masked]>
Date: Fri, 06 Mar 2020 08:37:48 -0800 (PST)

The main issue here is that encoded-words are are only defined in the
context of specific pieces of RFC 5322 syntax, and done in a way so that
they don't break the syntax. This is essential so that agents that don't
support RFC 2047 won't have a problem. So this business of encoding the
whole header field, which wrecks the syntax, is simply broken.

However, this doesn't mean something that things which meet the syntax criteria
for encoded words can't appear in other contexts. For example, nothing prevents
you from using an encoded word as a local-part of an address. But if you do,
its interpretation is just that of a local-part, which is a local matter by
definition. And it can't break the overall syntax of the field.

The larger problem is that a lot of agents blindly decode all the encoded words
they find in a field, rather than parsing first and only undoing the ones that
appear in specific places where they have meaning. This has the effect that a
lot of invalid encoded-word usage works, kinda sorta.

And yes, we were well aware that this would be a problem when the RFC was
written. The problem is every other approach we came up with had even bigger
issues. Keith deserves a lot of credit for this work.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Mar 2020 12:24:50 -0500

Ned,

Complete agreement.  And thanks for reminding me/ us about the
additional details.   An additional problem with at least the
ones of these Message-ID (and occasionally the mailbox addresses
in "From:" header fields) is that there is no "@" present in the
encoded form.  So anything that follows the principle of parsing
before trying to do any decoding will fail; only those systems
that, contrary to the clear intend of the spec (for the reason
you give), decode everything that looks like an encoded word
before examining or parsing the relevant fields has even a
chance of imagining that these things are valid.

best,
   john

--On Friday, March 6, 2020 08:37 -0800 Ned Freed
<[email-masked]> wrote:

>> > In the interest of clarity, what you are referring to as
>> > "MIME entities" are encoded words as specified in RFC 2231.
>> > Right?
> 
>> Right.
> 
>> > And because encoded words are not permitted in Message-IDs
>> > or the other fields you mention, those fields must conform
>> > to the specified syntax without decoding.   If they don't,
>> > it is pretty much up to the receiving system what to do
>> > about it. ...
> 
>> That was my understanding, too.  I'm finding these MIME
>> message-ID's in the IETF's IMAP server, so now I'm wondering
>> if they were in the original message.  Here's one that showed
>> up today.  I think they're all from Microsoft.
> 
>> John Levine, [email-masked], Taughannock Networks,
>> Trumansburg NY Please consider the environment before reading
>> this e-mail. https://jl.ly
>

---

From: Keith Moore <[email-masked]>
Date: Sat, 7 Mar 2020 19:59:59 -0500

> Message-ID: =?utf-8?q?=3CBN7PR11MB25474DF04998FF3AA9E0B80BC9E40=40BN7PR11MB2?= =?utf-8?q?547=2Enamprd11=2Eprod=2Eoutlook=2Ecom=3E?=

No, it's not valid.   The above field body scans as "atom atom", and a 
msg-id cannot begin with two atoms without some other symbol (e.g. ".") 
in between.

RFC 2047 encoded-words were designed to be compatible with the 
(then-current) RFC 822 grammar, so that they wouldn't break existing 
parsers.   That means that any decoding of encoded-words for 
presentation occurs after (header field-specific) parsing. Also, section 
5 of RFC 2047 limits use of encoded-words to very specific places in the 
message header.   Any text that parses as atom that appears elsewhere in 
a message header, should be treated as an ordinary atom, even if it 
looks like an encoded-word.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sat, 7 Mar 2020 20:00:36 -0500

> do this are deliberately breaking interoperability with old MUAs.

+1

Keith