Subject: [ietf-smtp] smtp extension model (was: Re: [Proposal] confusing parts of the mail system, was 250-MARKDOWN)

From: John Bucy <[email-masked]>
Date: Wed, 30 Jan 2019 11:38:48 -0800

On Thu, Jan 17, 2019 at 5:41 AM Ned Freed <[email-masked]> wrote:

> > On 17/01/2019 04.29, Ned Freed wrote:
>
> > > Clearly not, as you yourself have noted.
>
> > Microsoft supports BINARYMIME for received mail on port 25.
>
> > This something that, I hear, is very difficult for others to do.  I'm
> > told it's very complicated.  Somehow they have figured out how to make
> > this work.
>
> They have not figured out how to make it work, because it's not possible to
> make it work in general. Transcoding destroys DKIM signatures, period. So
> they
> either:
>
> (1) Reject messages sent with BINARYMIME and signatures, either in all
> cases
>     or in cases where they know they are going to forward, leading to
>     unncessary failures, or,
>
> (2) Accept such messages and trash the signatures, leading to unnecessary
>      failures, and,
>
> (3) Deal with dynamic forwarding cases either by not having any or
>     falling back to (1) or (2), leading to unnecessary failures.
>

Would it be useful to write an informational rfc "The SMTP Extensions Model
in 2019" to discuss these pitfalls in detail? 5321 2.2 doesn't have a lot
to say about what to do in this case.

cheers
john

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 30 Jan 2019 19:44:04 +0000

Hi John,

On 30/01/2019 19:38, John Bucy wrote:
> On Thu, Jan 17, 2019 at 5:41 AM Ned Freed <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     make it work in general. Transcoding destroys DKIM signatures,
>     period. So they
>     either:
>
>     (1) Reject messages sent with BINARYMIME and signatures, either in
>     all cases
> Would it be useful to write an informational rfc "The SMTP Extensions 
> Model in 2019" to discuss these pitfalls in detail? 5321 2.2 doesn't 
> have a lot to say about what to do in this case.

I think it would be a useful document.

Best Regards,

Alexey

---

From: "John Levine" <[email-masked]>
Date: 30 Jan 2019 15:39:03 -0500

>I think it would be a useful document.

Agreed.  There's a lot of folklore that could stand to be written down.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 Jan 2019 22:46:02 -0500

--On Wednesday, January 30, 2019 19:44 +0000 Alexey Melnikov
<[email-masked]> wrote:

>> On Thu, Jan 17, 2019 at 5:41 AM Ned Freed
>> <[email-masked]  <mailto:[email-masked]>> wrote:
>> 
>>     > This something that, I hear, is very difficult for
>>     > others to
>>     do.  I'm
>>     > told it's very complicated.  Somehow they have figured
>>     > out how
>>     to make
>>     > this work.
>> 
>>     make it work in general. Transcoding destroys DKIM
>>     signatures, period. So they
>>     either:
>> 
>>     (1) Reject messages sent with BINARYMIME and signatures,
>>     either in all cases
>>         or in cases where they know they are going to
>>     forward, leading to     unncessary failures, or,
>> 
>>     (3) Deal with dynamic forwarding cases either by not
>>     having any or     falling back to (1) or (2), leading
>>     to unnecessary failures.
>> 
>> 
>> Would it be useful to write an informational rfc "The SMTP
>> Extensions  Model in 2019" to discuss these pitfalls in
>> detail? 5321 2.2 doesn't  have a lot to say about what to do
>> in this case.
> 

While I agree that this would be useful, I don't think the title
suggested above properly captures the issue(s).  I think there
are three, quite separate, ones:

(1) Absolutely nothing prevents (and I can't imagine what we
could do that would prevent) an SMTP Server from offering
extension keywords whose effects would be inconsistent or even
contradictory nor an SMTP client from accepting/ requesting such
extensions.   IMO, we have been protected from many occurrences
of such situations by good sense rather than anything that
should have been said in the protocol spec.   If 5321 needs to
say that more explicitly, please file an "hold for update"
erratum to flag the issue so it does not get lost and, ideally,
suggest text.  If we need to spell out what should be done
and/or create a new reply code for the situation, that would be
a substantive update to 5321.  Let's get that I-D  written and
process it somehow (see below).

(2) If a particular extension/option implies formatting or other
information that will either be destroyed by message signing or
will cause the signature to fail or be trashed, that should
almost certainly be explained.  However, the explanation is a
matter for the spec for that particular extension or that for
the signature mechanism rather than a generic problem with the
extension mechanism or extensions in general.  It is hence not
really an issue for 5321 or its successor(s).

(3) SMTPUTF8 (non-ASCII addresses and headers), RFC 6531,
particularly Sections 1.2 and 3.6, requires support for 8BITMIME
and specifies a way to use BINARYMIME  for relevant body parts
(which, given message/global, may include non-ASCII,
non-encoded-word, header information).  When the EAI WG was
doing that work, it was clear that interactions with DKIM were
unspecified and fairly clear that DKIM was not going to work
with such addresses and message headers without some careful
thinking and specification adjustments.  At the time, the IETF
decided to move forward with both sets of specs, presumably on
the theory that we'd sort it out later.   Seven years later, it
is probably "later" and time to face the issue in some way.
Again, this doesn't have much to do with 5321 or its successors,
but could have profound effects on whatever recommendations we
make about use of SMTPUTF8 features and/or DKIM and DKIM-like
message signatures.

best,
   john

p.s. for those who don't know, I've been keeping and updating a
private copy of 5321bis for circa ten years now, modifying it
when required changes are clear and inserting notes about
unresolved issues that would need to be addressed if and when
5321 is revised.  I'm willing to get that version into shape and
post it any time one or more ART ADs indicate that I should do
that and that they have a plan for processing the document.  The
latter is not obvious.  While 5321 almost certainly needs more
work that 5322, it probably makes no sense to open one without
the other.  Even for 5321 alone, there are a number of issues
that ought to be addressed for which I do not believe there is
consensus about how to do that.  In addition, while others may
disagree, I think it makes little sense to open either 5321 or
5322 unless we are headed toward Full Internet Standard status
and that implies that anything that modifies ("updates") it
should be processed as Proposed Standard, implemented, etc.,
sufficiently to qualify for Full Standard status and then folded
in rather than our needing to produce another PS version of SMTP
because there are unimplemented loose ends in those specs.  One
of the issues in that category is whether we are ready to fold
SMTPUTF8, including non-ASCII headers (at least trace
information), into the base SMTP spec or whether we want a final
(Full Standard) SMTP specification that is restricted to ASCII.
I think those kinds of questions are going to require a WG
rather than, e.g., Pete and I getting together and putting
drafts together with while everyone will be comfortable.  YMMD
but be careful what you wish for or wish the ADs to wish for.

---

From: "John Levine" <[email-masked]>
Date: 31 Jan 2019 21:27:17 -0500

In article <F5A0D1E3B5FC14EEFBB6C7E7@PSB> you write:

skipping over all the parts I agree with ...

>...  When the EAI WG was
>thinking and specification adjustments. ...

DKIM turns out to be OK on SMTPUTF8 mail.  So long as you do the DKIM
signing after all of the cleanup that submission does, the signatures
are fine.  If something later recodes or reorders body parts, the
signatures will break, but that's hardly unique to EAI.  DKIM has some
optional canonicalization rules for computing the hashes which turn
out to be useless.  Hashes using the complicated fuzzy rules don't
work any better than the simple rule that says hash everything
verbatim except drop blank lines at the end of the message.

I have a tiny draft in process that says that in DKIM signature
headers in SMTPUTF8 messages, store IDNs as U-labels but that's not a
big deal. If someone uses A-labels anyway, it should still work.

R's,
John