Subject: Re: [ietf-smtp] Email standard revision, was address maximum length

From: John C Klensin <[email-masked]>
Date: Mon, 25 Nov 2019 21:45:38 -0500

--On Sunday, November 24, 2019 12:35 +0100 Alessandro Vesely
<[email-masked]> wrote:

> On Sat 23/Nov/2019 13:35:00 +0100 John C Klensin wrote:
>> there is an effort afoot to get revisions to 5321 and 5322
>> out there.
> 
> Where?

I will let the ADs address this at their convenience.  Any
questions about apparent top-down planning for WGs should also
be addressed to them and/or to the Nomcom as they consider
criteria for open slots.  In fairness, please give the ADs a
week or two to recover from IETF, any associated travel or
vacations, etc., before setting off a speculative storm of
postings on that topic and this list.

This rather long response is addressed rather more to a future
discussion about a scope statement for a possible WG to update
some mail documents as it is a specific answer to Alessandro.
As a result, he should probably be recognized for opening a can
of worms :-)

And see my personal comment and opinion that make up the last
half of this note.

>> [... guidelines for extension publishing omitted ...] because
>> there is about zero chance that anything would be included in
>> a 5321bis that would force it back to Proposed Standard.

> Yet, I think something will have to be said about the now
> common habit of From: rewriting.  Section 3.9, Mailing Lists
> and Aliases, still considers only rewriting of the envelope
> from.  Reality now differs.

Almost anything that addresses "From:" (I assume you are
referring to the message header field, not the MAIL command)
rewriting would need to be at least partially a 5322bis issue,
not a 5321bis one.  We've typically had to explain more times
than I care to remember (and in what has often turned out to be
a rather painful process) that SMTP is (still) allowed to
transport message content that does not conform to 2822/5382 and
that having MTAs mess with what 5321 thinks of as content
endangers that principle. 2821 and 5321 weakened that boundary
somewhat relative to the very clear boundary between 821 and
822, but it is not clear whether weakening it more is desirable.
A similarly-fuzzy boundary exists between 5322, which is very
clear that it is not in the MIME business, and the MIME specs,
noting that Content-Type:, Content-transfer-encoding:, and even
MIME-version: header fields are, structurally, very much part of
the main message headers and not material specific to body
parts.  

> I don't think such changes would imply going back to PS, given
> the loosely normative content of that section.

I had hoped to postpone this discussion or to never have it,
but, especially given that it bears on Viruthagiri's suggestion,
let me be clear on how I feel about opening 5321/5322.   This is
just a personal opinion and I'm not trying to avoid work.
Indeed, Barry, Pete, and Alexey already have a working draft and
collection of questions about 5321bis in hand although the
questions I'm about to raise are, IMO, the more serious and
complicated ones and _not_ in that draft and set of notes.  

First, I think it is bad for the IETF and probably bad for the
Internet if we create new standards that are generally ignored.
Old standards that practices have evolved away from are
unfortunate too but, IMO, less serious.  FWIW -- and it is
somewhat more than anecdotal evidence -- sites that track
academic papers and what they reference are still telling me
that far more references are being made to 2821 than to 5321.
That is more then eleven years after the latter was published.
One could claim that use of 5321 among those who are actually
implementing and deploying SMTP is proportionally much higher,
but the substantive technical differences are few anyway. 

Second and more important, we appear to be in a situation in
which a very small number of large email providers are doing
whatever they think serves their needs and their customers well,
independent of whether or not their actions are consistent with
the standards and at least equally independent of their effects
on email users who are not their customers.  One way to look at
those actions could lead us to the conclusion that concentration
is a good thing because, if links between MTAs are adequately
encrypted, having all mail traffic move among a small number of
providers prevent those who can sniff origins and destinations
on links from determining who is sending messages to whom.  If
that is our conclusion, and we also conclude that those
providers are perfectly capable of working out interchange
requirements with each other, than maybe further IETF work on
email standards, especially basic transport (and maybe header)
standards, is a waste of time (or worse).

If, instead, we conclude that concentration is not a good thing
and that many of the ideas those providers are coming up with
are bad for the Internet (or at least for the idea of a highly
distributed and robust mail system), it appears we have a choice
between trying to hold the line on what we believe is really
good for the Internet long-term (and having the near-certainty
of being ignored) and  trying to make minor adjustments to, or
compensations for, whatever is being done with a large fraction
of email in practice so as to make the effects less bad.  The
problem isn't just with the large providers but includes other
ways in which Internet mail has evolved.  Specifically, I
suggest that there are a small number of design principles or
assumptions that underlie the SMTP architecture.  One is that
once a message starts moving, via SMTP, across the Internet, no
one alters the message content or even inspects the content to
figure out if moving it onward is a good idea until after the
message arrives at the final delivery server.  If we still
believe that, we not only would not be rewriting header fields
like "From:" in transit, we wouldn't be doing in-transit
inspection or filtering for spam or malware.  We can get around
that requirement by splitting hairs about the definition of
administrative boundaries but the question is whether we are
ready to "face reality" and modify or remove that restriction
from SMTP.

Perhaps even more fundamentally, the Internet email architecture
assumes that, if a message is sent out, it will either be
delivered as specified or a non-delivery response or message
will get back to the message originator.  A version of this is
spelled out in Section 6.1 of RFC 5321.  Concerns about blowback
attacks, disclosure of information to actual or potential
attackers, and some tricky issues involving the assumption of
equal functionality on forward and reverse paths have
significantly reduced the validity of that assumption, possibly
getting worse with each succeeding year.   Sections 6.2, 7.8,
and 7.9 allow for messages to be rejected or discarded for
various operational reasons.  However, it seems to me that, if
we are going to open 5321 and try to move a successor document
to Internet Standard, we are obligated to ask ourselves whether
discarding messages that some system decides cannot or should
not be delivered has become enough of the current practice that
those sections should to be revised to be more prescriptive
about what should or should not happen rather than simply
providing a foundation for people discarding messages without
being in technical violation of the standard.

Again this is just personal opinion but it seems to me that, if
we are going to open 5321 and 5322 at this stage, we are under
some obligation to address those questions and try to reach some
sort of consensus about what to do about them.  And, if only
from observation of comments on this list in the last few years,
I predict that won't be easy.

    john

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 26 Nov 2019 14:37:41 +1100

> As a result, he should probably be recognized for opening a can

I think there's a set of three questions which are raised by this discussion:

1. Should we ever review/revise 5321 and 5322 within the IETF?
2. If yes, should we review/revise them NOW? (for some value of)
3. If yes, where should that revision work be done?

If anybody disagrees with me on this set of questions and ordering, I'd love to hear it! I can't think of any other questions or a better ordering, but I may well be wrong.

Assuming my questions are sane, let's look at them:

*1.* I'd say there's a very strong argument that we shouldn't leave those two documents as the final form "forever". As John pointed out, there are some good justifications for revising them at some time, including:
 * citations are still more common for 2821
 * there are various errata which it would be good to address
 * the documents don't necessarily describe actual practice in the real world (not necessarily an argument for changing them, but if we don't think the best solution is "push people to adopt the precise detail of the standard" then a better alternative is publishing the document that we do want to push people to follow).
 * (significantly tongue in cheek) It would be way cool to get numbers ending with 82X again instead of 32X. Everyone knows 82X.

If you disagree with my assumption that this is enough to justify "YES" to 1, then we stop here. Otherwise let's move on.

*2.* If not now, when? Are there any conditions which mean that a time in the future would be better for doing this work than now? Some examples might be:
 * experiments in progress with outputs which will inform the new documents
 * expected changes in the userbase which will obsolete work done now
 * excessive short-term workload on the people best suited to work on or review these documents, which means they are likely to have more availability in the future.
 * (ever so slightly tongue in cheek) a bunch of people are expected to retire/die in the near future who would otherwise gum up the works

Unless one of those exists to justify a delay in the work, a YES to 1 implies we might as well do it now!

Which brings us to:

* 3. *Where? Clearly this belongs in ART. Clearly this requires a working group, this is not AD-Sponsorship level work. I'm sure the ADs will have things to say here, but I think it makes sense to spin up a specific working group to handle just this piece of work.

Regards,

Bron.

P.S. I tell a lie above, I have thought of another question which belongs somewhere in a cloud with (1) and (2), which is "should these two documents be revised as a mini-cluster, or is there a scenario in which it makes sense to just revise one and leave the other untouched"?

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 26 Nov 2019 14:45:55 +1100

On Tue, Nov 26, 2019, at 13:45, John C Klensin wrote:
> [...] I think it is bad for the IETF and probably bad for the
> unfortunate too but, IMO, less serious. 

I just wanted to answer this one by itself :)

I have a lot of sympathy for this argument. It's sounds great and is easily justifiable!

As a counter argument, I would say that to not update our guidance is an abdication of responsibility.

 If we publish a standard which is widespread use but the behaviour in the real world has moved away from what we specify, then we are misleading those who come to learn from us. The IETF has no mechanism other than "publish a new document which obsoletes the old one" to revise our guidance to the world.

5321 and 5322 are widely enough used that I believe it's our responsibility to publish our best guidance to the world, trying to capture the use cases that are causing implementations to differ from our previous standards in a way that guides everybody back into compliance.

The alternative of throwing our hands up and saying "it's just an old standard, it's OK if nobody is following it any more" leads to us becoming irrelevant over time, which IMO is more serious, as we're *clearly* the best standards body. We don't want people looking elsewhere for the best guidance on how to use the protocols we used to be the authority on!

Regards,

Bron.

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Mon, 25 Nov 2019 23:25:36 -0500

On Tue, 26 Nov 2019 14:37:41 +1100, "Bron Gondwana" said:
> P.S. I tell a lie above, I have thought of another question which belongs
> somewhere in a cloud with (1) and (2), which is "should these two documents be
> revised as a mini-cluster, or is there a scenario in which it makes sense to
> just revise one and leave the other untouched"?

If we're going to the trouble of spinning up a formal working group,
we probably *should* (non-RFC meaning :) take the time to look at both
documents, and make sure they're consistent with each other (for instance,
if we document new features/semantics for a 822-ish From: field, we
need to make sure that the 821-ish MAIL FROM: is in sync, and we may
want to add verbage to the 821-ish reply codes in explanations, etc.

We may reach a state where we've modified one document, and surprisingly
enough, the other one needs no updates. And that's OK if that's where consensus
ends up. But we should open the entire can of worms assuming that we'll
update both (I haven't checked, but I suspect that at a minimum there's
errata in both to fold in)

---

From: Dave Crocker <[email-masked]>
Date: Mon, 25 Nov 2019 20:30:46 -0800

> I think there's a set of three questions which are raised by this 

These are good process questions to end with but not necessarily good 
technical ones to start with.

If the start is in developing a list of technical changes that are 
reasonable, with some assessment of their relative importance and 
difficulty -- and maybe even charting out the details of their solutions 
-- the sense of community pressure for the work will tend to dictate the 
process answers.

That is, focus on the substance first, then work out a way to pursue it.

d/

ps. I haven't studied all the messages, so this might have already been 
mentioned, but note that a possible step short of re-doing the base 
documents is, of course, creating addenda that modify only portions.

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 26 Nov 2019 15:40:56 +1100

> technical ones to start with.

I semi-agree with you. You clipped out my three questions, but I do agree that you have raised a question (0) which I did not address. The question zero is:

*0. Are there known deficiencies with the current documents which could be addressed by revising them.*

If the answer to that is NO then obviously the others don't matter.

The rest of your email raises the very sensible ordering question:

> If the start is in developing a list of technical changes that are reasonable, with some assessment of their relative importance and difficulty -- and maybe even charting out the details of their solutions -- the sense of community pressure for the work will tend to dictate the process answers. 
> 

I guess either way the question here is "should we charter a working group to develop this list of technical challenges with a charter which allows it to revise the existing documents if that's the result it comes to, or should we do that work somewhere else and only charter a working group if the work leads us there".

And I guess the answer to that is going to be another question "how much of this list of technical changes work has already been done either offlist or in somebody's head, and is just waiting for a place to be delivered"!

I legitimately don't know the answer to that last question. But you've hit the nail on the head when it comes to assessing the ripeness of this work. Once we have a clear idea of the scope, that will indicate whether complete document revision is the wisest path forwards.

Cheers,

Bron.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 25 Nov 2019 22:05:24 -0800

> If the answer to that is NO then obviously the others don't matter.

Hmmm.  What I advised was intentionally more elaborate than your terse 
summary and that's because the really important process is in the effort 
to answer the simple question you offer. That's why I think it is 
important to focus on the substance that permits being able to answer it 
compellingly.

There is a common view that the important work of an organization like 
the IETF is producing standards, but really those documents are just 
side-effects.  Those documents fail when the real work isn't done, 
namely developing meaningful community participation that establishes 
that it wants the documents and can (and will) use them.

The substance of figuring out what work on email specs needs to be done 
is developing a community assessment that has more than a tiny number of 
well-intentioned folk, and that actually represents desire of the larger 
email community.

The fact that there is -- and, unfortunately, always has been -- so much 
ad hoc deviation from the email specs suggests an ongoing failure to 
develop and use that community adequately.  There's no comfort in the 
fact that this is not just a problem for email specs, given how much 
proprietary ad hocery there is elsewhere in the stack.

d/

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Nov 2019 03:20:12 -0500

Bron,

I'm going to respond inline to parts of your message, but, after
this, I'm going to follow my own advice and go silent until we
hear from the ADs.  All I know at this point is that I was
approached by Barry, Alexey, and Pete about a possible plan to
spin up a WG to address some mail issues including producing and
processing Internet Standard documents to replace 5321 and 5322.
I am resistant to the idea for two sets of reasons.  One set was
more or less covered by my earlier note (and I may fill in some
extra details below).  The other is that I'm fairly unhappy,
editorially, with 5321 (and 2821 before it).  When 2821 was
being created, we (I'm complicit, but it was definitely a WG
decision), to essentially merge RFCs 821 and 1869, some material
from RFC 1123, and some new material.  The main reason was that
we were concerned that, if we started rewriting, we would
unintentionally introduce discrepancies that we wouldn't have
the resources to catch.  That concern was almost certainly
justified: the one big editorial change we made was to replace
the rather basic BNF of RFC 821 with ABNF.  The change was
marked by a lot of errors, some of which were caught before IETF
Last Call and some of which made it into 2821.  While that
strategy kept the number of inadvertent changes down, the result
is a document that is hard to read and understand and that is
written in a style (or three) that is inconsistent with how we
talk about things, with one important set of examples resulting
from the 15 years between the publication of RFC 821 and that of
RFC 2119.

For the record, I agree with Dave Crocker about the importance
of focusing on substance, not process.  One reason for this is
given in a reply to part of your later note below.

--On Tuesday, November 26, 2019 14:37 +1100 Bron Gondwana
<[email-masked]> wrote:

>...
> 2. If yes, should we review/revise them NOW? (for some value
> of) 3. If yes, where should that revision work be done?
> 
> If anybody disagrees with me on this set of questions and
> ordering, I'd love to hear it! I can't think of any other
> questions or a better ordering, but I may well be wrong.
> 
> *1.* I'd say there's a very strong argument that we shouldn't
> leave those two documents as the final form "forever". As John
> pointed out, there are some good justifications for revising
> them at some time, including:

>  * citations are still more common for 2821

Actually, there are two ways to interpret that bit of data.  One
is that, were we to issue 5321bis, all of the people who have
been ignoring 5321 would suddenly pick up the new version and
start referencing it.  The other is that, if 5321 has been
ignored for more than a decade, a successor could easily be
ignored for equally long.

>  * there are various errata which it would be good to address

Having addressed many of them in that working draft I mentioned,
most are trivial.  That draft also calls out the others that
have not been rejected; some of them raise complex issues about
which we may not easily reach agreement.

>  * the documents don't necessarily describe actual practice in
> the real world (not necessarily an argument for changing them,
> but if we don't think the best solution is "push people to
> adopt the precise detail of the standard" then a better
> alternative is publishing the document that we do want to push
> people to follow).
>  * (significantly tongue in cheek) It would
> be way cool to get numbers ending with 82X again instead of
> 32X. Everyone knows 82X.

Then we had best scurry to get 982x and then try to persuade the
RFC Editor to allocate those numbers further in advance than
they have, IIR, ever done before.  Or perhaps that is
justification for sitting around for another several years :-)

> If you disagree with my assumption that this is enough to
> justify "YES" to 1, then we stop here. Otherwise let's move on.

How about a resounding "maybe" followed by "let's see what the
ADs have to say".

> *2.* If not now, when? Are there any conditions which mean
> that a time in the future would be better for doing this work
> than now? Some examples might be:  * experiments in progress
> with outputs which will inform the new documents  * expected
> changes in the userbase which will obsolete work done now  *
> excessive short-term workload on the people best suited to
> work on or review these documents, which means they are likely
> to have more availability in the future.

Well, I am personally looking forward to the IETF's solving
internationalization (i18n) before I go back to editing long
email documents.  And the strong advocates for SMTPUTF8 (RFC
6530ff) would probably like to see it broadly deployed enough
that 5321bis and 5322bis can require that extension and thereby
get ride of a lot of cruft.

  * (ever so slightly
> tongue in cheek) a bunch of people are expected to retire/die
> in the near future who would otherwise gum up the works
 
> Unless one of those exists to justify a delay in the work, a
> YES to 1 implies we might as well do it now!
 
> Which brings us to:
> 
> * 3. *Where? Clearly this belongs in ART. Clearly this
> requires a working group, this is not AD-Sponsorship level
> work. I'm sure the ADs will have things to say here, but I
> think it makes sense to spin up a specific working group to
> handle just this piece of work.

So, let me anticipate the key question.  Let's pretend this is a
BOF and ask the obvious BOF-type question: RFC 5322 is 57 pages
long.  The working version of 5321bis contains questions,
annotations, and other material that will come out so the final
version will be shorter, but it is somewhat over 100 pages long
at present.  It would be astonishing if such a working group
didn't find a few other documents it needed to look at and
either revise or write to complete the picture.  So, to make an
estimate by rounding up a bit, who is willing to commit to doing
careful reviews of each of several probable iterations of up to
200 pages of documents, reviews that including digging in,
finding and discussing problems, and proposing text?

And, finally for now (and until we hear from the ADs) and noting
that I'm about to make a process comment but still agree with
Dave about the importance of looking at substance first:

--On Tuesday, November 26, 2019 15:40 +1100 Bron Gondwana
<[email-masked]> wrote:

> You clipped out my three questions, but I do agree that you
> have raised a question (0) which I did not address. The
> question zero is:
> 
> *0. Are there known deficiencies with the current documents
> which could be addressed by revising them.*
> 

Actually, there is a 0.1, which is "whether the known
deficiencies with the current documents could be addressed, in
the IETF, without revising them."

Independent of whether it is a good idea or not, we could
address at least a significant fraction of the issues by
generating an Applicability Statement that, if necessary, did a
little bit of document updating rather than doing more or less
comprehensive replacements for the two documents.   Such an
applicability statement might reasonably include advice about
which SMTP extensions are just about mandatory at this point.
For example, there is a strong case that an SMTP implementation
that does not support the 8BITMIME extension or a mail header
implementation that does not support encoded words is even more
badly outdated in the modern world than one that restricts
local-parts to 64 octets.

best,
   john

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 27 Nov 2019 13:09:08 +0100

On Tue 26/Nov/2019 03:45:38 +0100 John C Klensin wrote:> --On Sunday, November

> I will let the ADs address this at their convenience.

Right, I more or less agree with Bron's "NOW? (for some value of)"
(RFC 9821 would seem to be due in 2024, at current speed.)

> As a result, he should probably be recognized for opening a can

Thanks for your endorsement :-)

> not a 5321bis one.

Except that Section 3.9, Mailing Lists and Aliases, for example, says:

   However, in this case, the message header section (RFC 5322 [4]) MUST
   be left unchanged; in particular, the "From" field of the header
   section is unaffected.

Changing that sentence won't break any compliance.  Rather the opposite.

That short section is, AFAIK, the main mailing list spec.  As most readers
know, the catching on of DMARC brought serious troubles to many ML, especially
after some big players set up strict policies.  A WG was set up to mitigate ML
disruption.  Conditional DKIM signatures were considered the least broken
solution.  Although there was an implementation, that solution was abandoned in
favor of a different protocol, ARC, which does not solve the problem.  In that
situation, all what ML could do was From: rewriting.  Yet, nobody took the
burden to specify it.

I agree it is probably better to specify ML in a separate document.  Then
rfc5321bis can drop Section 3.9 entirely.  That won't reduce the number of
pages very much, but still help in that quest.

> complicated ones and _not_ in that draft and set of notes.

You don't mean the yam work, do you?

> but the substantive technical differences are few anyway. 

That practice to derive a paper worthiness by the number of citations is
questionable.  In some cases, the choice of citations should determine the
worthiness of the paper at hand.

> on links from determining who is sending messages to whom.

And then the harder they come
The harder they'll fall, one and all

> If that is our conclusion, and we also conclude that those

Yeah, in that hypothesis your second conclusion is correct, unless published
standards could be useful for law making (which doesn't seem to be, AFAICS.)

> ready to "face reality" and modify or remove that restriction

Nicely said.

> being in technical violation of the standard.

The current state of the art is that new mail operators can naively encounter
situations where their mail is not delivered.  Some times, that is because
their volumes are too small to be relevant.  Some times, they happen to inherit
bad IP addresses.  Not to mention filtering inbound traffic.  It is all too
easy to outsource.  That way, the path toward a very small number of large
email providers gets larger and larger.

Email authentication should be enough to overcome the risk of backscattering,
so there should be a way to let authors know that their mail was dropped, even
for messages being forwarded.  Actually, spammers are the most diligent
implementors of authentication protocols, using a number of trash, yet real,
domains.  To avoid gaming, however, receivers are not going to inform about
quarantined messages.  Perhaps, if email authentication was taken more
seriously, this kind of crippled behavior could be overcome.

Should the standard consider a database of pen pal domains?

> I predict that won't be easy.

Surely not

Best
Ale
--

---

From: Keith Moore <[email-masked]>
Date: Wed, 27 Nov 2019 10:09:32 -0500

This is not in response to any specific message in this thread, but 
rather to the general idea of revising the email standards yet again:

The last thing I think the Internet community needs is more pages of 
email standard to sift through.

It's tempting to think that revising the standard makes things simpler 
for implementors.   My take on the situation is that it does the 
opposite.   No matter how much care is taken, every revision has the 
potential to introduce new incompatibilities that can adversely affect 
interoperation.   A diligent implementor will consider all of these when 
making implementation choices.   For instance, a mail user agent written 
in 2019 may still be expected to deal correctly with email messages 
written in the 1980s.

As far as I can tell, the RFC process in which entire documents are 
revised, and entire documents are subjected to multiple layers of 
scrutiny for each revision, is not a good process for mature and 
relatively stable specifications.   (By "relatively stable" I mean that 
a very high degree of compatibility with previous implementations is 
expected, so changes are mostly in the form of slight tweaks and 
clarifications.)   We see the problems with the RFC process in the email 
realm more than in some other realms precisely because the email 
protocol specifications are so old and have been used for so long.

And as much as I respect the effort and diligence that has been put into 
all of the revisions of RFCs 821 and 822, it's hard to escape the 
impression (in hindsight) that we would be better off if instead of the 
last N revisions, we had instead stopped revising whole documents and 
instead had a way of annotating old RFCs.     If for instance we had 
decided to stop whole-document revision at RFC2821, we'd still be using 
that document, but whenever we viewed it we'd see the original 2821 text 
along with any subsequently approved annotations, say in a different 
color or typeface.   Instead of reviewing and approving entire new 
revisions, and subjecting every revision to de novo scrutiny, we'd just 
get consensus on the annotations.   Not only would this process be 
easier for everybody: WG members, editors, IESG, the RSE, and 
implementors, it would minimize the potential for incompatibility and 
gratuitous changes.

I might even suggest that the time to permit annotations as an 
alternative to whole-document revisions is when a document has attained 
full Standard status.   It should be possible to annotate a Standard 
with relatively low effort, as long as the changes made were limited in 
scope to say clarifications and implementation advice based on 
experience.  On the other hand a significantly incompatible revision 
should still require a reset to Proposed.

And this isn't just about email.   I really wish we had the ability to 
annotate the original IP, TCP, and UDP specifications for instance.

Of course this would require thinking about RFCs quite differently than 
we do now - an annotated RFC would presumably consist of the original 
document plus some set of annotations (maybe in XML) that could be 
layered on top of the original to form some presentation of the 
composite document.   I assume that there's already some well-defined 
way to specify annotations to an XML document in XML, but a separate 
format might be needed to specify annotations to RFCs that predate use 
of XML.

Keith

---

From: Hector Santos <[email-masked]>
Date: Thu, 28 Nov 2019 10:11:34 -0500

Just throwing it out there.

IMO, iff we just focus on 5321bis, it will difficult to remain focus 
on the SMTP protocol because there will be participants, especially 
those connected with DMARC futures, that will want it considerations 
being part of the SMTP update, same with SPF and most importantly TLS 
(encryption).

Over the years, I have peppered in my comments a few times, the idea 
of producing another "hosting requirements" documents similar to the 
1989 RFC 1123:

https://tools.ietf.org/html/rfc1123
RFC1123 Requirements for Internet Hosts -- Application and Support

I always considered RFC1123 as the "holy bible" for internet hosting. 
  With developing and supporting an internet hosting platform,  this 
document put all the internet hosting protocols together. Back then, 
it was SMTP, FTP, TELNET/RLOGIN and DNS.   This was the application 
layer. There was also the common guide, RFC1122 dealing with the IP 
transport layer, TCP/UDP which also helped with our RPC client/server 
framework.  But the work concentration was with the application layer.

There was many RFCs to cover all these protocols, many questions, many 
clarifications needed. You had the available outdated RFCs, the 
mailing list and the newsgroups. I found RFC1123 was a g-dsend.  While 
I didn't participate in the DRUMS WG, my understanding was that 
RFC1123 helped jump start DRUMS with much of the SMTP material in 
RFC1123 added to RFC2821.

For me, RFC1123 was a closer to a technical specification and 
reference guide with beautiful minimum requirement tables, things 
engineers look for.  It helped with implementation productivity, lower 
production cost, higher quality and better support, "Getting it Right 
... The First Time."

The RFC format, by design, targets a wider audience, it offered a 
blend of functional and technical guidance and specification.  But too 
often, its verbosity can be subjective with marginal rough consensus. 
Quite often this is intentional in the name of leaving proposed 
methods "open ended."  But today, with decades of industy-wide 
IETF-MAN-YEARS, we don't need all the subjective verbosity.  I would 
rather to get the straight forward protocol specifications, minimum 
requirements tables, explain how all the parts fit and leave 
subjective reasoning and background information out, put into an 
appendix or moved into informational support documents, if needed.

My proposal for a modern RFC similar to the format of 1123 that 
summarizes, clarifies, codifies all the MAIL Hosting related protocols 
and BCP methods as done today.  This is an example outline:

Requirements for Internet Mail Hosts -- Application and Support

- SMTP
- POP3
- IMAP
- DNS
--- DNANE?
- EXTENSIONS
--- TLS
--- AUTH
----  SUBMIT (port 587)
--- SPF
--- DKIM
------- DKIM-BASE
------- DKIM-POLICY
----------- DMARC
--- GREYLISTING
--- OTHERS??

The above is just a quick outline. Others can provided imagine a much 
better outline, but I suggest sticking with well-established protocols 
and BCP methods with the goal to consolidate what we know today, not 
repeating the verbosity, but providing the integrated summaries of 
what makes a modern mail hosting system for interoperability at ALL 
SCALES - small to large.

RFC1123 was used to help DKUMS WG produce RFC2821.  We can use this 
new RFC as input for a new WG to produce RFC5321bis.

I know there is also new HTTP-based Mail methods, JMAP?  Maybe the 
JMAP summary work can be added to this "Internet Mail Hosting" 
requirements guide.

Of course, it may sound like a good idea, and if so, who can help 
write it.  Maybe the above could be just an starting point for what a 
RFC5321bis WG Charter could consider.

Just throwing it out there.

On 11/27/2019 10:09 AM, Keith Moore wrote:
> interoperation.   A diligent implementor will consider all of these
> when making implementation choices.   For instance, a mail user agent
> written in 2019 may still be expected to deal correctly with email
> messages written in the 1980s.
>
> relatively stable specifications.   (By "relatively stable" I mean
> that a very high degree of compatibility with previous implementations
> is expected, so changes are mostly in the form of slight tweaks and
> clarifications.)   We see the problems with the RFC process in the
> email realm more than in some other realms precisely because the email
> And as much as I respect the effort and diligence that has been put
> into all of the revisions of RFCs 821 and 822, it's hard to escape the
> impression (in hindsight) that we would be better off if instead of
> the last N revisions, we had instead stopped revising whole documents
> and instead had a way of annotating old RFCs.     If for instance we
> had decided to stop whole-document revision at RFC2821, we'd still be
> using that document, but whenever we viewed it we'd see the original
> 2821 text along with any subsequently approved annotations, say in a
> different color or typeface.   Instead of reviewing and approving
> entire new revisions, and subjecting every revision to de novo
> scrutiny, we'd just get consensus on the annotations.   Not only would
> this process be easier for everybody: WG members, editors, IESG, the
> RSE, and implementors, it would minimize the potential for
> incompatibility and gratuitous changes.
>
> alternative to whole-document revisions is when a document has
> attained full Standard status.   It should be possible to annotate a
> Standard with relatively low effort, as long as the changes made were
> limited in scope to say clarifications and implementation advice based
> on experience.  On the other hand a significantly incompatible
> revision should still require a reset to Proposed.
>
> Of course this would require thinking about RFCs quite differently
> than we do now - an annotated RFC would presumably consist of the
> original document plus some set of annotations (maybe in XML) that
> could be layered on top of the original to form some presentation of
> the composite document.   I assume that there's already some
> well-defined way to specify annotations to an XML document in XML, but
> a separate format might be needed to specify annotations to RFCs that
> predate use of XML.
>
> Keith
>
>
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

-- 
HLS

---

From: "Pete Resnick" <[email-masked]>
Date: Fri, 29 Nov 2019 11:16:39 -0600

On 26 Nov 2019, at 2:20, John C Klensin wrote:

>  All I know at this point is that I was

Since my name was mentioned, let me give you my short take on this:

Both 5321 and 5322 have been in the limbo state of "Draft Standard" for 
some time now. Not wildly important, but it bothers my sense of order.

The new xml2rfc version is now out. I wanted to play with it, so I 
converted my copy of rfc5322.xml to v3.

There were less than a dozen errata against 5322 that could be trivially 
fixed. I fixed them as part of my conversion.

It seemed like a lovely time to publish this and get 5322 to full 
"Internet Standard". Other things (e.g., MIME, utf8 mail, etc.) could be 
progressed as desired. I asked Barry and Alexey about how we might do 
this. Dealing with 5321 at the same time seemed appropriate, so Barry 
contacted John.

Note that most of the tough "should we update or not" issues are 5321 
issues, not 5322. Even the "From: rewriting" issue is a gatewaying 
issue, not a message format issue per se. All of the 5322 issues 
(including "are we really documenting existing practice) seem to be 
handled by extensions to 5322 (again, like MIME) on the format side of 
things. So I contend that 5322 is pretty much ready to go, and 5321 is 
the more interesting problem. (Glad to hear disagreements with that 
position.)

So that's (in my view) how we got here. Barry and Alexey can jump in 
whenever they are over the jet lag and general wear of IETF week (which 
I am barely over myself).

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: John C Klensin <[email-masked]>
Date: Fri, 29 Nov 2019 18:12:53 -0500

--On Friday, 29 November, 2019 11:16 -0600 Pete Resnick
<[email-masked]> wrote:

>> spin up a WG to address some mail issues including producing
>> and processing Internet Standard documents to replace 5321
>> and 5322.
> 
> Both 5321 and 5322 have been in the limbo state of "Draft
> Standard" for some time now. Not wildly important, but it
> bothers my sense of order.

Mine too, fwiw.  On the other hand, doing 5321 without 5322 or
vice versa would also bother my sense of order, perhaps worse.

> The new xml2rfc version is now out. I wanted to play with it,
> so I converted my copy of rfc5322.xml to v3.
> 
> There were less than a dozen errata against 5322 that could be
> trivially fixed. I fixed them as part of my conversion.
> 
> It seemed like a lovely time to publish this and get 5322 to
> full "Internet Standard". Other things (e.g., MIME, utf8 mail,
> etc.) could be progressed as desired. I asked Barry and Alexey
> about how we might do this. Dealing with 5321 at the same time
> seemed appropriate, so Barry contacted John.

> Note that most of the tough "should we update or not" issues
> are 5321 issues, not 5322.

Which as you know from my first response to you, Barry, and
Alexey, I also agree with.

> Even the "From: rewriting" issue is
> a gatewaying issue, not a message format issue per se.

That is less clear.  It fits into the gray area that has existed
for years about just exactly what a mailing list exploder /
redistribution system really is.  We've traditionally threaded
that needle by saying that, if the message is simply
redistributed, without messing with content (or headers other
than trace ones), then it is an SMTP matter, and that is what
5321 talks about.  If more drastic changes are needed, the story
we have told is that the mailing list mechanism is acting as the
agent for the mailing list manager and really more like a
specialized MUA.  I don't need to remind you about this, but
that it the main reason 5322 specifies "Resent-" header fields
and 5321 at least implicitly forbids true, MTA-level exploder
from messaging with them.   So, as I say, not clear.  And, fwiw,
an argument that anything rewriting a "From:" field should be
identifying itself in Resent- header fields.

So there is at least one touch issue with 5322bis even though
I'm prepared to believe it is the only one that isn't an
Applicability Statement-like issue about what we are really
recommending what people do.

> All of
> the 5322 issues (including "are we really documenting existing
> practice) seem to be handled by extensions to 5322 (again,
> like MIME) on the format side of things. So I contend that
> 5322 is pretty much ready to go, and 5321 is the more
> interesting problem. (Glad to hear disagreements with that
> position.

See above and offlist note you have already seen, while
continuing to agreed with 5321 is the more interesting case and
set of problems.  It would be even if we decided to ignore the
"paste bits of multiple documents together" versus "write
something coherent" question for which we have chosen the first
options twice and ended up with errata and complaints both times.

Back to waiting for Barry and/or Alexey.
best,
   john

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Fri, 29 Nov 2019 21:00:51 -0500

> fixed. I fixed them as part of my conversion.

How large is the pile of errata that are non-trivial fixes?

---

From: John C Klensin <[email-masked]>
Date: Fri, 29 Nov 2019 22:54:33 -0500

--On Friday, 29 November, 2019 21:00 -0500 Valdis Klētnieks
<[email-masked]> wrote:

>> There were less than a dozen errata against 5322 that could
>> be trivially fixed. I fixed them as part of my conversion.
> 

To save Pete the trouble of answering unless he thinks I'm
wrong...

For 5322 just about zero. close to zero if not zero.  I think
that is what he intended to say above, i.e., there were less
that a cozen errata and all of them could be trivially fixed.

For 5321, several of the errata raise interesting issues that
probably need to be addressed.  For some of those (a subset, but
a large one), whether the errata submitters knew it or not, the
problem is ultimately the decision we made to merge documents
rather than to try to do a complete rewrite that would preserve
the same content.  I think that decision needs to be revisited
and reviewed even while I predict we will reach the same
conclusion we reached the last two times (a decision I believe
to be pragmatic but sad).  In other cases, the problem is
evolution in how we do email in practice give various
(relatively new) stresses and what to do about that.  

  best, 
   john

---

From: "Pete Resnick" <[email-masked]>
Date: Sat, 30 Nov 2019 00:19:50 -0600

> that a cozen errata and all of them could be trivially fixed.

Correct, all existing errata were trivially fixed. (Well, some were a 
bit trickier than others, but within a standard deviation of trivial.)

pr

---

From: Hector Santos <[email-masked]>
Date: Sat, 30 Nov 2019 11:49:08 -0500

On 11/29/2019 6:12 PM, John C Klensin wrote:
> 5321 talks about.

+1.

The separation problem can be felt depending on where the transaction 
payload processing is performed - Post SMTP or Dynamic SMTP processing.

With post-smtp processing, messages are first accepted at DATA and 
from the SMTP client perspective, the transaction is complete. The 
RFC5322 payload is queued for independent processors which could be a 
MDA, a MLS (Mailing List Server), a MFA (Mail Filtering Agent) or a 
gateway to a different mail format and even network.  These post mail 
agents probably also worked (ours did) in the UUCP days and were made 
to fit with SMTP when SMTP replaced UUCICO/UUCP mail transfer methods.

Once dynamic DATA processing became feasible (due to higher processing 
power and speeds allowing for fast shiming, hooking, running milters 
at the DATA state), it was now an 5321 issue because now it 
necessitated new security-related reply codes and responses.  With 
negative responses related to restrictive DKIM Author Domain Signature 
Practices, it could cause list problems that precipitated the "what 
choice do I have, the IETF has punted on this problem" 5322.Rewrite logic.

Besides accepting the appalling suggestion "5322.From Rewriting" is 
normal (list) practice today, I don't believe it is, the problem is it 
takes a secured domain message, rewrites the From: domain to correct 
the list distribution but it now opens a security loophole for the 
original domain with replays.

-- 
HLS

---

From: Dave Crocker <[email-masked]>
Date: Sat, 30 Nov 2019 09:09:45 -0800

On 11/29/2019 3:12 PM, John C Klensin wrote:
> redistribution system really is.

So you want to move RFC 5598 to historic?

d/

---

From: John C Klensin <[email-masked]>
Date: Sat, 30 Nov 2019 14:07:42 -0500

--On Saturday, 30 November, 2019 09:09 -0800 Dave Crocker
<[email-masked]> wrote:

>> That is less clear.  It fits into the gray area that has
>> existed for years about just exactly what a mailing list
>> exploder / redistribution system really is.
> 
> 

Dave,

While, as you know, I've never been a huge fan of parts of 5598,
no I don't think moving it to historic would be helpful.  I do
think that things have gotten more complicated (or at least
different) in the decade since it was published.   I also note
that it, particularly its Section 5.3, appears to say that 

* all mailing list redistribution activity occurs at MUA level
(that is inconsistent with 5321, which came earlier and which
5598 does not claim to update or otherwise supercede) 

* RFC5322.From: should remain as set by the original author
(inconsistent with current practice on many lists and part of
where this thread started) 

IMO, the inconsistencies between the first bullet and 5321 and
between the second and tampering with "From:" to work around the
difficulties between DMARC and mailing lists are evidence of
precisely what I described as a gray area.  I don't have an
opinion at the moment as to whether 5598 should be revised (or,
in principle, retired) as part of whatever the ADs think should
be addressed in a 2019/2020 email review and update.   Forming
such an opinion would require me to study it much more carefully
(I looked at it today for the first time in years) as well as
understanding what we actually intend to do with 5321 and 5322.

best,
   john