Subject: [ietf-smtp] Curious, with this now being associated to emailcore, should list name change?

From: Michael Peddemors <[email-masked]>
Date: Fri, 17 Jul 2020 14:57:08 -0700

Not sure what normally happens, but it might be confusing.

"Email Core" would have a wider scope, and it might be confusing if the 
list name was limited to 'smtp'.

Suggestion for topic for this group as well:

Unifying all the 'autodiscover' and 'autoconfig' methods currently in 
place.. email client developers have now a very convoluted set of 
requirements in order to find the 'recommended' settings for that domain 
or ISP etc..

There are several independent databases out there, eg Apple's own, the 
ISPDB, and even some of Microsofts' own email clients no longer follow 
traditional methods of lookups.. It is a bit of a mess, that maybe the 
IETF would like to weigh in on?

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: John C Klensin <[email-masked]>
Date: Fri, 17 Jul 2020 18:13:13 -0400

> Not sure what normally happens, but it might be confusing.

Independent of "normal", the name and mailing address of this
list is known by email developers and operators all over the
Internet.  It also consolidates some prior lists specifically
associated with mail headers, MIME, and non-ASCII addresses and
headers (and maybe others, probably including the lists for the
DRUMS and YAM WGs).   Changing its name (effectively killing the
list and starting another) would be disruptive in the extreme.

Perhaps "emailcore" should be given a list of its own, but I
think that would not be helpful either.
 
> "Email Core" would have a wider scope, and it might be
> confusing if the list name was limited to 'smtp'.

Consider it a historical artifact and, like WG names (and
corresponding mailing list) that are chosen more for cuteness
than actual semantic value, accept it and move forward.

Please.

I will leave it to the BOF Chairs and/or ADs to comment on the
rest of this but my understanding is that they want to keep the
scope of "emailcore" as narrow as possible, at least initially,
rather than having it expand into "any email topic that would be
worth addressing".

Speaking only for myself, I note that the IETF has tried very
hard over the years to stay out of MUA design and issues.
Perhaps it is time to change that and take on at least some MUA
requirements (work is badly needed, IMO, in the non-ASCII
addresses and header space although I don't know if the IETF as
the right expertise to do it) but it would be a rather large
step.

> Unifying all the 'autodiscover' and 'autoconfig' methods
> currently in place.. email client developers have now a very
> convoluted set of requirements in order to find the
> 'recommended' settings for that domain or ISP etc..
> 
> There are several independent databases out there, eg Apple's
> own, the ISPDB, and even some of Microsofts' own email clients
> no longer follow traditional methods of lookups.. It is a bit
> of a mess, that maybe the IETF would like to weigh in on?

best,
   john

---

From: E Sam <[email-masked]>
Date: Sat, 18 Jul 2020 14:55:13 -0400

Hello all,

I'm out of the loop of the plans for the (future?) emailcore working group

Any links where I can catch up and read more about this before the
IETF 108 meeting (if I can make it of course)

YES i DuckDuckGoed some information about it but I am still out of the
loop a little bit

Thank you all

On Fri, Jul 17, 2020 at 6:13 PM John C Klensin <[email-masked]> wrote:
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: John C Klensin <[email-masked]>
Date: Sat, 18 Jul 2020 15:15:38 -0400

You may want to have a look at the draft agenda, now posted at
https://www.ietf.org/proceedings/108/agenda/agenda-108-emailcore-00,
and noting that the example topics for rfc5321bis are references
to Appendix G of draft-klensin-rfc5321bis-03 (the agenda sort of
says that, but, IMO, it is easy to miss).  It is vaguely
possible that a new version of the I-D will be posted before the
BOF meets, but it is thoroughly unlikely that Appendix
names/numbers will change.

Alexey or Seth may be able to add to that, especially if you
have specific questions.

Beyond the agenda, "emailcore" seems to be a title or framework
for a lot of different ideas that may not have focused or
converged yet.  The purpose of a BOF is to move that process
along.

best,
  john

--On Saturday, July 18, 2020 14:55 -0400 E Sam
<[email-masked]> wrote:

> Any links where I can catch up and read more about this before
> the IETF 108 meeting (if I can make it of course)
> 
> YES i DuckDuckGoed some information about it but I am still
> out of the loop a little bit
> 
> Thank you all
> 
>> associated with mail headers, MIME, and non-ASCII addresses
>> and headers (and maybe others, probably including the lists
>> for the DRUMS and YAM WGs).   Changing its name (effectively
>> killing the list and starting another) would be disruptive in
>> the extreme.
>> 
>> rest of this but my understanding is that they want to keep
>> the scope of "emailcore" as narrow as possible, at least
>> initially, rather than having it expand into "any email topic
>> that would be worth addressing".
>> 
>> Perhaps it is time to change that and take on at least some
>> MUA requirements (work is badly needed, IMO, in the non-ASCII
>> addresses and header space although I don't know if the IETF
>> as the right expertise to do it) but it would be a rather
>> large step.
>> 
>> 
>> 
>> > There are several independent databases out there, eg
>> > Apple's own, the ISPDB, and even some of Microsofts' own
>> > email clients no longer follow traditional methods of
>> > lookups.. It is a bit of a mess, that maybe the IETF would
>> > like to weigh in on?
>> 
>> best,
>>    john
>>

---

From: E Sam <[email-masked]>
Date: Sat, 18 Jul 2020 18:56:11 -0400

I think the max size of the header name does need to be addressed in RFC 5322.
Seeing how email and Usenet can sometimes be the "wild west" I have
seen really long headers and developing mail software the question of
header max limits have also come into my mind.

The other issues on the list are important too but this one is one
that I have personally dealt my hands on.

I look forward to seeing what comes out of the BOF.

Best,
 Sam

---

From: John C Klensin <[email-masked]>
Date: Sat, 18 Jul 2020 20:57:53 -0400

--On Saturday, July 18, 2020 18:56 -0400 E Sam
<[email-masked]> wrote:

> I think the max size of the header name does need to be
> addressed in RFC 5322. Seeing how email and Usenet can
> sometimes be the "wild west" I have seen really long headers
> and developing mail software the question of header max limits
> have also come into my mind.

Out of curiosity and just to be precise, are you talking about
the header field name or the field value?

> The other issues on the list are important too but this one is
> one that I have personally dealt my hands on.

I trust Pete and the co-chairs will take note.

> I look forward to seeing what comes out of the BOF.

Time zones, etc., permitting, please try to participate.  Much
of the BOF is precisely about deciding what is in or out of
scope.

best,
   john

---

From: E Sam <[email-masked]>
Date: Sat, 18 Jul 2020 22:04:32 -0400

> the header field name or the field value?

I was originally talking about the header field name, but I think the
field value could use some addressing. I once saw a spam email in my
inbox which 95 percent of the actual email was an extremely long
header value.

All the big email providers like Gmail and new email providers with
considerable leverage like hey.com and Protonmail keep their header
value sizes to reasonable amounts so I don't think its as important as
the header field.

The problem with header value limits is the length of some headers are
deterministic (like the Path header in Usenet, X-Beenthere in mailing
lists, and to an extent dkim signatures).

I think a header value limit for important headers stored in overview
systems (From:, To:, Newsgroups:, Bcc:, etc...) could be a good
idea... this could be difficult but it is something that we could
consider - I don't know if there is already something out there that
already sets the rules for this and I don't know about it. SMTP
software developers  - anyone from EXIM, Sendmail, OpenSMTPD, or
Postfix on this list can elaborate if they want in my reply.

> of the BOF is precisely about deciding what is in or out of

I'll try my best. Would I see the meeting link here or will I see it
after I purchase my registration to IETF 108?

best,
 sam

---

From: "John Levine" <[email-masked]>
Date: 18 Jul 2020 23:26:41 -0400

>inbox which 95 percent of the actual email was an extremely long

I don't understand the problem. Spammers will send all sorts of junk
which we need to defend against no matter what since they will ignore
the specs.

Each line is limited to 998 octets plus CR-LF, but headers can be
arbitrarily long and folded at spaces. These limits date back to RFC
780 forty years ago so it seems kind of late to try to change them now.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 19 Jul 2020 02:19:02 -0400

--On Saturday, July 18, 2020 22:04 -0400 E Sam
<[email-masked]> wrote:

> I'll try my best. Would I see the meeting link here or will I
> see it after I purchase my registration to IETF 108?

The IETF isn't nearly as good at keeping secrets as your
question suggests might be possible.  The main meeting page
(which you will probably have to go to in order to pay or ask
for a waiver) is at https://www.ietf.org/how/meetings/108/ .
Under "Agendas and Meeting Materials", you will find links to
the agenda.  To save you navigating that if it is all you need,
there is an agenda at
https://datatracker.ietf.org/meeting/108/agenda/.  If you scroll
it down to find the emailcore entry.  Clicking on "emailcore"
will get the proposed charger, clicking on " 	Revision of core
Email specifications" will get the current agenda, and the
little row of icons to the right include the meeting link as the
next-to-last entry ("Meetecho video stream").  

All pretty straightforward once one gets used to it.  Getting
used to it is another matter but just keep asking questions and
you will find that many people are willing to help.

best,
   john

---

From: Jeremy Harris <[email-masked]>
Date: Sun, 19 Jul 2020 11:18:39 +0100

> Postfix on this list can elaborate if they want in my reply.

Exim limits the aggregate size of the headers of a message; this
implicitly limits any single header.

The limit is changeable in the configuration file; the default
is 1MB.
-- 
Cheers,
  Jeremy

---

From: "John Levine" <[email-masked]>
Date: 19 Jul 2020 10:43:57 -0400

In article <CAKFo7wk+jLGqjs6mU=[email-masked]> you write:
>idea... 

There's no formal limit other than the line length, but there is a registry of 
header names, shared among mail, netnews and uucp:

https://www.iana.org/assignments/message-headers/message-headers.xhtml

Adding new names to the registry requires expert review and I expect
the expert would be sceptical of proposed names that were a lot longer
than the existing ones. I don't see any over 35 characters which seems
reasonable.

R's,
John

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Jul 2020 18:57:10 +0200

> than the existing ones. I don't see any over 35 characters which seems

My personal server currently has 559 field names longer than that. The 25 
worst offenders:

 X-Offlineimap-X706593913-6d61646475636b2e6e6574-494e424f582e647261667473
 X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner-Information
 X-Ms-Exchange-Crosstenant-Originalattributedtenantconnectingip
 X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner-Spamcheck
 X-Ssjmail.Ssjfinance.Com-Mail-Server-Mailscanner-Information
 Staticcontent1_Header1_F731d3dc-Fd31-4161-Ad91-1083ba56853f
 Staticcontent2_Header2_F731d3dc-Fd31-4161-Ad91-1083ba56853f
 X-Mimedefang-Relay-15b21d6f94afe8e768c451e09085c007047aae7e
 X-Mimedefang-Relay-89167b66339720c294cd81d33948afd6488b114f
 X-Ssjmail.Ssjfinance.Com-Mail-Server-Mailscanner-Spamcheck
 X-Ssjmail.Ssjfinance.Com-Mail-Server-Mailscanner-Spamscore
 X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner-From
 X-Mailscan-242.Hostingdynamo.Net-Mailscanner-Information
 X-Content-Pgp-Universal-Saved-Content-Transfer-Encoding
 X-Kypusserverappliance-Kypus-Mailprotection-Information
 X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner-Id
 X-Mailscan-242.Hostingdynamo.Net-Mailscanner-Spamscore
 X-Mailer.Unfpa-Bangladesh.Org-Mailscanner-Information
 X-Nugget-Enterprises-Antispam-Mailscanner-Information
 X-Ssjmail.Ssjfinance.Com-Mail-Server-Mailscanner-From
 X-Bangladesh-Open-University-Mailscanner-Information
 X-Ironport.Danmargroup.Co.Za-Mailscanner-Information
 X-Mail01lehostingservicesnet-Mailscanner-Information
 X-First-Flight-Couriers-Ltd-Mailscanner-Information
 X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner

Arnt

---

From: "John Levine" <[email-masked]>
Date: 19 Jul 2020 13:46:30 -0400

In article <[email-masked]> you write:
> ...

Right, that's what I meant when I said that no matter what the specs say,
we have to defend against junk.  Those don't look particularly spammy,
just clueless, but we have to deal with them anyway.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 19 Jul 2020 14:19:52 -0400

--On Sunday, July 19, 2020 18:57 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

>> Adding new names to the registry requires expert review and I
>> expect the expert would be sceptical of proposed names that
>> were a lot longer than the existing ones. I don't see any
>> over 35 characters which seems reasonable.
> 
> My personal server currently has 559 field names longer than
> that. The 25 worst offenders:
> 

<Sounds of virtual vomiting>

It is interesting that every one of these starts in "X-".

Presumably, by putting "X-" in front of their field names, the
perpetrators believe that they are exempt, not only from the
registry and its rules, but any rules at all.  That would be my
concern here: maybe it would be worth getting a stake in the
ground that long, or very, long field names are a bad idea
and/or a bigger stake in the ground about "X-" prefixes.  Unless
we accompany it with a really strong, clear, and persuasive
discussion of the problems it causes, which might help, I wonder
if the offenders would pay any attention at all?  

The rest of this note is more or less a rejection of the notion
that because there are no protocol police, we can't do anything
(or anything useful), with some questions, speculation, and
examples about what might be done.  I haven't made my mind up
yet whether doing something would be worthwhile; I just want to
get past the idea of impossibility.  Those who aren't interested
in that speculation, or who would prefer to leave it for the BOF
or WG, should probably stop reading here.

    -----

Perhaps more important than what the perpetrators do today,
user-defined-fields using "X-" were deprecated by RFC 2822
nineteen years ago, with that message reinforced by RFC 6648
more than eight years ago.  Much as, on principle, I hate the
idea of intermediate MTAs messing with non-trace headers, would
relays or anti-malware systems consider just deleting the things
as potentially-dangerous noise?  Do they check now for embedded
nastiness in header field data as well as body parts?   Should
we be encouraging Submission Servers to discard them (there is
enough flexibility in RFC 6409 to allow that and, given that
such servers are not supposed to inject non-conforming stuff
into the public Internet SMTP system, our banning them would
make a Submission Server that let them pass non-conforming)? How
about the interface between the mailstore and receiving MUAs:
could these things be tossed out there?  Would any of that help
and would our saying something even stronger than what RFC 6648
says (it mostly applies to new header field names, not older
ones, but I'd guess the vast majority of these were invented
after June 2012 and almost certainly after April 2001) be
useful?  Even if the fields are, as John Levine points out,
probably just clueless rather than spammy, it is probably not a
wonderful idea to encourage noise in mail headers, those whose
intent is malicious are likely to uncover opportunities sooner
or later, and perhaps (just perhaps) people getting a little
proactive now might prevent or head off future attacks.

There aren't any protocol police, but there are certainly
implementers who can decide, especially if we make a strong
recommendation in that area to help them defend against angry
folks who produce this stuff, to not tolerate this particular
type of nonsense any more.  Should we register a
"Deleted-NonConforming-Headers:" field, treat it as an optional
trace field with "By" and timestamp information, and then have
the rest of the value showing either the number of such fields
tossed out or the header field names (but not values)?

Would the suggestion that started this thread be consistent with
an applicability statement, either in 2822bis or the separate
document for which draft-klensin-email-core-as-00 is a
placeholder, that said something like "Header field names
starting in 'X-' or longer than NNN are prohibited unless they
are legacy registrations in the IANA registry.  They MUST NOT be
transmitted in messages and Submission Servers and other systems
encountering them MAY, and in most cases SHOULD, delete them"? 

At a whole different level, at least several of these examples
appear to be loop detection tools.  Should we be taking another
look at the loop detection support machinery in 5321 (IIR,
although there have been some clarifications and details, that
machinery is not significantly changed since RFC 821) either as
part of the proposed emailcore effort or separately?

Again, I don't yet have an opinion as to whether we should take
those, or other, actions on this topic.  I'm just wondering what
actions might be effective if we did decide to take them.

best,
   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Jul 2020 21:10:33 +0200

> It is interesting that every one of these starts in "X-".

It looks shorter, but uglier, if I exclude X-. Avert your eyes now:

 Staticcontent1_Header1_F731d3dc-Fd31-4161-Ad91-1083ba56853f
 Staticcontent2_Header2_F731d3dc-Fd31-4161-Ad91-1083ba56853f
 {Emails_Custom_Header_Prefix}Subscriber-Uid
 {Emails_Custom_Header_Prefix}Campaign-Uid
 {Emails_Custom_Header_Prefix}Customer-Gid
 {Emails_Custom_Header_Prefix}Customer-Uid
 {Emails_Custom_Header_Prefix}Delivery-Sid
 {Emails_Custom_Header_Prefix}Tracking-Did
 /Usr/Usr/Lib/Ecartis/Ecartis.Logreceived
 /Usr/Local/Ecartis/Ecartis.Logreceived
 Antivirus.Cortanet.Com.Mx-Mail-From
 {Emails_Custom_Header_Prefix}Mailer
 (Anti-Spam-(Anti-Spam-Mime-Version
 Lcs-Mailantivirus-System-Mail-From
 {Emails_Custom_Header_Prefix}Ebs
 Prayed-Neutralize-Impressionable
 Authentication-Results-Original
 Original-Authentication-Results
 Regenerates-Root-Insensitivity
 Discarded-X400-Mts-Extensions
 Require-Recipient-Valid-Since
 Academia-Accosts-Occidentals
 Forward-Confirmed-Reversedns
 Sun-Java-System-Smtp-Warning
 Antivirus-Sistema-Mail-From

Some might say that these are the worst even including X-. I, uh, I don't 
know what to say, but maybe doing these queries may not have been a good 
idea and you should shoot me now. "Authentication-Results-Original 
Original-Authentication-Results" oh god.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Sun, 19 Jul 2020 15:45:02 -0400

--On Sunday, July 19, 2020 21:10 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> Some might say that these are the worst even including X-. I,
> uh, I don't know what to say, but maybe doing these queries
> may not have been a good idea and you should shoot me now.
> "Authentication-Results-Original

No, I think the data are always helpful.  So much for my guess
that we could address the problem by going after the "X-" stuff.
That also may suggest a review of whether deprecating the use of
"X-" and substituting a registration requirement with expert
review just encourages name-squatting without any
warning/indication.

   john

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 20 Jul 2020 09:51:00 +0200

On Sun 19/Jul/2020 20:19:52 +0200 John C Klensin wrote:
> Would the suggestion that started this thread be consistent with an
> applicability statement, either in 2822bis or the separate document for
> which draft-klensin-email-core-as-00 is a placeholder, that said something
> like "Header field names starting in 'X-' or longer than NNN are prohibited
> unless they are legacy registrations in the IANA registry.  They MUST NOT
> be transmitted in messages and Submission Servers and other systems 

Such a drastic stance may make sense if we need to guard against some kind of 
violation.  For example, header fields which aim at capturing some traits of 
the body of a message which are considered relevant for tracking, profiling, or 
similar activities which might be considered intrusive.  Such fields can allow 
a mail system to link non-delivery notifications and DMARC/ DKIM/ SPF failure 
notices which contain text/rfc822-headers only to their missing bodies.  That 
way, they'd defeat the omission of the body due to privacy concerns.

Just ruminating...
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 20 Jul 2020 14:27:00 -0400

In article <B7E061A14E80279E1E14D92F@PSB> you write:
>registry and its rules, but any rules at all. ...

No, they heard somewhere that you put x- in front of header names you
make up and the only registry they know about is the one in MS
Windows.

I did a similar sweep through my mail archive and found 1388 different
headers, all but 144 of which start with x-. Here's the longest ones,
all of which thoughtfully tell who to blame:

x-ms-exchange-crosstenant-originalattributedtenantconnectingip
x-ms-exchange-crosstenant-rms-persistedconsumerorg
x-ms-exchange-transport-crosstenantheadersstamped
x-ms-exchange-transport-crosstenantheadersstripped
x-white-heron-it-services-mailscanner-information
x-white-heron-it-services-mailscanner-spamscore
x-white-heron-it-services-mailscanner-watermark

My experience with people from Microsoft is that they are plenty smart
but often have no idea what they don't know. It might be productive to
make a few queries and see if we can encourage them to register them a
provisional names so at least there's less chance of collision.

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 20 Jul 2020 14:47:17 -0700

Given that so many headers are out there, that are not on any registry 
list, other than name space collisions, what other reasons are out there 
that can help encourage registration..

And BTW, we add some X-headers, specific to our email platforms, that 
are only meant to be added for MTA->MTA final destination, rather than 
headers that are being inserted on outbound traffic..

Those in theory don't need the same level of registration, correct? 
Comments? Back when we first started our MagicMail platforms, the 
recommendation was to use X- headers, however that recommendation has 
been countered since.. we left them as X- headers, since it was more 
backwards compatible, trying to see an argument why we should change now 
after 18 years..

If we see these headers on inbound traffic, they are 'stripped' as we 
know they shouldn't really exist in the wild in MTA->MTA traffic, and 
they no longer can be trusted..

Most are for internal use, eg whether the content was detected as spammy 
by the SMTP engines etc, remote MTA's operating system and other 
identifiers or tools for metrics, they all start with X-MagicMail to 
make it easy to protect it as a named space, eg we know they only should 
be insert on acceptance by our MTA's

---

From: "John Levine" <[email-masked]>
Date: 20 Jul 2020 22:07:36 -0400

>they no longer can be trusted..

That's probably OK. I can sort of imagine a situtation where you have
a customer who is annoyed at you because he was using the same header
names for something that a friend sends him and and those headers
always disappear.

If they start with something like x-linuxmagic- or just linuxmagix-
I'd say you can ignore chance of innocent collision.

R's,
John

---

From: Dilyan Palauzov <[email-masked]>
Date: Tue, 21 Jul 2020 07:37:49 +0000

Hello,

my reading is that this thread questions the usefulness of the X-  
headers, spread over the internet.  The message below contains:

X-headers are useful to insert information about what and why  
intermediate hosts think about the spam probability of a message.   
They can prevent manual communications on this.  Unfortunately the  
above X-Spam headers do not include information about which host added  
that headers.

As useless mail headers do make emails heavier, I am in favour of  
removing DKIM-Signature headers, that are known to be broken, e.g.  
because the current host has modified (and resubmitted) the message.   
Or if not completely removed, then at least shortened by substituting  
to “b=invalided” or “b=invalidated-on-host-A.B”.  The latter is more  
useful than just having an invalid dkim-signature. (or removing the  
b=/bh= tags and putting instead a new tag containing the host where  
the signature was broken, which not really an Authenticated Receiver  
Chain, and does make the massage shorter).

Greetings
   Dilyan

----- Message from John C Klensin <[email-masked]> ---------
    Date: Sun, 19 Jul 2020 14:19:52 -0400
    From: John C Klensin <[email-masked]>
Subject: Re: [ietf-smtp] Curious, with this now being associated to  
emailcore, should list name change?
      To: Arnt Gulbrandsen <[email-masked]>, [email-masked]

----- End message from John C Klensin <[email-masked]> -----

---

From: Michael Richardson <[email-masked]>
Date: Tue, 21 Jul 2020 08:55:01 -0400

Dilyan Palauzov <[email-masked]> wrote:
    > As useless mail headers do make emails heavier, I am in favour of  removing
    > DKIM-Signature headers, that are known to be broken, e.g.  because the
    > current host has modified (and resubmitted) the message.   Or if not
    > completely removed, then at least shortened by substituting  to “b=invalided”
    > or “b=invalidated-on-host-A.B”.  The latter is more  useful than just having
    > an invalid dkim-signature. (or removing the  b=/bh= tags and putting instead
    > a new tag containing the host where  the signature was broken, which not
    > really an Authenticated Receiver  Chain, and does make the massage shorter).

"I, host-A.B, (failed-to) validated the DKIM header from C.D, but I removed it when I
broke the message"

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Hector Santos <[email-masked]>
Date: Tue, 21 Jul 2020 12:10:07 -0400

Microsoft and Windows does not have a copyright with "X-" junk headers.

Junk is generated from everywhere, including from the *nix weenies 
camp & especially from Google and the IETF creating protocols with 
useless overhead.  You, yourself, have added a new "X-" header with 
"X-Original-From" while making 5322.From useless to fix a mess you are 
partly responsible for where more DKIM signatures and related patch 
work headers are expected to be needed.  Let just wait for ARC per 
node wasted overhead to be widely spilled over into the email network. 
  I already have plans for it but not what you think:

Today, the majority of headers are useless information to the MDA. 
The most important are:

Date:
From:
To:
Subject:

Just like it has always has been since the beginning of electronic 
communications time.

Networking headers like Reply-To: are needed too.  MIME headers will 
be needed for rendering.

For the record, our mail software, "older than dirt," has two modes, 
one to keep a raw "as is" import storage and one to import/convert 
into a proprietary format where unnecessary RFC5322 headers are 
pruned. Once mail is received by the MDA, validated, spam or 
otherwise, all the extra overhead is not needed.

-- 
Hector Santos,
https://secure.santronics.com
https://twitter.com/hectorsantos

On 7/20/2020 2:27 PM, John Levine wrote:
> https://twitter.com/hectorsantos

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 12:13:56 -0400

Dilyan,

It is a bit more than "questions the usefulness of".  Please see
the explanation in RFC 6648.  I'll let others respond to the
DKIM header suggestion.

best,
   john

--On Tuesday, 21 July, 2020 07:37 +0000 Dilyan Palauzov
<[email-masked]> wrote:

> Hello,
> 
> my reading is that this thread questions the usefulness of the
> X- headers, spread over the internet.  The message below
> contains:
> 
> Received: from localhost (localhost [127.0.0.1])
>   by ietfa.amsl.com (Postfix) with ESMTP id 6B1623A08A2
>   for <[email-masked]>; Sun, 19 Jul 2020 11:20:01
> -0700 (PDT)
> X-Spam-Flag: NO
> X-Spam-Score: -1.897
> X-Spam-Level:
> X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5
>   tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001,
>   URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
> Received: from mail.ietf.org ([4.31.198.44])
>   by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port
> 10024)
>   with ESMTP id g-1uauMMW0n1 for <[email-masked]>;
>   Sun, 19 Jul 2020 11:20:00 -0700 (PDT)
> 
> intermediate hosts think about the spam probability of a
> message.  They can prevent manual communications on this.
> Unfortunately the above X-Spam headers do not include
> information about which host added that headers.
> 
> As useless mail headers do make emails heavier, I am in favour
> of removing DKIM-Signature headers, that are known to be
> broken, e.g. because the current host has modified (and
> resubmitted) the message.  Or if not completely removed, then
> at least shortened by substituting to "b=invalided" or
> "b=invalidated-on-host-A.B".  The latter is more useful
> than just having an invalid dkim-signature. (or removing the
> b=/bh= tags and putting instead a new tag containing the host
> where the signature was broken, which not really an
> Authenticated Receiver Chain, and does make the massage
> shorter).
> 
> Greetings
>    Dilyan
> 
> Subject: Re: [ietf-smtp] Curious, with this now being
> associated to emailcore, should list name change?
>>>> Adding new names to the registry requires expert review and
>>>> I expect the expert would be sceptical of proposed names
>>>> that were a lot longer than the existing ones. I don't see
>>>> any over 35 characters which seems reasonable.
>>> 
>>>  X-Offlineimap-X706593913-6d61646475636b2e6e6574-494e424f582
>>>  e6 47261667473
>>>  X-Gemstorm-Computing-T/A-The-It-Company-Mailscanner-Informa
>>>  ti on
>> registry and its rules, but any rules at all.  That would be
>> my concern here: maybe it would be worth getting a stake in
>> the ground that long, or very, long field names are a bad idea
>> and/or a bigger stake in the ground about "X-" prefixes.
>> Unless we accompany it with a really strong, clear, and
>> persuasive discussion of the problems it causes, which might
>> help, I wonder if the offenders would pay any attention at
>> all?
>> 
>> The rest of this note is more or less a rejection of the
>> notion that because there are no protocol police, we can't do
>> anything (or anything useful), with some questions,
>> speculation, and examples about what might be done.  I
>> haven't made my mind up yet whether doing something would be
>> worthwhile; I just want to get past the idea of
>> impossibility.  Those who aren't interested in that
>> speculation, or who would prefer to leave it for the BOF or
>> WG, should probably stop reading here.
>> 
>>     -----
>> 
>> idea of intermediate MTAs messing with non-trace headers,
>> would relays or anti-malware systems consider just deleting
>> the things as potentially-dangerous noise?  Do they check now
>> for embedded nastiness in header field data as well as body
>> parts?   Should we be encouraging Submission Servers to
>> discard them (there is enough flexibility in RFC 6409 to
>> allow that and, given that such servers are not supposed to
>> inject non-conforming stuff into the public Internet SMTP
>> system, our banning them would make a Submission Server that
>> let them pass non-conforming)? How about the interface
>> between the mailstore and receiving MUAs: could these things
>> be tossed out there?  Would any of that help and would our
>> saying something even stronger than what RFC 6648 says (it
>> mostly applies to new header field names, not older ones, but
>> I'd guess the vast majority of these were invented after June
>> 2012 and almost certainly after April 2001) be useful?  Even
>> if the fields are, as John Levine points out, probably just
>> clueless rather than spammy, it is probably not a wonderful
>> idea to encourage noise in mail headers, those whose intent
>> is malicious are likely to uncover opportunities sooner or
>> later, and perhaps (just perhaps) people getting a little
>> "Deleted-NonConforming-Headers:" field, treat it as an
>> optional trace field with "By" and timestamp information, and
>> then have the rest of the value showing either the number of
>> such fields tossed out or the header field names (but not
>> values)?
>> 
>> Would the suggestion that started this thread be consistent
>> with an applicability statement, either in 2822bis or the
>> separate document for which draft-klensin-email-core-as-00 is
>> a placeholder, that said something like "Header field names
>> are legacy registrations in the IANA registry.  They MUST NOT
>> be transmitted in messages and Submission Servers and other
>> systems encountering them MAY, and in most cases SHOULD,
>> delete them"?
>> 
>> appear to be loop detection tools.  Should we be taking
>> another look at the loop detection support machinery in 5321
>> (IIR, although there have been some clarifications and
>> details, that machinery is not significantly changed since
>> RFC 821) either as part of the proposed emailcore effort or
>> separately?
>> 
>> Again, I don't yet have an opinion as to whether we should
>> take those, or other, actions on this topic.  I'm just
>> wondering what actions might be effective if we did decide to
>> take them.
>> 
>> best,
>>    john
>> 
>>

---

From: Hector Santos <[email-masked]>
Date: Tue, 21 Jul 2020 12:40:14 -0400

> Chain, and does make the massage shorter).

See my last post on this.

My mail software has been pruning useless headers since the 80s when 
we began to import RFC822 mail.  It has grown tremendously ever since 
and its gotten worst.

The idea of "Who cares?" that RFC5322 meta-header waste is widely 
acceptable, is part of the problem.

RFC5322 Header Pruning SHOULD be a new common practice to be evaluated.

Among others, DKIM-Signature is certainly a prune candidate once 
validation has taken place a the MDA, i.e. transports have ended.

---

From: "John Levine" <[email-masked]>
Date: 21 Jul 2020 16:19:38 -0400

>because the current host has modified (and resubmitted) the message.   

The amount of bandwidth used by e-mail is a rounding error of the
Internet's total, which is mostly video these dayts, and the amount
used by broken headers is a rounding error on that rounding error.

Look at the headers of the mail in your inbox, particularly mail from
large providers, and you'll find megabytes of headers that nobody is
ever likely to look at or use.  This battle was over decades ago.

R's,
John

---

From: Brandon Long <[email-masked]>
Date: Tue, 21 Jul 2020 13:35:29 -0700

> ever likely to look at or use.  This battle was over decades ago.

A better (but annoying) reason is there are a smattering of servers which
reject messages
based on broken DKIM signatures, against the rfc.

Brandon

---

From: "John R Levine" <[email-masked]>
Date: 21 Jul 2020 16:38:01 -0400

> reject messages based on broken DKIM signatures, against the rfc.

Is that still the case?  Wow, brokenness never ceases.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Hector Santos <[email-masked]>
Date: Tue, 21 Jul 2020 16:45:19 -0400

> ever likely to look at or use.  This battle was over decades ago.

I heard this stated by you a few times. I never brought into it.  It 
wasn't a problem to be concern about a few decades ago.  Now it is. 
It's junk and its growing.  The more it grows the more overhead we 
have. It makes systems who are suppose to be ignorant about the junk, 
work harder.  It makes your "View Source" more cryptic.

Simply put, for the MDA/MUA it is overhead not required. It can be 
pruned by the MDA passing it on to a MUA.

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 16:48:25 -0400

John,

While I mostly agree, we should not lose sight of the fact that
there still people using the Internet in less-developed
countries, isolated locations, over intermittent connections,
etc., where bandwidth tends to be scarce and expensive
Whatever they are using it for, it probably isn't dominated by
high resolution video and, assuming we care about them, saying
"well, it doesn't make a lot of difference to those who have
essentially unlimited bandwidth and who are streaming lots of
video, so no one and nothing else else counts" seems a
unfortunate to me.

In addition, if a possibly-legitimate, possibly-phishing,
message comes along, I'd really like to see us (and our MUAs)
encourage users to look at headers, understand what can be
trusted and what can't, and so on.  Perhaps the educational
effort that would require is hopeless under the best of
circumstances, but it is certainly impossible if we facilitate
and encourage an ever-rising collection of noise header fields.

  john

--On Tuesday, 21 July, 2020 16:19 -0400 John Levine
<[email-masked]> wrote:

> In article
> <[email-masked]
> g> you write:
>> As useless mail headers do make emails heavier, I am in
>> favour of   removing DKIM-Signature headers, that are known
>> to be broken, e.g.   because the current host has modified
>> (and resubmitted) the message.   
> 
> The amount of bandwidth used by e-mail is a rounding error of
> the Internet's total, which is mostly video these dayts, and
> the amount used by broken headers is a rounding error on that
> rounding error.
> 
> Look at the headers of the mail in your inbox, particularly
> mail from large providers, and you'll find megabytes of
> headers that nobody is ever likely to look at or use.  This
> battle was over decades ago.
> 
> R's,
> John
>

---

From: "Pete Resnick" <[email-masked]>
Date: Tue, 21 Jul 2020 16:15:34 -0500

On 19 Jul 2020, at 11:57, Arnt Gulbrandsen wrote:

> My personal server currently has 559 field names longer than that. The 
> 25 worst offenders:
>

Other than X- field names not doing what people think they're doing, I 
don't see the problem here. None of them are over 77 characters, and 
none (including the ones you showed later with curly braces in them) are 
using problematic characters. So what's the problem?

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: "Pete Resnick" <[email-masked]>
Date: Tue, 21 Jul 2020 16:18:07 -0500

> intermediate hosts think about the spam probability of a message.

But why start them with "X-"? Why not just "Spam-Flag" and "Spam-Score", 
etc.?

pr

---

From: Paul Smith <[email-masked]>
Date: Tue, 21 Jul 2020 22:41:41 +0100

> work harder.  It makes your "View Source" more cryptic. 

Hmm, my view is that the headers should be at least vaguely useful at 
the recipient end. That means that OK examples could be:

- an old (non validating) DKIM record
- X-Virus-Scanned: amavisd-new at amsl.com
- X-BeenThere: [email-masked]
- X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5
   tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
   DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001,
   SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no

But useless ones could be:
- X-Data: E5-cx76azc4-nr189/4/434-00019lcm
- X-GRID-REF: 
8A66B7582E9A48E5AC85DC32E810B6FA.PATT3L315_20414934_HkCfoHWEBm/iKqy8JQBwBgX1wZg0SxXH8yjogkuTlEYPlJqQWxslg0xZdvOP09bi
- X-MA-Instance: 
SIM_TeUTj%2BiE4AhZRTmhfTY44Pwa8ILjXaS6WkWQguAtRkV9EKsTj.f8a41b40b7c841fef95f404ee3efc8c7
- X-PP-Email-transmission-Id: 407b928e-cb6d-11ea-a29f-b875c0aa69dc

(The above are real examples from some recent legitimate messages I've 
received)

The former ones tell me something. It may or may not be that useful. It 
may nor may not be true, but it tells me something. The latter ones tell 
me (the recipient) nothing at all. I'm honestly not sure why the latter 
data is in the headers at all.

It *may* be that if I ask for support from Paypal about an email, 
they'll ask me 'What was the "X-PP-Email-Transmission-Id'' header in 
that message? But I seriously doubt that would happen. If it did, then 
adding that header is OK, otherwise, no. In any case, the Message-ID 
should be sufficient for them to trace it if that's what they need to do.

The former set may be more useful if it was tagged with WHICH server 
added those headers, but that's it.

Paul

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 18:01:29 -0400

--On Tuesday, 21 July, 2020 16:15 -0500 Pete Resnick
<[email-masked]> wrote:

> Other than X- field names not doing what people think they're
> doing, I don't see the problem here. None of them are over 77
> characters, and none (including the ones you showed later with
> curly braces in them) are using problematic characters. So
> what's the problem?

Pete, I think the core issue is the amount of trash we are
accumulating and passing around.  I don't see that as a 5322bis
problem.  

It might be a problem with how the registry is defined (probably
out of scope for emailcore but anyone who believes it isn't
should make a case at the BOF.   

Questions of how and what header fields a final delivery MTA
should discard before the message is dropped in the mailstore
(as I think Hector has been suggesting) because they might be
useful in transit but are not once received and just clutter up
long-term storage of mail (lots cheaper than it used to be but
still not free), or even what IMAP or POP servers should
suppress before hand-off to their clients might be A/S topics
too, although that would be getting, IMO, a bit far afield (and
adding additional options to IMAP to control delivery of this
stuff is almost certainly out of a reasonable interpretation of
emailcore's scope).

But the original inquiry was whether we need to restrict the
length of header field names.  That almost certainly is an
5322bis issue even though the examples given so far don't make a
strong case for doing it.

    john

---

From: "Pete Resnick" <[email-masked]>
Date: Tue, 21 Jul 2020 17:06:51 -0500

> strong case for doing it.

Sure, and in fact is already listed as an issue we'll mention at the 
BOF. (See 
https://www.ietf.org/id/draft-resnick-rfc5322bis-01.html#erratum5918.)

pr

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 18:12:30 -0400

--On Tuesday, 21 July, 2020 22:41 +0100 Paul Smith
<[email-masked]> wrote:

>...
> The former ones tell me something. It may or may not be that
> useful. It may nor may not be true, but it tells me something.
> The latter ones tell me (the recipient) nothing at all. I'm
> honestly not sure why the latter data is in the headers at all.
> 
> It *may* be that if I ask for support from Paypal about an
> email, they'll ask me 'What was the
> "X-PP-Email-Transmission-Id'' header in that message? But I
> seriously doubt that would happen. If it did, then adding that
> header is OK, otherwise, no. In any case, the Message-ID
> The former set may be more useful if it was tagged with WHICH
> server added those headers, but that's it.

Is it the case that, if we wanted to make these useful, we'd be
considering a way to make those fields into trace fields, with
all of the time-stamping and server (or other supplying entity)
information that implies?    Non-trivial work and something that
would certainly have to be done as a new Proposed Standard or
Experimental spec, not in 5321bis, but maybe an interesting
question.

FWIW, it is a question: I have no idea whether it would be a
good idea.

    john

---

From: Michael Richardson <[email-masked]>
Date: Tue, 21 Jul 2020 18:25:30 -0400

Paul Smith <[email-masked]> wrote:
    >> I heard this stated by you a few times. I never brought into it. It wasn't
    >> a problem to be concern about a few decades ago.  Now it is. It's junk and
    >> its growing.  The more it grows the more overhead we have. It makes systems
    >> who are suppose to be ignorant about the junk, work harder.  It makes your
    >> "View Source" more cryptic.

    > Hmm, my view is that the headers should be at least vaguely useful at the
    > recipient end. That means that OK examples could be:

...

    > But useless ones could be:

But, how to tell which is which, and your X-PP-Email-Transimission-Id
scenario is actually a point.

    > The former set may be more useful if it was tagged with WHICH server added
    > those headers, but that's it.

It seems that we could write a BCP that says that X-Foo is bad, but
X-foo.example.com: is better?

I think that there is information in the DKIM header which can tell us what
we need to keep.
Any header mentioned in a validatable DKIM header needs to be kept, right?
That also tells us which server added, or was aware of those headers.

If the message goes through "mailman" or some other processor, then it seems
like it ought to rip pretty much every X-FOO out.  The rest of them ought to
be known headers at the time the processor was written, and it ought to
either know what they are, or it does not, in which case, it shouldn't pass
on things it does not know about.

I suspect that this would at least make a lot of us less grumpy about excess headers.

I don't subscribe to the idea that the target MTA should rip everything other
than From/To/Subject/Date out.  That would limit our ability to innovate, and
there sure has been a lot of that!  But, surely, any system that knows what
some header is, (such as X-BeenThere or X-GRID-REF) *is* ought to rip that
out as it knows whether or not its useful.

---

From: "Kurt Andersen (b)" <[email-masked]>
Date: Tue, 21 Jul 2020 16:03:52 -0700

On Tue, Jul 21, 2020 at 3:25 PM Michael Richardson <[email-masked]>
wrote:

>
> any system that knows what

That's a highly presumptuous stance to take. Knowing what a header is and
making a judgement as to whether or not it is useful are not at all the
same thing. Amongst other relevant points is the question of "useful to
whom (or what)"? Something that is not useful to 99.9% of recipients might
be very useful to the remaining 0.1% (witness the current thrash in the
dmarc-wg regarding components within the 5322.From header field)...

--Kurt

---

From: Keith Moore <[email-masked]>
Date: Tue, 21 Jul 2020 20:48:43 -0400

> accumulating and passing around.  I don't see that as a 5322bis

I think it's slightly deeper than that, which is that every time one 
party adds a new header field, some number of other parties decide that 
they can delete or modify that field, or use it to validate or 
invalidate the message.   So random parties making random decisions 
about message headers are detrimental to email reliability.

Keith

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 21:08:43 -0400

--On Tuesday, 21 July, 2020 20:48 -0400 Keith Moore
<[email-masked]> wrote:

>> accumulating and passing around.  I don't see that as a
>> 5322bis problem.
> 
> I think it's slightly deeper than that, which is that every
> time one party adds a new header field, some number of other
> parties decide that they can delete or modify that field, or
> use it to validate or invalidate the message.   So random
> parties making random decisions about message headers are
> detrimental to email reliability.

I agree with that.  But that is the justification for registered
header field names, preferably well-documented and stable ones.

I contend that:

X-I-know-what-this-means-and-I-am-not-going-to-tell-you: (Longer-
   than 35 characters but much shorter than 78)

does no one (other than maybe me) any good (with or without the
"X-".  If there are sufficiently many like that, they are
invitations to exactly the behavior to which you object.

I think this is probably a bad idea (or just short of nightmare)
but I wonder whether if we created a registration tree for
headers named
     private.* :
required registration of whatever came next, and then declared
open season on the rest if that would make things better.  Or
worse.

E.g.
    private.<antispam>.MyCOMPANY.MYSTUFF:
    private.<antispam>.YourCOMPANY.YourStuff:
    private.<trace>.YOURSTUFF:

Noting (to my surprise, I had forgotten) that section 3.6.8 of
RFC5322 allows any printable ASCII character other than SP or
colon in field names.

As I said, probably a bad idea, but the idea that these things
were all going to be registered and documented has obviously
failed.

   john

---

From: Dave Crocker <[email-masked]>
Date: Tue, 21 Jul 2020 18:31:35 -0700

> used by broken headers is a rounding error on that rounding error.

Folks,

My understanding is that the goal of the emailcore work is to do the 
minimum necessary to enable promoting the core email specifications to 
full standard.

Each of us has a worthy list of wonderful enhancements, cleanups, and 
the like, that we would love to see pursued for email.  None of them is 
likely to be within scope for this round of effort.

To the extent that someone thinks otherwise, consider that each item 
added to the task list delays completion of this round of effort, and 
possibly for quite awhile.

So, as fascinating as this thread has been, unless someone can explain 
how it is essential to the work of the emailcore effort, I'll ask why it 
is worth taking up this much list energy? Absent compelling 
justification, is there a chance people could stop feeding this thread?

As an aside: X- was deprecated.  It's not illegal, but it's no longer a 
formal construct.  I thought it quite a clever suggestion, when we were 
doing RFC 822 -- though I don't remember who suggested it -- but as with 
many good suggestions, it had downsides we hadn't anticipated.

Also, as for cleanup of email header fields in transit, I'm pretty sure 
that it is fully out of scope for SMTP, IMAP, Mail Format and any other 
emailcore work.  It certainly should be, since it isn't causing actual 
operational problems.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Michael Richardson <[email-masked]>
Date: Tue, 21 Jul 2020 21:45:51 -0400

> dmarc-wg regarding components within the 5322.From header field)...

my appologies... let me a use a few more words here.
(because I agree with you, and you clearly didn't understand my point)

A system that knows what X-GRID-REF is, because it use that header itself,
ought to know where it is a useful field or not, and what the implications
are of removing it.

This might deal with some of the X-Foo-bar-SOMEHEXSTUFF: that seems to
accumulate from multiple hops what I suspect is an internal load balanced set
of systems.

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 21 Jul 2020 21:47:58 -0400

> RFC5322 allows any printable ASCII character other than SP or

Why not borrow the general idea from Java and Android, and just incorporate  
DNS itself into header names.

com.courier-mta.MYSTUFF: whatever

I can use any header that starts with my domain name, for whatever I want.

No need to maintain a registration service of any kind. DNS is the  
registration service. If you own the domain name, use any header that starts  
with it.

---

From: Hector Santos <[email-masked]>
Date: Tue, 21 Jul 2020 21:48:15 -0400

On 7/21/2020 5:41 PM, Paul Smith wrote:
> - X-GRID-REF:  ...
> - X-MA-Instance: ....

My point is that at a minimum, if you were to write an MUA, once also 
called Mail Reader/Writer, the fields for the Reader layout viewport 
would be, in no specific order, Date:, From:, To:, Subject: and you 
really don't need anything else.

But at a local MUA level, there are other things as you pointed out. 
Your package may be utilizing extra concepts, Status, Multi-Mime, X-* 
related stuff, including DKIM.

A lot depends on the mail is storage. It can be prune is all I am 
saying and we have been doing it for a long time.   Its a fundamental 
feature in Wildcat! where the user has the option to "[_] Preserve 
Mime." If enabled, then the mail is stored as is, raw RFC5322 format. 
  If disabled, the minimum required fields are extracted and then some 
for supported features.  The rest are not stored.  The only time a 
user really need to Preserve Mime is when the mail is not rendered 
locally, but passed to an offline MUA. The offline MUA has to deal 
with the display.  With the Online MUA, which I suspect is a major 
direction, back to a centralized concept, the backend has all the 
power to do whatever it needs to do from a display standpoint. It 
doesn't all the extra meta headers.

---

From: Hector Santos <[email-masked]>
Date: Tue, 21 Jul 2020 21:55:08 -0400

> out as it knows whether or not its useful.

A MUA does not display all headers. It only extract and displays what 
it needs. For anymore starting to write an MUA from scratch, the 
fields you will need to first deal with for display are 
From/To/Subject/Date.

The OffLine MUA will need all the headers because it handles the 
rendering. The Online MUA will not need all the headers because the 
centralized backend handles the rendering.

---

From: "John R Levine" <[email-masked]>
Date: 21 Jul 2020 22:03:36 -0400

On Tue, 21 Jul 2020, John C Klensin wrote:
> etc., where bandwidth tends to be scarce and expensive ...

Sure, but if that's the issue, you don't address it by messing around
with individual mesage headers.

Back in the day we sent compressed batches of messages over uucp.  I 
expect something like that (or maybe exactly that) still works.

> and encourage an ever-rising collection of noise header fields.

That battle was lost over a decade ago. See for example the headers
below that I took from a message that came through one of my mailing
lists today. They'd be even longer if the message had started at
Google's webmail rather than a MUN on the author's phone.

R's,
John

---- snip ----

From: Mike Radow <[email-masked]>
Date: Tue, 21 Jul 2020 13:28:48 -0700
  <CAEWkM+BtrMcVumz4P=[email-masked]> <[email-masked]>
In-Reply-To: <[email-masked]>
To: [email-masked]
X-Mailer: iPhone Mail (16D39)
Subject: Re:[circletime] 
Reply-To: [email-masked]
X-Loop: [email-masked]
X-Sequence: 3553
Errors-To: [email-masked]
Precedence: list
Precedence: bulk
Sender: [email-masked]
X-no-archive: yes
List-Id: <circletime.lists.iecc.com>
List-Help: <mailto:[email-masked]?subject=help>
List-Subscribe: <mailto:[email-masked]?subject=subscribe%20circletime>
List-Unsubscribe: <mailto:[email-masked]?subject=unsubscribe%20circletime>
List-Post: <mailto:[email-masked]>
List-Owner: <mailto:[email-masked]>
List-Archive: <https://sympa.services.net/sympa/arc/circletime>

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2020 22:55:29 -0400

--On Tuesday, 21 July, 2020 18:31 -0700 Dave Crocker
<[email-masked]> wrote:

> My understanding is that the goal of the emailcore work is to
> do the minimum necessary to enable promoting the core email
> specifications to full standard.
> 
> Each of us has a worthy list of wonderful enhancements,
> cleanups, and the like, that we would love to see pursued for
> email.  None of them is likely to be within scope for this
> round of effort.
> 
> To the extent that someone thinks otherwise, consider that
> each item added to the task list delays completion of this
> round of effort, and possibly for quite awhile.
> 
> So, as fascinating as this thread has been, unless someone can
> explain how it is essential to the work of the emailcore
> effort, I'll ask why it is worth taking up this much list
> energy? Absent compelling justification, is there a chance
> people could stop feeding this thread?
> 
> As an aside: X- was deprecated.  It's not illegal, but it's no
> longer a formal construct.  I thought it quite a clever
> suggestion, when we were doing RFC 822 -- though I don't
> remember who suggested it -- but as with many good
> suggestions, it had downsides we hadn't anticipated.
> 
> Also, as for cleanup of email header fields in transit, I'm
> pretty sure that it is fully out of scope for SMTP, IMAP, Mail
> Format and any other emailcore work.  It certainly should be,
> since it isn't causing actual operational problems.

Dave,

This collection of threads actually started with a claim that
long header field names were causing operational problems. It
was followed, I think, by at least an implicit claim that the
proliferation of unregistered header fields with very long field
bodies has caused operational problems.

I'm not convinced.  I infer that you are not convinced.  From
his notes, I'm nearly certain that John Levine is not convinced.
However, if I may paraphrase the rules crudely, we don't advance
documents to Internet Standard that specify things that don't
work.  So, if someone really wants to make the case that the
language about <optional-field>s in 5322 causes problems without
some addition specification or restrictions, I think that is,
sadly, in scope.  I would hope that we could handle it in the
Applicability Statement that I hope will accompany
5321bis/5322bis but that is ultimately up to the WG.

And I hope we don't have to fuss with optional header fields at
all and will conclude that the text in 5322 is adequate without
even a reference to the registry.   But I don't think we can
shut off the discussion and prevent those who think there are
operational problems from making their case until someone is in
a position to make a consensus call.

best,
   john

---

From: Keith Moore <[email-masked]>
Date: Tue, 21 Jul 2020 23:16:37 -0400

> header field names, preferably well-documented and stable ones.

I suspect this is not an issue that can be addressed by any combination 
of registration and/or field-naming constraints in the absence of 
instructions to message-handling software about which fields (if any) 
they can add, modify, or delete.

I love the format's extensibility but I suspect there's a point of 
diminishing returns.   The message header has become a garbage dumping 
ground, and it might need some cleaning up.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Tue, 21 Jul 2020 20:53:41 -0700

> long header field names were causing operational problems. It

When someone makes a claim like that, they need to document it.  Nothing 
in this thread has done anything like documenting the existence of an 
operational problem.  Unless I missed it, but I don't think I did.

Rather, what happened was a claim that then led to lots of /other/ 
people putting quite a bit of energy into evaluation of all sorts of 
information, none of which demonstrated any operational problems.

Or even tried to, IMO.

> I'm not convinced.  I infer that you are not convinced.  From

Things that cause operational problems are described very differently 
than what's been happening here.  They state the problem(s) that are 
caused and then give details about how, when, and where.

>    So, if someone really wants to make the case that the
> sadly, in scope. 

There is a difference between saying "I think bad-thing-x is happening 
and causing problems" from saying "bad-thing-x is causing the following 
problems, in the following ways and with the following systems."

Someone claiming a problem has the burden of documenting that the 
problem is real.

It is not reasonable for them to raise a question and expect other 
people to do the work of proving/disproving.  The latter approach is, 
effectively, a denial of service attack. (There are, of course, 
exceptions, but the framework of legitimate vs. questionable is really 
quite straightforward.)

  I would hope that we could handle it in the
> 5321bis/5322bis but that is ultimately up to the WG.

People continue to love ASs, but my impression is that they have no 
field utility.

> a position to make a consensus call.

In formal terms, that is of course correct.

But people can choose to not respond to postings lacking in-scope 
relevance or substance.

That's why I refrained from posting, but the fact that this thread 
persisted suggested a meta-issue worth broaching early in the wg effort, 
in the possibility it has continuing relevance.  I mean, it's just 
possible that people will have other opportunities to lose focus from 
the actual goals of emailcore.  Shocking, I know, but...

d/

---

From: Dilyan Palauzov <[email-masked]>
Date: Wed, 22 Jul 2020 07:37:29 +0000

Hello,

> based on broken DKIM signatures, against the rfc.

This does not have to be against the RFC.

An open source software for doing DKIM, that was last released  
(tagged/a tarball was created) in May 2015, got a bugfix in October  
2015.   Whoever uses the released version, does for some emails the  
DKIM calculations wrong.  This gets evident, after evaluating a lot of  
aggregate reports on a semi-busy server.  Since the users of the  
software do not understand DKIM/do not evaluate the aggregate reports  
for their own servers, their keep running the unpatched software.  So  
some correct DKIM headers are validated as wrong, and bad DKIM headers  
are inserted.  When DMARC says to reject messages that do not validate  
DKIM, and either the recipient considers valid DKIM as invalid, or the  
sender inserts invalid DKIM, then advancing from DKIM to DMARC/ARC  
does not make sense.

SMTP-rejecting a suspicious message is much better that delivering the  
message to a recipient so, that the recipient is likely to overlook it.

To sum up, if a message is rejected because
- the DKIM was broken, so the rejection is against the RFC (no DMARC  
involved), or
- the DKIM was correct, but the DKIM evaluation software on the  
recipient site has bug in the calculation algorithm and the sender  
published DMARC reject, or
- the DKIM was inserted by software that does the signing wrongly, the  
sender publishes DMARC reject, and the recipient applies the RFCs  
correctly

has all the same consequences.

Finding and resolving DKIM bugs, not limited to DNS troubles, is  
essential to bring DKIM/ARC/DMARC forward.  My opinion is that there  
is no willingness in email operators to assist each other on this.   
Cooperation on the matter means, more or less, sending individual  
failure reports (otherwise one is not going to acknowledge, that s/he  
uses DKIM-software with bugs).

Greetings
   Дилян

----- End message from Brandon Long <blong=[email-masked]> -----

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Jul 2020 13:31:38 +0100

> https://www.ietf.org/proceedings/108/agenda/agenda-108-emailcore-00,

Thank you for the introduction, John.

The above was updated a bit:

https://www.ietf.org/proceedings/108/agenda/agenda-108-emailcore-01

> Alexey or Seth may be able to add to that, especially if you
I don't have much to add at this point. Seth and I will send the 
conference call link for the BOF once we know it.

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Jul 2020 08:14:29 -0700 (PDT)

> ground, and it might need some cleaning up.

Doors, once opened, are hard to close.

Building a door where none existed before is even more difficult. Especially
when the costs of construction have yet to be justified.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 22 Jul 2020 17:48:30 +0200

> on things it does not know about.

YES!

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 22 Jul 2020 13:15:38 -0400

>This does not have to be against the RFC.

Sections 6.1 and 6.3 of RFC 6376 say that an unverifiable DKIM
signature is treated as though the signature was not present.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "John Levine" <[email-masked]>
Date: 22 Jul 2020 13:29:39 -0400

>on things it does not know about.

That's not gpod advice. The point of the mystery headers is to tell
what happened to the message during its trip, and the part of the trip
before it hit the list manager is as important as the part after. When
I'm trying to figure out why something undesirable leaked through the
list manager, I need the original headers to figure out what happened.

I also don't understand what problem this is supposed to address. No
MUA of which I'm aware displays junk headers, so it can't be user
confusion.  If the concern is bandwidth, why aren't we resuscitating
the cascaded sub-list managers from prior decades, or inventing something
like STARTDEFLATE for SMTP?  Headers are in the noise when people are
shipping around giant blobs of HTML and attached pictures.

R's,
John

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 22 Jul 2020 13:11:17 -0500

On 21 Jul 2020, at 22:53, Dave Crocker wrote:

> When someone makes a claim like that, they need to document it.  
> Nothing in this thread has done anything like documenting the 
> existence of an operational problem.

Yep, I think that's what led several of us to ask, "What exactly is the 
problem?" People are implying that there's some sort of operational 
problem, but nobody's saying what it is.

And I agree that it is using quite a bit of unnecessary energy.

> People continue to love ASs, but my impression is that they have no 

I think it's worth a discussion here, or in the BOF, of what's meant by 
an "Applicability Statement" in this context, because people use that 
term very loosely. My sense is that it is not meant to be a "problem 
statement" or a "technology review" or a "requirements document" (as 
people often intend "AS"), but rather a document giving a description of 
how the protocol pieces are used together in the real world, with a bit 
of operational guidance and some solid terminology. Perhaps "AS" is the 
wrong term, but I think what the charter describes would be useful. We 
should probably work out both what would be useful, and what to call it 
to avoid confusion.

pr

---

From: Dave Crocker <[email-masked]>
Date: Wed, 22 Jul 2020 11:20:42 -0700

On 7/22/2020 11:11 AM, Pete Resnick wrote:
> a document giving a description of how the protocol pieces are used 
> together in the real world, with a bit of operational guidance and some 
> solid terminology.

Not offered as satisfying what's needed for the current effort, but 
wondering about the, ummm... applicability... of this approach:

    https://bbiw.net/clusters/#suitemail

d/

---

From: Michael Richardson <[email-masked]>
Date: Wed, 22 Jul 2020 14:33:04 -0400

> list manager, I need the original headers to figure out what happened.

Right. You need the standard "Received:" lines, which would be a known
header at this point, so it would remain.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "John R Levine" <[email-masked]>
Date: 22 Jul 2020 14:55:48 -0400

> header at this point, so it would remain.

Really, I find the other stuff to be useful.  I understand that you don't, 
but you're not the only one here.

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 22 Jul 2020 14:57:12 -0400

Also, can we return to the bit you snipped out?

What problem is addressed by removing junk mail headers?

Given that our time is limited, I would prefer not to spin our wheels on 
non-problems.

R's,
Joh

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 22 Jul 2020 14:50:51 -0500

On 22 Jul 2020, at 13:20, Dave Crocker wrote:

>> together in the real world, with a bit of operational guidance and 
>> some solid terminology.
>

I had completely forgotten about that web page. I saw it several years 
ago. I think it's a great starting point for a bunch of what I was 
thinking about.

pr

---

From: Steve Atkins <[email-masked]>
Date: Thu, 23 Jul 2020 11:07:35 +0100

On 22/07/2020 19:55, John R Levine wrote:
>>    > That's not gpod advice. The point of the mystery headers is to tell
>>    > what happened to the message during its trip, and the part of 
>> the trip
>>    > before it hit the list manager is as important as the part 
>> after. When
>>    > I'm trying to figure out why something undesirable leaked 
>> through the
>>    > list manager, I need the original headers to figure out what 
>> happened.
>>
> Really, I find the other stuff to be useful.  I understand that you 
> don't, but you're not the only one here.

Autoresponders and list managers that snip out headers they don't 
understand, particularly those that were added as part of a 
belt-and-braces attempt to avoid mail loops and message amplification, 
can cause problems.

Those not-described-by-RFC headers - e.g. "X-BeenThere: 
[email-masked]" - are not just useful for manual diagnostics, but 
also for automated mitigation of operational problems.

Cheers,
   Steve