Subject: [ietf-smtp] Email explained from first principles

From: Kaspar Etter <[email-masked]>
Date: Fri, 14 May 2021 15:27:07 +0200

Dear IETF community,

I spent several months researching and summarizing all aspects of modern email in an attempt to make the email standards more accessible to a wider audience. You find the result of my work at https://explained-from-first-principles.com/email/.

I'm interested in your feedback and criticism: Did I get anything wrong? Is an important aspect still missing?

Besides summarizing more than a 100 RFCs, my article contains many suggestions for how existing standards could be improved. This mailing list is probably not the right place to discuss them, but I would still like to point out a few of them:

1. I made a security analysis of SCRAM and provided several suggestions for how its security could be improved even further: https://explained-from-first-principles.com/email/#desirable-properties-of-authentication-mechanisms and https://explained-from-first-principles.com/email/#salted-challenge-response-authentication-mechanism

2. SMTP for Submission should make it clear that the IP address of the submitter SHOULD NOT be included in the Received header field. In my view, the current practice of many email service providers violates the European General Data Protection Regulation (GDPR): https://explained-from-first-principles.com/email/#sender-towards-recipients

3. This ship sailed a long time ago, but remote content violates three fundamental principles of email and should thus never have been supported by mail clients. At the very least, subresource integrity (SRI) should be introduced and be required for all remote content: https://explained-from-first-principles.com/email/#remote-content

4. HTML emails are broken. Mail clients struggle to quote them securely, and the same message can appear differently to different recipients: https://explained-from-first-principles.com/email/#quoting-html-messages and https://explained-from-first-principles.com/email/#different-appearances

5. To fix the problem of different appearances for different recipients, a profile of CSS which ensures that content cannot be hidden should be specified and then implemented by all mail clients: https://explained-from-first-principles.com/email/#hide-content-with-css

6. Are SPF verifiers supposed to follow CNAME records and if so, do they count towards the lookup limit? I couldn't find an answer in the RFC. In my opinion, resolving CNAME records during SPF validation is undesirable for security reasons: https://explained-from-first-principles.com/email/#protecting-subdomains

7. I don't understand why Authenticated Received Chain (ARC) is necessary or even desirable: https://explained-from-first-principles.com/email/#authenticated-received-chain

8. It's rather complicated in which cases DANE applies to ESMTP. I hope I got all the cases right: https://explained-from-first-principles.com/email/#name-matching

9. RFC 8461 states explicitly that MTA-STS may not override a failed DANE validation. As far as I can tell, it isn't specified anywhere whether DANE may override a failed MTA-STS validation. In my opinion, this would be desirable because it would allow domain owners to configure transport security without the support of their email service provider: https://explained-from-first-principles.com/email/#coexistence-with-dane

As a feedback from my side to the IETF community: 6 out of the 8 errata which I reported are still not "resolved" (see https://www.rfc-editor.org/errata_search.php?submitter_name=Kaspar+Etter). I hope the errata process is revised rather sooner than later.

Best regards,
Kaspar Etter

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 17 May 2021 11:33:20 +1000

> I'm interested in your feedback and criticism: Did I get anything wrong? Is an important aspect still missing?

Thanks Kaspar,

It's clear you've put a lot of work into putting this together.  Unfortunately I don't have time right now to do it justice and review the whole thing!  I'm not sure that anyone else is likely to either.  This is one of the ongoing challenges for any work, it's more interesting to do the work than to review it!

But that actually raises the more interesting question out of all of this.  Are you interested in engaging with the IETF community and helping to progress your ideas within the IETF?  Because that's going to involve the messiness of human interaction and building relationships so that people make the time to review your work because they know you will return the favour when they need reviews

> 1. I made a security analysis of SCRAM and provided several suggestions for how its security could be improved even further: https://explained-from-first-principles.com/email/#desirable-properties-…

I wonder if you'd be interested in the work happening in the GNAP working group.

> 2. SMTP for Submission should make it clear that the IP address of the submitter SHOULD NOT be included in the Received header field. In my view, the current practice of many email service provider…

This is an interesting question.  Are you entitled to such privacy from the recipient?  If so, on what basis?  This kind of thing is very much a tradeoff of various concerns, and by removing your IP address, the server to which you are making the submission is taking on additional responsibility for the content of your message.

> 3. This ship sailed a long time ago, but remote content violates three fundamental principles of email and should thus never have been supported by mail clients. At the very least, subresource inte…

I strongly agree with you on this one :)   Immutable content and completeness are a big deal to me.  It's somewhat solveable by clients fetching the content and caching it immediately - or servers doing the same and rewriting the links.

But upgrading from where we are to there is a big project - who's going to do the work?  Both in defining something better, and persuading the world to move to it.  These are both hard problems, because the something better needs to be persuasive.

> 4. HTML emails are broken. Mail clients struggle to quote them securely, and the same message can appear differently to different recipients: https://explained-from-first-principles.com/email/#quot…

Oh yeah, HTML is a horrible choice for this.  But, like democracy, it's better than all the alternatives!  Defining a subset and getting people to use it would be great, but see my response to 3.  How do you sell this to all the involved parties and have an upgrade path which doesn't lead to all multipart emails having 3 parts, text/plain, text/html and text/the-better-one ?

> 5. To fix the problem of different appearances for different recipients, a profile of CSS which ensures that content cannot be hidden should be specified and then implemented by all mail clients: h…

See previous.

> 6. Are SPF verifiers supposed to follow CNAME records and if so, do they count towards the lookup limit? I couldn't find an answer in the RFC. In my opinion, resolving CNAME records during SPF vali…

As with anything where it's not really clear, the only safe assumption is to assume the least favourable to you of all possibilities, so - don't use CNAME records and if you do, they'll be counted towards the lookup limit!  Which I see you've already mentioned.

> 7. I don't understand why Authenticated Received Chain (ARC) is necessary or even desirable: https://explained-from-first-principles.com/email/#authenticated-received-chain

It's an attempt to solve the mailing list problem and the forwarder problem.  At least for pure forwarding the simple solution is to keep the bytes intact so that DKIM still passes, but for messages without DKIM it will at least maintain some integrity tracking on the SPF check that was done when it entered ARC-supporting servers.

I have written a bit on this, and I do feel that having an additional DMARC policy of reject-unless-arc-trusted is necessary for the intermediate to know whether the far end will reject - but the problem is that there's no clean bootstrap path either way, because you fail either too open or too closed.  The intermediate doesn't know if it needs to address-rewrite anyway to make sure the mail gets through.

Note for example that this email is listed as being:

From: Kaspar Etter <kaspar=[email-masked]>

Because the IETF list has to rewrite due to your domain's policy.

> 9. RFC 8461 states explicitly that MTA-STS may not override a failed DANE validation. As far as I can tell, it isn't specified anywhere whether DANE may override a failed MTA-STS validation. In my…

I'm not an expert in this area so I won't try to respond to these two.

> As a feedback from my side to the IETF community: 6 out of the 8 errata which I reported are still not "resolved" (see https://www.rfc-editor.org/errata_search.php?submitter_name=Kaspar+Etter). I h…

There's a discussion in the IETF generally about how to deal with the backlog of errata.  Again as I mentioned right up the top - getting people to do the boring grunt work of reading others' work and providing insightful feedback is tricky!  Finding time to process things is tricky.

Clearly you've found an enormous amount of time to put all this together, and it's really excellent work.  It feels like the kind of thing that you should be getting academic credit for, to the extend of being a published paper, at least to the "Honours Degree" level - it's carefully researched and meticulous.

I hope you have the time and the willingness to engage with the IETF community as a contributor and to work within the ebb-and-flow of human relationships to progress some of your ideas and your work further.  We could do with the energy that you're bringing - it's just a lot to bite off so much all in one go!

Cheers,

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 17 May 2021 12:25:30 +0200

On Mon 17/May/2021 03:33:20 +0200 Bron Gondwana wrote:
>> I spent several months researching and summarizing all aspects of modern 
>> email in an attempt to make the email standards more accessible to a wider 
>> audience. You find the result of my work at 
>> https://explained-from-first-principles.com/email/ 

Interesting effort.  I just gawked at it.  I hope I'll find time to read some 
more...

>> an important aspect still missing?

One thing is the definition of /email service provider/.  The wikipedia article 
you link to is correctly named /Mailbox provider/.  A number of people consider 
ESPs a different kind of business.  Others, especially foreigners, don't grok that.

>> 2. SMTP for Submission should make it clear that the IP address of the 
>> submitter SHOULD NOT be included in the Received header field. In my view, 
>> the current practice of many email service providers violates the European 
>> General Data Protection Regulation (GDPR): 
>> https://explained-from-first-principles.com/email/#sender-towards-recipients 
> 
> This is an interesting question.  Are you entitled to such privacy from the 
> recipient?  If so, on what basis?  This kind of thing is very much a tradeoff 
> of various concerns, and by removing your IP address, the server to which you 
> are making the submission is taking on additional responsibility for the 
> content of your message.

I'd note that the author's email address is usually considered more private 
than the (temporary) IP address used for submission.  Yet, in this respect, 
protocols are not clear about what submitters are allowed to put in the 
From:/MAIL FROM addresses.  Even if the SMTP operator DKIM signs From: and 
restricts MAIL FROM by SPF, it is not formally required to check those values.

>> 3. This ship sailed a long time ago, but remote content violates three 
>> fundamental principles of email and should thus never have been supported by 
>> mail clients. At the very least, subresource integrity (SRI) should be 
>> introduced and be required for all remote content: 
>> https://explained-from-first-principles.com/email/#remote-content 
> 
> I strongly agree with you on this one :)   Immutable content and completeness 
> are a big deal to me.  It's somewhat solveable by clients fetching the content 
> and caching it immediately - or servers doing the same and rewriting the links.

+1, perhaps that's why it's not much used.

>> 4. HTML emails are broken. Mail clients struggle to quote them securely, and 
>> the same message can appear differently to different recipients: 
>> https://explained-from-first-principles.com/email/#quoting-html-messages 
>> <https://explained-from-first-principles.com/email/#quoting-html-messages> 
>> and https://explained-from-first-principles.com/email/#different-appearances

For a nit, s/they will see USD 100/she will see USD 100/.

The usual scam is to invite readers to go to <a href="hell">paradise</a>.

Did you find clients that execute Javascript?

> Oh yeah, HTML is a horrible choice for this.  But, like democracy, it's better 
> than all the alternatives!

Text/plain _is_ a better alternative (to HTML, not to democracy...)

>> 6. Are SPF verifiers supposed to follow CNAME records and if so, do they 
>> count towards the lookup limit? I couldn't find an answer in the RFC. In my 
>> opinion, resolving CNAME records during SPF validation is undesirable for 
>> security reasons: 
>> https://explained-from-first-principles.com/email/#protecting-subdomains 
> 
> As with anything where it's not really clear, the only safe assumption is to 
> assume the least favourable to you of all possibilities, so - don't use CNAME 
> records and if you do, they'll be counted towards the lookup limit!  Which I 
> see you've already mentioned.

OTOH, resolvers can deliver the target without requiring an extra call.  And 
the number of calls is the only thing an SPF verifier can/should count.

>> 7. I don't understand why Authenticated Received Chain (ARC) is necessary or 
>> even desirable: 
>> https://explained-from-first-principles.com/email/#authenticated-received-chain 
> 
> It's an attempt to solve the mailing list problem and the forwarder problem.  
> At least for pure forwarding the simple solution is to keep the bytes intact so 
> that DKIM still passes, but for messages without DKIM it will at least maintain 
> some integrity tracking on the SPF check that was done when it entered 
> ARC-supporting servers.
> 
> I have written a bit on this, and I do feel that having an additional DMARC 
> policy of reject-unless-arc-trusted is necessary for the intermediate to know 
> whether the far end will reject - but the problem is that there's no clean 
> bootstrap path either way, because you fail either too open or too closed.

It is interesting to consider the similarity between ARC and Dave's Use of 
Sender[*].  Both aim to allow an intermediate to add something to the header so 
that the recipient should put a DMARC=pass notwithstanding the letter of RFC 
7489 would mandate DMARC=fail given the final From:.  In both cases, the 
legitimacy of the stuff added to the header cannot be verified by the recipient.

[*] https://datatracker.ietf.org/doc/html/draft-crocker-dmarc-sender

> The intermediate doesn't know if it needs to address-rewrite anyway to make
> sure the mail gets through. >
> From: Kaspar Etter <kaspar=[email-masked] 
> <mailto:[email-masked]>>
> 

The latter point, the need to munge From:, proves that ARC is not necessary.

>> 8. It's rather complicated in which cases DANE applies to ESMTP. I hope I got 
>> all the cases right: 
>> https://explained-from-first-principles.com/email/#name-matching

IMHO, putting *Trust anchor* first would better convey the intention of DANE.

Then, there are some other alternatives (!=3) to generate TLSA RRs.  Are they used?

>> 9. RFC 8461 states explicitly that MTA-STS may not override a failed DANE 
>> validation. As far as I can tell, it isn't specified anywhere whether DANE 
>> may override a failed MTA-STS validation. In my opinion, this would be 
>> desirable because it would allow domain owners to configure transport 
>> security without the support of their email service provider:
>> https://explained-from-first-principles.com/email/#coexistence-with-dane 
> 

Neither am I.  However, I'd note a couple of points:

* It's not clear who provides web hosting for MTA-STS.  Is it a third role?

* Domain owner should prepare new TSLA RRs /before/ the MX operator changes 
key/certificate.  Failure to sync might entail a few days of blackout.

Best
Ale
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 17 May 2021 17:23:46 -0400

On Fri, May 14, 2021 at 03:27:07PM +0200, Kaspar Etter wrote:

> 8. It's rather complicated in which cases DANE applies to ESMTP. I
> hope I got all the cases right:

I guess this one falls on me.

>   If a certificate usage other than DANE-EE is being used, DANE clients
>   have to verify that the validated certificate matches the server’s
>   identity. In order to do so, they carry out the following, rather
>   complicated and boring procedure:

This can be made more intuitive (if not necessarily simpler).

>       Determine the TLSA domain: The TLSA lookup is usually performed
>       on the server’s domain after resolving any CNAME, MX, and SRV
>       records. RFC 7672, however, makes things a bit more complicated
>       for ESMTP.

One simplifying observation is that the "TLSA base domain" (what you've
shortened to the TLSA domain) is wherever the TLSA records were found!

A. If TLSA records were found via a query for "_25._tcp." prepended
   to some "base domain", then the TLSA base domain is THAT domain.
   [ This is the name used to make the query, not any CNAME expansion
     that may happen as part of resolving that name to a an actual TLSA
     RRset. ]

B. In addition, the TLSA base domain is always associated with the
   logical nexthop server to which the mail will be relayed, i.e. the
   "MX host".  Which may be implicitly equal to the recipient domain if
   that domain has no MX records (per RFC5321, 2821 and 821).

   In a DANE-enabled SMTP client the MX lookup is performed via a
   security-aware or validating stub resolver, that can determine
   whether the MX records (or their denial of existence) were signed.

   If the MX lookup leads to an insecure answer, DANE does not apply.
   [ See "H." below for a note on DNS error semantics. ]

C. Regardless of whether that logical nexthop domain is explicit or
   implicit, since a connection is going to be made to it, one needs
   to obtain its address records (some combination of A/AAAA).

D. Resolving that logical name to an address RRset (or two) via
   a validating resolver will also reveal whether those addresses
   lie in a signed zone.

E. If one or more of the address RRSets (IPv4 "A" and/or IPv6 "AAAA") is
   signed (it is possible, e.g. for the IPv4 address RRSet to come
   secure, while denial of existence of IPv6 addresses to come back
   insecure, or vice versa), then check for TLSA records at the fully
   nexthop server domain (see "H." below).

   * If found (see "A." above), that's the TLSA base domain.
   * If none found (NODATA or NXDOMAIN), continue with "F." or "G."
     as appropriate.

F. If no address RRSet zone is signed, and there is no CNAME expansion
   involved in getting from the logical name to the addresses, DANE is
   not possible, proceed without DANE.

G. When either:

   i.  "E." above yields no TLSA records, or

   ii. In "F." above, CNAMEs lead to all addresses being insecure, it is
       possible that the origin zone is signed, and it is only the
       target zone with the addresses (or some CNAME in a multi-step
       CNAME chain) that is insecure.

       To determine whether that's the case, a security-aware stub
       resolver needs to issue an explict "CNAME" query for the
       unexpanded name, and determine its security status.  A validating
       stub resolver (not widely used) may already know this as part of
       validating the query response.

       If the original unexpanded name is also unsigned, DANE is not
       possible, proceed without DANE. Otherwise,

    Look for TLSA records of the original unexpanded server name
    (prepending _25._tcp. ...).

H.  TLSA record lookup failures are security sensitive.  Acceptable
    outcomes are:

    i. Success, signed validated TLSA records.  DANE MUST be used.
    ii. Unsigned TLSA records.  DANE does not apply.
    iii. Signed or unsigned denial of existence (NODATA or
         NXDOMAIN).  DANE does not apply.

    All other outcomes (SERVFAIL, REFUSED< BOGUS, timeout, ...)
    are potential downgrade attacks, and the MX host in question
    is skipped, possibly leading to the message being deferred if
    no other candidate MX hosts remain.

I.  By this point we have one of:

    - TLSA records and an associated "TLSA base domain"
    - Proof that the records don't exist, or that all zones in
      which we might have tried to look are unsigned.
    - A lookup failure, which means we skip the MX host.

    In either of the latter cases no DANE.

J.  The TLSA base domain is therefore either:

    i. A CNAME expanded nexthop server
    ii. The original nexthop server.

    Either could be the same as the recipient domain, or its CNAME
    expansion.  Note that the TLSA RRset itself may involve CNAME
    expansion, this is normal, and DOES NOT affect the TLSA base
    domain which is always the query name, not the domain

    Wherever the TLSA base domain came from, the SMTP client uses this
    TLSA base domain as the SNI value for the TLS handshake with the
    target SMTP server.

K.  For purposes of certificate verification the client builds a list
    "reference identifiers" that will compared against the names found
    in the peer certificate (the "presented identifiers").

    The list of "reference identifiers":

        - Always includes the TLSA base domain

        - Will also include the unexpanded name of the MX host, if
          different from the TLSA base domain.

        - Will also include the unexpanded recipient domain if different
          from those above.

L.   The presented identifiers are compared against the reference
     identifiers until one pair matches, per RFC6125, wildcards in
     presented identifiers, ...

Twelve steps should about (re)cover it.

>     First of all, the domain found after the @ symbol in the email
>     address is CNAME-expanded. If the whole expansion was secure, you
>     look up its MX records.

Well, you look up MX records either way, it is just that if they or
their absence is insecure, DANE is off the table.

>     Any domain name listed in an MX record has to resolve directly to
>     A/AAAA records according to RFC 2181 and RFC 5321.

True, but most MTAs are more liberal, and support CNAME expansion of
also the MX hosts (though a my survey shows only ~1% of DANE MX hosts to
be CNAME-valued).

>     If the expanded @ domain has no MX records, you fall back to its
>     A/AAAA records.  The TLSA domain is determined as follows:

See above for the the full procedure.

>       ------------------------------------------------------------------------------------
>       Non-expanded @ domain | Expanded @ domain | MX domain        | TLSA domain
>       ====================================================================================
>    1. Secure CNAME record   | Secure MX records | Secure A/AAAA    |
>                             |                   | and TLSA records | MX domain
>       ------------------------------------------------------------------------------------
>    2. Secure CNAME record   | Secure A/AAAA     | -                | Expanded @ domain
>                             | and TLSA records  |                  |
>       ------------------------------------------------------------------------------------
>    3. Secure CNAME record,  |                   | -                | Non-expanded @ domain
>       secure TLSA records   | Insecure A/AAAA   |                  |
>       records               |                   |                  | [some glyph]
>       ------------------------------------------------------------------------------------
>    4. Secure MX records     | -                 | Secure A/AAAA    | MX domain
>                             |                   | and TLSA records |
>       ------------------------------------------------------------------------------------
>    5. Insecure MX records   | -                 | Secure A/AAAA    | MX domain
>                             |                   | and TLSA records | [some glyph]
>       ------------------------------------------------------------------------------------
>    6. Secure A/AAAA         | -                 | -                | Non-expanded @ domain
>       and TLSA records      |                   |                  |
>       ====================================================================================

The above does not cover short-circuiting TLSA lookup when both the A
and AAAA RRsets come back insecure.

It also assumes that the MX host is NOT CNAME expanded (per 5321), in
practice many MTAs (e.g. at least Postfix and Exim) allow this.

Case "5" is not possible, the MX records and A/AAAA records are obtained
from the same zone, and so the security status should be the same, also
insecure TLSA records are not used.

>       In all other cases, DANE does not apply. I’ve marked the
>       unexpected cases with [some glyph] 

It is not clear what's "unexpected" about them.

>       RFC 7672 allows ESMTP clients to use DANE even if the MX records
>       were in a zone whose records aren’t authenticated with DNSSEC.

True, provided that the MX host (different from the recipient domain
or its CNAME expansion) is in a signed zone.  But this is optional,
and the MTAs logs must not claim secure delivery in this case.  DANE
protects the transport to a possibly forged MX host in this case.

>   If you implement an ESMTP client, you can avoid all of this
>   complexity by supporting only the certificate usage DANE-EE.

Actually, you don't avoid most of the work, because the only thing that
changes is construction of the reference identifier list.  One must
still locate the TLSA records correctly, construct the correct SNI name,
otherwise you may get the wrong certificate and fail to match even a
DANE-EE RRset.

So the above is bad advice.  Many domains are using DANE-TA(2), which
SHOULD be supported.

-- 
        Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 17 May 2021 20:07:55 -0400

On Fri, May 14, 2021 at 03:27:07PM +0200, Kaspar Etter wrote:

> 9. RFC 8461 states explicitly that MTA-STS may not override a failed
> DANE validation. As far as I can tell, it isn't specified anywhere
> whether DANE may override a failed MTA-STS validation. In my opinion,
> this would be desirable because it would allow domain owners to
> configure transport security without the support of their email
> service provider:

One more thing.  This is not entirely surprising, since the DANE spec
came out well before MTA-STS.  If Postfix were to officially support
MTA-STS (as opposed to bolted-on external TLS policy tables), then
I'd probably have to implement a more advanced logic for policy
selection, were DANE is used when applicable, else MTA-STS is used
when applicable, ...

It is unlikely I'd implement support for both on the same connection, as
that would make the TLS setup logic rather too complex.  The two may
imply different SNI choices, require different APIs for chain
validation, ... so I just don't see much opportunity for realistic
conflict.  The client will typically pick just security mechanism to
apply up front, and succeed or fail the connection based on that.

-- 
    Viktor.

---

From: Kaspar Etter <[email-masked]>
Date: Sun, 23 May 2021 14:15:47 +0200

Dear Bron,

Thanks a lot for your response and please excuse my late reply.

> On 17 May 2021, at 03:33, Bron Gondwana <[email-masked]> wrote:

Yes, I totally understand this. While I asked for feedback and criticism, I also shared my work with this mailing list for two other reasons:

1. My work could be really useful to the IETF community by allowing you to refer people to this comprehensive introduction to modern email. (For some people here, it’s also a summary of and a tribute to their life’s work.)

2. I’m very much aware of the epistemic damage one can cause as an author, and I take this responsibility seriously. For example, confusing opportunistic use of TLS with explicit use of TLS was a big mistake. I don’t know who started this, but I tried my best to rectify this misconception. If my article becomes the definitive guide to email for newcomers (I know that this is a big “if”), any error in my text can cause a lot of work for people on this mailing list. Differing terminology is a good example of harm I might cause to the email community, and I’ve already received two remarks in this direction.

> But that actually raises the more interesting question out of all of this.  Are you interested in engaging with the IETF community and helping to progress your ideas within the IETF?  Because that'…

You raise a valid point. I don’t mind the messiness of human interactions and I can imagine getting involved in some IETF efforts. I’m somewhat hesitant for two reasons:

1. I might be wrong, but my impression is that most people are paid for their standardization work by a company or a university. I won’t be able to contribute a significant amount of work unless an organization is willing to sponsor me.

2. Standards alone don’t matter. During my research, I learned about many great standards which haven’t found significant adoption yet. Examples of this are SRV records for autoconfiguration and the REQUIRETLS extension to ESMTP. Innovation has to be driven by the relevant players in the field, and I’m not one of them.

Having said this, I’d be open to work on the following, relatively low hanging fruits if enough people here encourage me to do so:

1. Formalizing no reply: Many automated systems send emails but cannot handle replies. A new RFC could either acknowledge that no reply shall be sent if the local part of the sender is “no-reply” or introduce a new header field to achieve the same. (See https://explained-from-first-principles.com/email/#no-reply <https://explained-from-first-principles.com/email/#no-reply> for more context. The idea is that mail clients wouldn’t even offer a reply button for such messages.)

2. List-Name header field: Mailing lists shouldn’t rewrite the messages of others and break DKIM signatures in the process. Since my email address was rewritten due to my domain policy, I didn’t receive some of the replies to my original message because I didn’t subscribe to this mailing list immediately. The List-Unsubscribe header field makes it unnecessary to include an unsubscribe footer in forwarded messages. A new List-Name header field would make it unnecessary to prefix the Subject with the name of the list. Mail clients could simply prefix the Subject with the value of the List-Name header field according to the user’s preferences locally if desired. Anyone who moves messages to separate folders with filtering rules might not even be interested in this. In my opinion, DKIM signatures are broken for no good reason.

> I wonder if you'd be interested in the work happening in the GNAP working group.

Thanks for the hint, I wasn’t aware of this working group.

>> 2. SMTP for Submission should make it clear that the IP address of the submitter SHOULD NOT be included in the Received header field. In my view, the current practice of many email service providers violates the European General Data Protection Regulation (GDPR): https://explained-from-first-principles.com/email/#sender-towards-recipients <https://explained-from-first-principles.com/email/#sender-towards-recipients>
> 

Leaking the IP address of the sender to the recipient has serious implications for privacy (geolocation of the sender and tracking the sender across services) and security (denial-of-service attacks and targeted remote attacks). With broadband Internet, many households have the same IP address for months or even years. As a consequence, IP addresses are justifiably treated as personally identifiable information. Given laws such as GDPR, many lawmakers think that I’m entitled to such privacy. For similar reasons, some companies proxy remote content for their users (see https://explained-from-first-principles.com/email/#proxying-remote-content <https://explained-from-first-principles.com/email/#proxying-remote-content>), but this is pointless if they share the user’s IP address when the user hits on “reply”. And as far as reputation and domain authentication are concerned, outgoing mail servers take responsibility for my messages anyway. (To be clear: I don’t mind if companies keep logs of email submissions. There’s just no reason to share the most confidential part of such logs with all recipients.)

>> [Remote content]

I don’t have the answer either, but convincing authors of mail clients to disable remote content by default seems like a good place to start. :-) (And we already have “something better” in the form of aggregate documents: https://explained-from-first-principles.com/email/#aggregate-documents <https://explained-from-first-principles.com/email/#aggregate-documents>.)

>> 7. I don't understand why Authenticated Received Chain (ARC) is necessary or even desirable: https://explained-from-first-principles.com/email/#authenticated-received-chain <https://explained-from-first-principles.com/email/#authenticated-received-chain>
> 

Yes, I understand this, but the forwarding mail system could just add a DKIM signature using its own domain and take responsibility for the message (as mentioned by the DKIM RFC if I remember correctly). Recipients are free to trust such signatures.

> There's a discussion in the IETF generally about how to deal with the backlog of errata.  Again as I mentioned right up the top - getting people to do the boring grunt work of reading others' work…

Here is what I replied to someone from this mailing list who contacted me privately:

It would be valuable if RFCs could reflect the current (and original) sentiment of IETF members. A designation, such as “deprecated” or “no longer maintained”, could help external people like me a lot. You could also stop accepting errata for such RFCs then. And to most of my errata, I did get a response (see https://mailarchive.ietf.org/arch/search/?q=Kaspar%20Etter <https://mailarchive.ietf.org/arch/search/?q=Kaspar%20Etter>). But while it’s easy to find an RFC and its errata, it’s difficult for others to find the corresponding conversation. Automatically referencing the corresponding thread in the mail archive would go a long way towards making RFCs and their errata more accessible to outsiders.

> Clearly you've found an enormous amount of time to put all this together, and it's really excellent work.  It feels like the kind of thing that you should be getting academic credit for, to the ext…

Thank you for this compliment.

> I hope you have the time and the willingness to engage with the IETF community as a contributor and to work within the ebb-and-flow of human relationships to progress some of your ideas and your wo…

See my reply above. If someone has the capacity to mentor me, I’m confident that we can make something happen. :-)

Best regards,
Kaspar

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 24 May 2021 13:27:17 +0200

On 2021-05-23 14:15:47 +0200, Kaspar Etter wrote:
> 2. List-Name header field: Mailing lists shouldn’t rewrite the messages of
> others and break DKIM signatures in the process. Since my email address was
> rewritten due to my domain policy, I didn’t receive some of the replies to my
> original message because I didn’t subscribe to this mailing list immediately.
> The List-Unsubscribe header field makes it unnecessary to include an
> unsubscribe footer in forwarded messages.

Not really, because many MUAs don't support the List-Unsubscribe header.
So for the benefit of the users of those MUAs it still makes to include
such a footer. 

> A new List-Name header field would make it unnecessary to prefix the
> Subject with the name of the list.

Same problem. Technically the prefix in the subject is already
unnecessary. We have List-Id header which the MUA could use to filter
messages, rewrite the subject, display an avatar or whatever. But most
don't, so mailing-list administrators fall back to rewriting the
subject. I doubt a new List-Name header field would change that.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: "John Levine" <[email-masked]>
Date: 24 May 2021 10:03:14 -0400

It appears that Kaspar Etter  <[email-masked]> said:
>2. List-Name header field: Mailing lists shouldn’t rewrite the messages of others and break DKIM signatures in the process.

Sorry, but this shows some serious misunderstandings about both DKIM and mailing lists.

DKIM is a transport signature, which in this case shows that the message was sent from the author
to the mailing list system.  List apply their own DKIM signature on the mail they send.
Mailing lists have been editing messages for 40 years, long before anyone
ever thought of DKIM or DMARC.   It is a well known DMARC failure that it doesn't work with mailing lists.
Some people have tried to rewrite history and claim that it is the lists' fault but they are wrong.

The whole point of ARC is to provide recipient systems with info to help recognize when they should
ignore DMARC and deliver mail from lists and other legitimate senders that don't happen to match the
assumptions that DMARC makes.

R's,
John

---

From: Sam Varshavchik <[email-masked]>
Date: Mon, 24 May 2021 18:06:29 -0400

> doesn't work with mailing lists.

s/that it doesn't work with mailing lists//.

I'm struggling to identify some tangible value-added that DKIM/DMARC brings  
to the table.

Ostensibly, these signatures prove that the mail really comes from the  
domain it purported to come from.

Ok, that's cool, but what is the point?

I'm told that this is to block spam that forges others domains.

Splendid, but I can't help but notice that spam that makes it past my spam  
filters features a shiny signature more often than not.

Here's a small sample from today's batch. I've masked the domain to avoid  
triggering someone's OCD's spam filter:

DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=[spam domain]
                [blah blah blah]
From: "Mail-Admin courier-mta.com" <mailer-daemon@[spam domain]>
To: [email-masked]

And here's one more:

DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=[spam domain]
        [blah blah blah]
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=k1; d=[spam domain];
        [two for the price of one] 
Subject: Strange Liver-Hormone Helps You Burn Fat 20 Hoursa Day

It seems very obvious to me that DKIM/DMARC has been a complete failure,  
even ignoring mailing list-related breakage.

They had some initial success, when they were a novelty. That changed as  
soon as their implementations gained some foothold. Spam senders figured out  
that spam filters are whitelisting signed domains. Therefore, all they have  
to do is use their own domain, sign their spam, and they's whitelisted!

Pure comedy gold.

---

From: "John Levine" <[email-masked]>
Date: 24 May 2021 21:23:45 -0400

>Ok, that's cool, but what is the point?

For DKIM, the point is to have a reliable identifier for the message
that is better than an IP address. Small mailers that share an IP can
use separate signing domaisn to separate their mail streams, large
mailers can aggregate the repuation of all of their outbound IPs.

As you say, merely having a DKIM signature tells you nothing, but
after you watch a mail stream for a while, you see that some DKIM
signers send clean mail and some send lousy mail and adjust your
filters appropriately. Large mail systems all do this. We hoped that
there would be shared DKIM reputation lists like there are shared IP
lists but so far that hasn't happened.

The original point of DMARC was for B2C or B2B mail from heavily
phished domains like Paypal, that could say please discard anything
from us that fails DMARC and we understand that might be some real
mail. (All of Paypal's mail just says "something happened, look at our
web site".) It still works pretty well for that.

Unfortunately, AOL and Yahoo had separate giant security failures and
allowed crooks to steal people's address books, so spammers could take
pairs of addresses and send spam that appeared to be from a friend,
leading to huge numbers of support calls at AOL and Yahoo. They
decided to outsource the cost of their security failures to the rest
of the Internet by abusing DMARC p=reject to make that spam disappear,
along with a lot of real person-to-person and list mail. They knew
this would break every mailing list and they didn't care, according to
someone who was in the room at the time.

The people who designed DKIM and DMARC knew then and know now what it
can and can't do, but we have done a poor job of explaining them to
people who want them to be a magic FUSSP.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Mon, 24 May 2021 18:35:45 -0700

On 5/24/2021 7:03 AM, John Levine wrote:
> to the mailing list system.  

Since DKIM is often misunderstood -- and especially with beliefs that go 
far beyond what it actually does -- it is important that we be very 
careful in its description.

Although DKIM is, indeed, applied and interpreted by operators rather 
than users, it is /not/ a transport mechanism.

DKIM does not know or care about sessions or transport (for any 
definition of transport.)  DKIM is an object-level mechanism. At an 
architectural level, it is comparable to OpenPGP and S/MIME, though of 
course its semantics and algorthms are wholly different.

In technical terms, DKIM actually /can/ be created and processed by 
end-user software.  That it isn't is a matter of operational 
preferences, rather than technical design.

Further, DKIM does not say anything at all about the author.  Nothing. 
Nada.  Some signers have stringent policies that well might permit 
making assessments relative to the author, but that is fully and 
completely outside the DKIM specification.

> List apply their own DKIM signature on the mail they send.
> ever thought of DKIM or DMARC. 

Yup.

Any attempt to pretend that 'we' can dictate much to the list 'them' has 
proven to be a non-starter.  For ever.

> assumptions that DMARC makes.

Yeah.  No.  It does not say you can ignore DMARC.

Rather it says that the list that broke DMARC is reporting what their 
DMARC evaluation was.  If you trust that list's reporting, then indeed 
you can very much pay attention to DMARC.

That is, the main motivation for ARC is to provide a plausible basis for 
paying attention to DMARC, even when its underlying authentication 
mechanism are broken by the time they show up at the receiver.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Dave Crocker <[email-masked]>
Date: Mon, 24 May 2021 18:42:12 -0700

On 5/24/2021 6:23 PM, John Levine wrote:
> that is better than an IP address.

yup.

d/

---

From: "John R Levine" <[email-masked]>
Date: 24 May 2021 21:49:19 -0400

On Mon, 24 May 2021, Dave Crocker wrote:
> paying attention to DMARC, even when its underlying authentication mechanism 
> are broken by the time they show up at the receiver.

Yes. that's a better way to put it.

I asked somone from a Large Mail Provider that since they know where the 
mailing lists are, why not just whitelist them.  The answer was that a lot 
of spam leaks through lists, since they typically just check that the 
From: address is a subscriber.  ARC lets them look back and see if the 
message was DMARC aligned when it arrived at the mailing list and do the 
filtering that the mailing list didn't.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 25 May 2021 06:52:12 -0400

John Levine writes:

> filters appropriately.

To me that's not fundamentally different from filtering based on the sending  
IP address.

I do not see both bad and clean mail coming out of the same IP address,  
differing only in the sending domain. In that situation, and where the  
signatures are applied by the mail host to their users, then I could see  
this argument.

I should clarify this. I see that occasionally. But when it does, I seem to  
always end up moving my goalposts, and conclude that the mail provider  
itself is rogue, and made a business decision to go into the business of  
providing spam outsourcing services, with some non-spam mail services on the  
side. So I treat it as a bad mail source.

Ss long as mail recipients are willing to tolerate spam-friendly mail  
service providers, and relying on the domain signature to filter out their  
spamming customers, this situation will never change.

I don't accept the premise that accepts bad and clean mail coming out of the  
same IP address using "oh well just use a domain signature" as a solution.

> Large mail systems all do this. We hoped that

This is never going to happen. Domains are relatively cheap. If a domain  
acquires negative social credit it'll be discarded and replaced by a new one.

> web site".) It still works pretty well for that.

Eh, no. A large majority of user-facing mail clients are now hiding the  
sending mail address, and showing only the name, up front.

From: "Paypal Customer Service" <[email-masked]>

Most people will see "Paypal Customer Service". Valid domain signature for  
934iowero.us, and straight it goes into your Inbox.

---

From: "John Levine" <[email-masked]>
Date: 25 May 2021 14:43:03 -0400

>side. So I treat it as a bad mail source.

You and I do not run large mail systems.  The bigger you are, the less latitude
you have to punish senders (even places like Sendgrid that deserve it) if it
means also losing the real mail they send.  On my small system I have elaborate
rules to post-filter Sendgrid's mail because there is useful stuff in with the crud
and my users are sad if it disappears.

>same IP address using "oh well just use a domain signature" as a solution.

You certainly don't have to use DKIM if you don't want to, but large providers seem to feel differently.

>acquires negative social credit it'll be discarded and replaced by a new one.

Reputation goes both ways.  If a domain has a good reputation you can accept its
mail even if some of it smells sort of spammy.

>> from us that fails DMARC ...

>From: "Paypal Customer Service" <[email-masked]>

Yeah, we know.  But large providers tell me that DMARC still blocks a great
deal of phish that uses the target's actual domain name.

>934iowero.us, and straight it goes into your Inbox.

You're making the same mistake again.  DMARC is not a whitelist.  If something is
DMARC aligned, all that means is that it was really sent by the domain in the From:
header.  You still apply reputaiton and other spam filters to it.  It's not a FUSSP.

R's,
John

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 25 May 2021 17:54:58 -0400

> and my users are sad if it disappears.

And that's why Sendgrid – and everyone else in their position – has no  
incentive to clean up their act. Why should they, when most recipients on  
the other side of the earth are willing to expend their effort to accomodate  
Sendgrid – or anyone else in their position?

It used to be that they would just segregate their dirty mail from their  
clean mail on different IP addresses, and that was their preferred solution:  
go ahead, feel free to just block those guys over there, they'll continue to  
pay us anyway.

Now, they don't even have to do that. They can simply say: hey, here's the  
sender, feel free to sort the senders yourself. The burden on you to go  
through the hassle of identifying the sender, and filtering them out.

I think what's happening, in the grand scheme of things, is the elimination  
of small to medium E-mail providers. E-mail is being split between the  
extreme low end: individuals running their own E-mail for either themselves  
or their organization. They don't realy on E-mail, don't consider it  
critical, and can afford to brutally filter their incoming E-mail. They  
don't mind an occasional loss of an attempted personal contact from the  
public. They just want to avoid having their mailbox getting filled with  
junk every day.

And then we have the large E-mail providers. They can afford to invest  
resources in implementing sophisticated, score-based mail filtering  
techniques, employing DKIM, DMARC, and paying for a non-free blacklist.

A small-to-medium E-mail provider just won't have the resources to compete,  
I believe, over the long term. What we're talking about here, will /not/ be  
sufficient to clean their incoming E-mail, by itself. They'll still have to  
constantly waste time constantly adjusting their filters and figure out what  
to do with each new source of mail they haven't seen before.

> deal of phish that uses the target's actual domain name.

So does SPF, actually.

May 25 07:10:29 shorty courieresmtpd[28905]: error,relay=::ffff: 
185.222.57.81,port=56202,from=<[email-masked]>: 517 SPF fail courier- 
mta.com: Address does not pass the Sender Policy Framework
May 25 07:50:58 shorty courieresmtpd[29001]: error,relay=::ffff: 
195.140.213.254,port=54000,from=<[email-masked]>: 517 SPF fail  
bflhydro.com: Address does not pass the Sender Policy Framework
May 25 08:57:31 shorty courieresmtpd[29918]: error,relay=::ffff: 
36.92.9.83,port=7329,from=<[email-masked]>: 517 SPF fail [email-masked]: Address  
does not pass the Sender Policy Framework

SPF is heck of a lot easier to implement.

And, thinking about this more: if something will fail a DMARC check, it's  
almost a certainty that it'll fail an SPF check too. Chances are very good  
that a domain that uses DMARC will also have an SPF record.

> You're making the same mistake again.  DMARC is not a whitelist.  If

I didn't say that it was.

> something is
> header.

Right, but my point was DMARC will verify 934iowero.us, and the recipient  
sees that the mail is from "Paypal Customer Service".

And if 934iowero.us is a brand new domain, with no reputation, then what?

>        You still apply reputaiton and other spam filters to it.  It's not  
> a FUSSP.

And if it's a brand new domain and has no reputation?

So, what's the starting point here. What are we going to do with new domains  
that validate in DMARC/DKIM, but have no reputation.

A) Vote them down. This winds up with large E-mail providers keeping  
everyone off their turf. You have poor deliverability with a new domain,  
unless you pay the danegold to big E-mail providers to send from their  
infrastructure.

B) Do not vote them down. Factor in bad reputation only. Do not mark down in  
absence of no reputation. This winds up with cheap domain recycling. Spam  
the crap out of everyone using a new domain, until its reputation sinks.  
Then switch to the next throwaway domain.

My point is: the domain-based reputation approach is being pushed by large  
email providers because they don't want to terminate their paying customers,  
who are paying them for the privilege of sending spam. These large email  
providers would much rather prefer to do nothing more than an absolute  
minimum amount of work to merely identify who is sending mail from them.  
Then they can wash their hands, proclaim that their job is done; and it is  
up to you to block their spambags, if you wish. You do all the work of  
figuring out each domain's reputation.

Now, in that scenario, I'll agree, domain authentication would be very  
useful for them. But only for them, and nobody else, except for those that  
are willing to expend their own resources to filter the spam sources that  
they should not have to filter in the first place.

And as long as this situation is acceptable by everyone, I don't see the  
situation changing, since the situation is acceptable to everyone.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 25 May 2021 16:18:23 -0700

On 5/25/2021 3:52 AM, Sam Varshavchik wrote
> To me that's not fundamentally different from filtering based on the 
> sending IP address.

In its simplest terms, it isn't.  But then, simplest is not enough here.

First, IP breaks after one MTA hop and DKIM doesn't.

Second, IP mixes all sorts of traffic and DKIM doesn't (or, at least, it 
doesn't have to).  That is, DKIM can be used to highly partition and 
identify content streams.  This allows clean, accurate narrow-band 
reputation analysis.  IP allows only a very coarse reputation grain.

How signers actually use DKIM might well be different from how they 
/could/ use it, of course...

> acquires negative social credit it'll be discarded and replaced by a new 

One of the continuing, strategic challenges in anti-abuse work is that 
people who work in it necessarily have a primary focus on bad actors.  A 
collaborative mechanism -- such as DKIM, where the originating site 
literally signs up for identification and assessment -- creates a 
challenge, in that evaluating good actors is quite a different job from 
evaluating bad actors.  It's not that good actors are perfect, but that 
they are less likely to act badly and typically it won't be intentionally.

Think misdemeanor rather than felony...

So the fact that domains are cheap is less relevant than a good actor 
wanting to create a clean record of being a good actor.

> sending mail address, and showing only the name, up front.

Users are pretty much irrelevant to DMARC.  DMARC is for use by the 
receiving filtering engine.  It doesn't matter what From: field data is 
displayed to users.  (Really.  It.  Does.  Not.  Matter.)

> Most people will see "Paypal Customer Service". Valid domain signature 
> for 934iowero.us, and straight it goes into your Inbox.

Noting that operators continue to claim benefit in supporting DMARC, the 
fact that it is easy to circumvent means that its utility is tactical 
rather than strategic.  I'm not a fan of tactical (ie, limited) benefit 
in standards work, but I didn't have a vote...  More importantly, they 
claim they /do/ see real filtering benefit.

d/

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 25 May 2021 20:50:28 -0400

Dave Crocker writes:

> literally signs up for identification and assessment -- creates a challenge,  
> in that evaluating good actors is quite a different job from evaluating bad  
> actors.  It's not that good actors are perfect, but that they are less  
> likely to act badly and typically it won't be intentionally.

So what you're saying is that usage of DKIM is more indicative of a good  
actor than a bad actor.

> wanting to create a clean record of being a good actor.

I just did a rough search of my mailbox, looking at the proportion of non- 
spam mail with DKIM-Signature: field versus the spam bin.

Would you believe that the difference, between the two categories, was a  
rounding error? For both spam and non-spam mail, roughly 60% of it carried a  
DKIM-Signature.

But what was interesting is that the majority of spam without a DKIM- 
Signature were either lame advance fee fraud attempts, or a lame ransomware  
or virus infestation attempts. I find that subcategories of spam to be far  
less annoying than others.

Some percentage of advance fee fraud attempts and ransomware were from large  
mail providers that stamped a DKIM-Signature: header.

But nearly all other spam, the kind that I do have a major problem with, the  
specific type that I'm bitching about, nearly all of it carries a DKIM- 
Siganture: field. I only found very, very few exceptions to that.

So, my direct experience is that it's other way around: a presence of a DKIM- 
Signature: header indicates that this is more likely to be spam than not.  
Not by a lot, as it turns out to be, but it is distinguishing.

Now, to John's point, that DKIM alone is not indicative of reputation, that  
it only serves to ascertain identity, and with that out of the way you can  
now evaluate the proven identity's reputation. Well, the problem with that  
is twofold:

1) There are no known (at least to me) established reputation providers. And  
even if there are some that claim to be, history teaches that they don't  
really accomplish much.

2) So you're left with building and maintaining your own reputation database.

That seems like a lot of work to me. Perhaps a large mail provider might  
have resources to dedicate to this, but even then I'll be quite skeptical of  
how well it would work, for a variety of reasons.

And for non-large provider, I just don't see it working. And with that, what  
good is a proven identity, when you have no easy and low-cost way of  
evaluating the reputation of that identity? And that, in a nutshell, is the  
problem I see with proving domain identities.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 25 May 2021 18:14:34 -0700

On 5/25/2021 5:50 PM, Sam Varshavchik wrote:
>>> This is never going to happen. Domains are relatively cheap. If a 
>>> domain acquires negative social credit it'll be discarded and 
>>> replaced by a new one.
>>
>> people who work in it necessarily have a primary focus on bad actors.  
>> A collaborative mechanism -- such as DKIM, where the originating site 
>> challenge, in that evaluating good actors is quite a different job 
>> from evaluating bad actors.  It's not that good actors are perfect, 
>> but that they are less likely to act badly and typically it won't be 
>> intentionally.
> 

Actually, no, that's not what I said.  Bad actors are always the first 
to adopt the newest anti-spam technologies, to abuse those unfortunates 
who interpret DKIM the way you described.

DKIM establishes a clean (noise-free) channel from the signer, which 
means that any assessment about them really is about them.  If they are 
bad actors, that is a lot easier to assess, as is if they are good actors.

> I just did a rough search of my mailbox, looking at the proportion of 
> non-spam mail with DKIM-Signature: field versus the spam bin.

cf, above, about bad actors.

> But nearly all other spam, the kind that I do have a major problem with, 
> the specific type that I'm bitching about, nearly all of it carries a 
> DKIM-Siganture: field. I only found very, very few exceptions to that.

For those assessed as bad actors, was any of their mail mixed in with 
mail from a different signer who was assessed to be a good actor?

That differentiation is the value DKIM can provide.  It eliminates or 
reduces noise.

> Now, to John's point, that DKIM alone is not indicative of reputation, 
> that it only serves to ascertain identity, and with that out of the way 
> you can now evaluate the proven identity's reputation. Well, the problem 
> with that is twofold:
> 
> 1) There are no known (at least to me) established reputation providers. 
> And even if there are some that claim to be, history teaches that they 
> don't really accomplish much.

Gosh, you mean that each evaluator needs to formulate their own 
criteria, about a complex, fuzzy topic?  Yup!

> That seems like a lot of work to me.

It is.  Sad reality.  Lot of criminals on the streets make safe 
navigation challenging.  Most people need to outsource their safety efforts.

d/

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 25 May 2021 22:11:54 -0400

Dave Crocker writes:

> Actually, no, that's not what I said.  Bad actors are always the first to  
> adopt the newest anti-spam technologies, to abuse those unfortunates who  
> interpret DKIM the way you described.
>
> DKIM establishes a clean (noise-free) channel from the signer, which means  
> that any assessment about them really is about them.  If they are bad  
> actors, that is a lot easier to assess, as is if they are good actors.

Ah, but the first paragraph's the rub. That's why I saw DKIM-Signature: as a  
spam indicator: the bad actors' initial take-up of DKIM-Signature: was quite  
noticable.

That was definitely true at one point. Based on today's numbers that I  
looked at the mainstream adoption of DKIM sadly diluted its early value as a  
spam indicator, ironically.

> For those assessed as bad actors, was any of their mail mixed in with mail  
> from a different signer who was assessed to be a good actor?

My sample wasn't large enough for that. I have no recollection of seeing  
this; except I have a dim recollection of receiving something non-spam from  
Sendgrid a very, very long time ago, before I wrote them off as damaged  
goods.

Interestingly enough, while researching this response, I found a copy of a  
sendgrid-sourced spam from December 2020, from a previously unknown (to me)  
IP address range (it was spamming an SMS-spam service). It did not have a  
DKIM/DMARC signature of any kind. Nothing from Sendgrid since then until  
today, when Sendgrid attempted to spam one of my Sourceforge mailing lists,  
with a monstrous DKIM-signed spam in Spanish.

So, looks like Sedngrid is
> Gosh, you mean that each evaluator needs to formulate their own criteria,  
> about a complex, fuzzy topic?  Yup!
>
>>
> It is.  Sad reality.  Lot of criminals on the streets make safe navigation  
> challenging.  Most people need to outsource their safety efforts.

You can't really have both. Either you "formulate your own criteria", or  
you'll outsource your spam filtering.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 25 May 2021 19:19:44 -0700

On 5/25/2021 7:11 PM, Sam Varshavchik wrote:
>> to adopt the newest anti-spam technologies, to abuse those 
>> unfortunates who interpret DKIM the way you described.
>>
>> means that any assessment about them really is about them.  If they 
>> are bad actors, that is a lot easier to assess, as is if they are good 
>> actors.
> 
> Ah, but the first paragraph's the rub. That's why I saw DKIM-Signature: 
> as a spam indicator: the bad actors' initial take-up of DKIM-Signature: 
> was quite noticable.
> 
> looked at the mainstream adoption of DKIM sadly diluted its early value 
> as a spam indicator, ironically.

You seem to have fixated on using the existence of a signature as 
indicating goodness or badness.  Since it isn't intended to do that, 
please stop casting the issue in terms of whether it accomplishes that.

One more time:

      1)  DKIM creates a noise free channel of mail associated with the 
signing identifier.

      2) A noise-free channel permits accurate assessment of the actor 
associated with the identifier.

      3) Assessment is a separate process from identification; DKIM does 
identification.

> you'll outsource your spam filtering.

Well, actually, you CAN do both, if you want.  But whether you can or 
not is irrelevant to any of the points I was making...

d/

---

From: Matthias Leisi <[email-masked]>
Date: Wed, 26 May 2021 09:16:40 +0200

> 2) So you're left with building and maintaining your own reputation database.

That’s the sort of problem we are trying to solve as a collaborative, better scalable, solution at dnswl.org <http://dnswl.org/>. But uptake by users of positive domain reputation has been relatively poor so far, even though I believe it would be beneficial especially for small-ish sending and receiving systems.

— Matthias (with the dnswl.org <http://dnswl.org/> maintainer hat)

---

From: Sam Varshavchik <[email-masked]>
Date: Wed, 26 May 2021 11:40:58 -0400

Dave Crocker writes:

> You seem to have fixated on using the existence of a signature as indicating  
> goodness or badness.  Since it isn't intended to do that, please stop  
> casting the issue in terms of whether it accomplishes that.

I understand, very well, what the intent of DKIM is.

However, it is what it is. If I observe what the practical reality of the  
past or current usage of DKIM is, then this speaks for itself.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 26 May 2021 08:44:33 -0700

On 5/26/2021 8:40 AM, Sam Varshavchik wrote:
> However, it is what it is. If I observe what the practical reality of 
> the past or current usage of DKIM is, then this speaks for itself.

Your comments seem to demonstrate an assessment of what isn't.

d/

---

From: "John Levine" <[email-masked]>
Date: 26 May 2021 16:22:58 -0000

>past or current usage of DKIM is, then this speaks for itself.

I honestly do not undersatand what your point is.  It sounds like you're saying
that since it is not a magic FUSSP, it must be useless, but that would be dumb
so I doubt that's it.

For example, do you believe that people whitelist mail using DKIM
signatures from signers with good reputations?  Because we do.

R's,
John
-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Sam Varshavchik <[email-masked]>
Date: Wed, 26 May 2021 16:28:01 -0400

> signatures from signers with good reputations?  Because we do.

I'm sure that's the case.

But I believe I made my point clear: that (the same) people can end up using  
the DKIM in exactly the opposite manner, for the simple reason that they see  
DKIM used more by unwanted mail, than not, and I explained why that turned  
out to be: a much higher adoption rate of DKIM by mail outsourcing providers  
than by other users. And when you are seeing most of your bad mail come from  
third party mail providers, then the combination of the two factors turns  
DKIM into a spam indication.

Again, I am not passing judgement or arguing DKIM's lofty goals. I  
understand what they are, what DKIM is, and its purported use cases. It's  
really no different than judging the presence of

X-PHP-Originating-Script: 33:SimpleMailInvoker.php

or

X-Mailer: PHPMailer

as also a high spam indication. I am struggling to find any message in my  
archive with those two headers that's not spam. Does this simple observation  
pass any judgement on those two fine products' stated use cases? No, it does  
not. But, again, in the real world things don't always work according to  
plan. And, all I'm saying, is that some may find DKIM-Signature: to be  
awfully similar to those two.

---

From: Nathaniel Borenstein <[email-masked]>
Date: Wed, 26 May 2021 16:36:50 -0400

> DKIM used more by unwanted mail, than not,

It seems to me you can make the same claim for almost any feature of email
(or anything else) that is used by both the good and bad guys.  "Most of
the mail we receive that uses multipart/related comes from good/bad guys,
so I will/won't block it."  The fact that others are doing things stupidly
isn't a good reason for the rest of us not to do them intelligently.  If it
were, it would be time to give up on email altogether.

On Wed, May 26, 2021 at 4:28 PM Sam Varshavchik <[email-masked]>
wrote:

> the
> > I honestly do not undersatand what your point is.  It sounds like
> you're
> > saying
> > that since it is not a magic FUSSP, it must be useless, but that would
> be
> > dumb
> > so I doubt that's it.
> >
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 May 2021 23:28:34 -0400

> It seems to me you can make the same claim for almost any
> feature of email (or anything else) that is used by both the
> good and bad guys.  "Most of the mail we receive that uses
> multipart/related comes from good/bad guys, so I will/won't
> block it."  The fact that others are doing things stupidly
> isn't a good reason for the rest of us not to do them
> intelligently.  If it were, it would be time to give up on
> email altogether.

FWIW, a very strong +1.

IMO, we should be specifying thing to work in the best way
possible.  I suspect we could probably do a better job of
explaining decisions and the tradeoffs that went into them so as
to reduce the odds that good people will make bad decisions in
the hope of avoiding the evildoers.  Explaining particular
decisions will probably serve us better, at least with those who
want to Do the Right Thing than taking positions that are too
easily interpreted as "because we said so".  

best,
   john

---

From: Kaspar Etter <[email-masked]>
Date: Thu, 27 May 2021 17:55:09 +0200

This conversation went off on a tangent, but I also want to comment on a couple of things:

> On 24 May 2021, at 16:03, John Levine <[email-masked]> wrote:
> 

I disagree. My impression is that some people think that the world can be divided into good guys and bad guys and that email is supposed to work as a cooperation among well-intended postmasters. I respect this view, but I also think it’s naive and outdated. IT security is about minimizing trust and attack surfaces. Domain authentication with SPF, DKIM, and DMARC is an important step in this direction. Domain authentication isn’t enough to prevent phishing, of course, since homograph attacks and malicious display names are still possible, but it allows mail clients to put only messages from approved senders into the user’s inbox. HEY is the only mail client I know of which does this (see https://www.hey.com/features/the-screener/ <https://www.hey.com/features/the-screener/>), but I’m confident that we’ll see similar features in other clients in the future. I believe that mechanisms like this and showing only the display names of approved senders would go a long way in fighting phishing – but only in combination with domain authentication. Other efforts to curb phishing, such as BIMI, also require domain authentication. (BIMI also addresses homograph attacks with verified mark certificates.)

One of the nice things about DMARC is that it makes domain authentication opt-in for those who want this (even if the reason for this feature is mainly backward compatibility). I’m honestly surprised to see that almost no one here uses DMARC with a policy of reject or quarantine, but I don’t mind this because it’s your call. What I do mind is when people deny the legitimacy of other viewpoints. I don’t care whether it was a conscious design decision that the sender of a message can be spoofed. “It appears that someone said” is simply not how I want to use email, and given that email is an open system which I’m forced to use, I think that I should be allowed to have this view.

Making it easier for mailing lists to keep DKIM signatures intact wouldn’t force anything upon anyone. If mailing lists prefer to rewrite the sender address, they’re free to do so. (I acknowledge that a DMARC policy other than none forces mailing lists to do something, but backward compatibility shouldn’t be sacred and this ship has sailed anyway.) ARC, on the other hand, subverts the little progress towards trust minimization that some of us gained with DMARC. For my email addresses, I’m the only legitimate sender. I don’t want to be obliged to trust people I have no business with not to impersonate me. If ARC gains adoption, the servers of reputable mailing lists will be a prime target for scammers. ARC only makes sense with a good guys versus bad guys mentality in a world with no IT vulnerabilities but not with a security mindset. Introducing a DMARC policy of “reject-unless-arc-trusted”, as mentioned by Bron Gondwana, might be a good compromise between the two camps. For the reasons I just mentioned, I wouldn’t use this policy on my domains, though.

On a different note, the problem with malicious display names is much worse than many people are aware: Most of the mail clients which display only the display name do so even if the display name is itself an email address. Emails from `"[email-masked]" <[email-masked]>` are displayed identically to emails from [email-masked] <mailto:[email-masked]>`. See https://explained-from-first-principles.com/email/#malicious-display-names <https://explained-from-first-principles.com/email/#malicious-display-names> for more context.

---

From: "John R Levine" <[email-masked]>
Date: 27 May 2021 12:23:10 -0400

On Thu, 27 May 2021, Kaspar Etter wrote:

> I disagree.

You're welcome to your opinion, but you don't get to rewrite 40 years of 
history.  We've spent a decade with people insisting that the entire 
e-mail world has to change the way it works to conform to the lastest 
FUSSP.  We saw it with SPF, we're seeing it with DMARC.  It didn't hapen 
with SPF, and it's not happening with DMARC.  For example, mailing lists 
have been editing messages for a long time, they have good reasons to do 
so, and they're not going to stop.

I'm not guessing about the reasons for ARC, I know the people who 
developed it and I believe I am correctly describing the reason they 
invented it.  It includes some of the largest mail providers in the world. 
They all use DMARC, and they understand that DMARC's limitations cause a 
lot of gratuitous pain for their users who've been using mailing lists for 
a long time.  It's not a large fraction of the total amount of mail, but 
it's a large fraction of the mail they care about.

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 27 May 2021 12:39:19 -0400

PS:

On Thu, 27 May 2021, Kaspar Etter wrote:
> On a different note, the problem with malicious display names is much 
> worse than many people are aware: ...

I can assure you that everyone here is quite aware of the issue.  We're 
also aware that a large number of mail users pay no attention to the 
sender's name at all.  If you send a message from Evil Russian Criminal 
telling people to click on this link to verify your Paypal payment, lots 
of people will click anyway.

Regards,

---

From: Kaspar Etter <[email-masked]>
Date: Thu, 27 May 2021 19:15:16 +0200

On 27 May 2021, at 18:23, John R Levine <[email-masked]> wrote:
> We've spent a decade with people insisting that the entire e-mail world has to change the way it works to conform to the lastest FUSSP. 

Why is domain authentication framed as a spam prevention technique? Any messaging service which is popular, open, and free will have spam. Spam is a problem of quantity: You just have to bring down the amount of unsolicited messages to a bearable level, be it with domain or IP reputation, challenge-response mechanisms, or proof of work. Phishing, on the other hand, is a problem of quality: A single successful attack can do immense harm. It’s not just large organizations which are being impersonated. A popular scam is to impersonate the victim themself, claiming that their account has been compromised and blackmailing them into paying a ransom.

Just because some people will always fall for scams doesn’t mean that we shouldn’t try to reduce the number of victims. Otherwise, cars wouldn’t need safety measures because some people will always die in car accidents. Priming plays a huge role in human psychology and there’s a lot that mail clients could do in this regard: Separate messages from unknown senders, don’t display the display name of unknown senders, warn users when they click on links from unknown senders, warn users if a previously authenticated sender could not be authenticated, etc.

> They understand that DMARC's limitations cause a lot of gratuitous pain for their users who've been using mailing lists for a long time.  

I fully understand this pain and respect the motivation behind ARC, but you cannot have (strict) domain authentication and message rewriting. I want the former and don’t care about the latter. Maybe the solution will be that people use two different addresses: One with domain authentication enabled for direct conversations and one without domain authentication for mailing lists.

---

From: "John R Levine" <[email-masked]>
Date: 27 May 2021 13:20:47 -0400

> Why is domain authentication framed as a spam prevention technique?

Don't take FUSSP too literally.

> I fully understand this pain and respect the motivation behind ARC, but 
> you cannot have (strict) domain authentication and message rewriting.

Right.  So since sensible mail systems deliver the mail that their users 
want, strict domain authentication loses.

Regards,

---

From: Richard Clayton <[email-masked]>
Date: Fri, 28 May 2021 09:27:13 +0100

In message <[email-masked]>, Kaspar Etter
<kaspar=[email-masked]> writes

>Other efforts to curb phishing, such as 
>BIMI, 

You will not find any claim on the BIMI website that says that it will
curb phishing (and, in my view, with good reason)

>also require domain authentication. (BIMI also addresses homograph attacks 
>with verified mark certificates.)

It has always been unclear how similar looking trademarks (indeed
identical ones, since trademarks are only unique within a single
economic sector within a single jurisdiction) will be handled by BIMI

... but while only a small number of brands, mainly from one country,
are the only users of the mechanism this is not an issue that they have
to resolve so far

>One of the nice things about DMARC is that it makes domain authentication opt-in 
>for those who want this (even if the reason for this feature is mainly backward 
>compatibility). I’m honestly surprised to see that almost no one here uses DMARC 
>with a policy of reject or quarantine, but I don’t mind this because it’s your 
>call.

you'd be even more surprised if you started to investigate how few
mailbox providers honoured a reject request

>On a different note, the problem with malicious display names is much worse than 
>many people are aware: Most of the mail clients which display only the display 
>name do so even if the display name is itself an email address. Emails from 
>`"[email-masked]" <[email-masked]>` are displayed identically to emails from 
>[email-masked] <mailto:[email-masked]>`. See https://explained-from-first-
>principles.com/email/#malicious-display-names <https://explained-from-first-
>principles.com/email/#malicious-display-names> for more context.________________

I think most people who have thought about how to tackle the issue of
misleading display names are well aware of this relatively simple issue,
it's all the other complexity that has hindered attempts to tackle the
issue...

... and of course the real issue with phishing is that systems are
(still being) built with the assumption that end users will be able to
reliably identify legitimate communications and disregard the messages
from the bad guys. Since that will never be true, any system which
relies on the assumption will fail.  viz: trying to help the user make
better decisions is merely a sticking plaster

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 28 May 2021 11:33:37 +0200

On Thu 27/May/2021 17:55:09 +0200 Kaspar Etter wrote:
> Making it easier for mailing lists to keep DKIM signatures intact wouldn’t 
> force anything upon anyone. If mailing lists prefer to rewrite the sender 
> address, they’re free to do so.

Mailing lists often do keep DKIM signatures intact.  They only alter the rest 
of the message, but the transformations they do are stylized enough to allow to 
revert them and verify the original signatures.  That requires just some 
cooperation, because DKIM is so flexible.  Both the mailing list and its 
posters can operate so as to make reversion possible or not.

For example, Kaspar's message that I'm replying to could not be reverted 
because the original signature by ef1p.com signed MIME-Version: and 
Content-Type:.  Mailing lists need to control the values of those fields.  To 
allow this kind of interoperability, an author's domain shouldn't sign them.

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 28 May 2021 14:18:19 -0400

It appears that Alessandro Vesely  <[email-masked]> said:
>revert them and verify the original signatures. ...

We've been around this barn a few times before. What you say may seem
true if you have only seen a few lists handled by a single list
package. In reality, I have seen lists add headers and footers, add,
delete, and reorganize MIME parts while changing the MIME boundary
strings, rewrite URLs in message bodies, and flatten HTML bodies into
text. There is no way to list all of the common changes, much less
describe how to reverse them.

In any event, whatever you might want to do by reviving DKIM signatures is already
handled by ARC, so it'd be a lot more productive to encourage ARC adoption than
to invent yet another half-hearted DMARC workaround.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 29 May 2021 11:58:24 +0200

> describe how to reverse them.

There are also lists that gave up making any change in order to not break 
DMARC.  Some other lists cannot afford that, and turn to From: rewriting.  If 
they can manage to restrain from irreversible changes, the original From: can 
be safely restored.

Reverting From: rewriting is a practical approach which works for cooperating 
lists and cooperating author domains.

> to invent yet another half-hearted DMARC workaround.

ARC is a completely different thing.  It introduces a third party, the sender, 
which modifies the message header in such a way to induce receivers to accept a 
message even if it fails DMARC.  Semantically, it is very similar to Dave's Use 
of the Sender Header Field.

ARC is good for global providers who are able to rate intermediate senders.

I see no conflict between these two methods.

Best
Ale
--

---

From: "John R Levine" <[email-masked]>
Date: 29 May 2021 10:23:22 -0400

On Sat, 29 May 2021, Alessandro Vesely wrote:
> ARC is a completely different thing.  It introduces a third party, the 
> sender, which modifies the message header in such a way to induce receivers 
> to accept a message even if it fails DMARC.  Semantically, it is very similar 
> to Dave's Use of the Sender Header Field.

Ho, it is really nothing like that.  It provides log info to so that the 
final recipient can retroactively do filtering that the intermediate hosts 
didn't.  It is true that you do have to know which intermediate hosts 
might plausibly do that, but small systems rarely get ARC'ed mail from 
more than a handful of places.

> I see no conflict between these two methods.

Just wondering, but has anyone implemented the mutant DKIM you are 
proposing?  Is there a spec or an I-D?

Regards,

---

From: Dave Crocker <[email-masked]>
Date: Sat, 29 May 2021 08:32:13 -0700

On 5/29/2021 7:23 AM, John R Levine wrote:
>> sender, which modifies the message header in such a way to induce 
>> receivers to accept a message even if it fails DMARC.  Semantically, 
>> it is very similar to Dave's Use of the Sender Header Field.
> 
> final recipient can retroactively do filtering that the intermediate 
> hosts didn't.  It is true that you do have to know which intermediate 
> hosts might plausibly do that, but small systems rarely get ARC'ed mail 
> from more than a handful of places.
> 

Me either.

One says "Here is the original author information."

The other says "1) When I got this message, here is how the DMARC 
evaluation came out, and you can trust these results if you trust me; 
and 2) Here is the collaborative handling sequence since that evaluation"

Very different semantics.

d/

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 30 May 2021 16:59:20 +0200

On Sat 29/May/2021 16:23:22 +0200 John Levine wrote:> On Sat, 29 May 
2021, Alessandro Vesely wrote:
> proposing?

I added that to zdkimfilter.

>  Is there a spec or an I-D?

https://datatracker.ietf.org/doc/draft-vesely-dmarc-mlm-transform/

Best
Ale
--