Subject: Re: [ietf-smtp] [Shutup] Proposed Charter for the "SMTP Headers Unhealthy To User Privacy" WG (fwd)

From: "John Levine" <[email-masked]>
Date: 30 Nov 2015 04:28:19 -0000

>> Spam filters have been doing Received chain analysis for about 20 years.
>
>Yes, I know.   A friend of mine founded a company that worked using this principal.   Unfortunately, it got less and less effective as
>spammers got better and better at faking things.   The reason I asked for recent experience is that I'm curious if anyone is _still_
>getting real benefit from this.

Yes.  See the message you just replied to.

>Since the only header-field you can actually trust is the first one that your own MTA adds, ...

No, that's not correct.  See recent message.

SPF works just as well (actually, a _lot_ better) as a validation mechanism.

What?  Header chains say "this message came through this IP".  SPF
says "I assert that these IPs are allowed to send mail for this
domain."  They're completely unrelated.

We don't use header chains for validation, we use it to figure out who
to blame, who to alert, and who to block.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 30 Nov 2015 05:52:20 -0000

>  I can't see how what you described could
>possibly scale to anything a large email provider would ever do, ...

This argument is going into the weeds.  The question is whether the
information in Received headers is useful for mail system management
(which is not a synonym for spam filtering) and your apparent
assertion that it only matters what giant mail systems do is pretty
creepy.

> Do you seriously think that Google has special-case header parsing to deal with spam from Cornell students' infected computers? No, they just use machine learning.

Unfortunately, due to NDA's I can't talk about Google's spam
filtering.

>SPF allows me to discard all messages that claim to be from domain X but come from IP addresses not listed for domain X,

Yeah, we know what SPF does, and the many and wonderful ways that it
doesn't quite do what it's supposed to.  But since nobody has ever
said that we use Received headers for sender authorization, there's
still no point here.

> And if the site _is_ trustworthy, then modulo a few small exceptions
> like Cornell, it's not originating anything that can be reasonably
> identified as spam, because if it could have been reasonably
> identified as spam, it would never have been forwarded.

Aw, come on.  I get plenty of spam from Gmail and Yahoo, all of which
is 100% SPF, DKIM, and DMARC compliant and has 100% real Received
headers.  Unless you are extremely unusual, you do too.  Crooks sign
up for public mail systems to send spam, and on mail systems of all
sizes they steal or guess AUTH credentials to spam through compromised
accounts, or compromise web servers and spam through buggy old
drupal and wordpress setups.

>I don't know who "we" is here.   Is this really how Google, et al., operate?

I can't talk about Google, but I can tell you from direct experience
working around their DMARC damage that some other large mail systems
use Received headers as part of their spam filtering process, which is
not just checking sender authorization.

Some of us go to conferences like MAAWG where we spend a lot of time
talking to people who run all sorts of medium and large mail systems,
and other conferences where we talk about security problems and
responses to them with various combinations of cops and nerds.  It's
OK that you're not as familiar with all this stuff, just as I am not
as familiar with a lot of the DNS and IPv6 stuff you do.  But you
might consider the possibility that we actually do know something
about the areas in which we work.

R's,
John

---

From: Richard Clayton <[email-masked]>
Date: Mon, 30 Nov 2015 12:04:36 +0000

In message <[email-masked]>, Ted
Lemon <[email-masked]> writes

>Do 
>you seriously think that Google has special-case header parsing to deal with 
>spam from Cornell students' infected computers?   No, they just use machine 
>learning.

... and one of the things that the ML will be processing will be the
(tokenised contents of the) header fields... so having a pattern (of any
kind) within the header fields has the potential to be extremely helpful
in distinguishing good from bad

>SPF allows me to discard all messages that claim to be from domain X but come 
>from IP addresses not listed for domain X, which means that I never have to 
>write a Received: header for that message.

It rather escapes me how one of your users will be able to determine
whether you received the email from a domain which had SPF at the time
at which you received it unless you record that information along with
the email (or do you think that DNS results are constant for all time?)

If you're relaying the email on to somewhere else then you're assuming
that there's a mechanism by which your policy regarding SPF becomes
known to those other people.

I'm unaware of such a mechanism existing at the moment -- and (this
might be relevant to charters, albeit I really don't think that there's
any real mileage in this topic at all) that although we have a scheme
for originators of email to publish policy recommendations about the
handling of email from their domains we don't have a similar policy for
relaying machines (and absent a Received header field it's unclear to me
what indicator we'd use to look up what that policy was).

>   If there is no SPF for the domain 
>that sent the message, I would like to just discard it as spam, but that's not 
>safe to do because so many small sites don't implement SPF or get it wrong.   
>But in any case where there is no SPF record, the site is definitely not 
>trustworthy:

that's a shame, I consider myself very trustworthy and I've never
bothered with SPF :-(

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

From: Ted Lemon <[email-masked]>
Date: Mon, 30 Nov 2015 04:46:14 +0000

> Yes.  See the message you just replied to.

The experience you related sounds like hobbyist activity, no offense.   I can't see how what you described could possibly scale to anything a large email provider would ever do, and indeed it's not something I would ever take the trouble to do, even though my domain has exactly two users.   If I get spam, I delete it.   I do not complain about it.  I do not attempt to understand where it originated.

> No, that's not correct.  See recent message.

I saw the recent message, and again, what you described is something that doesn't scale.   If that is your model for how header-field messages are used for validation, I think what I said is actually more generally accurate.   Do you seriously think that Google has special-case header parsing to deal with spam from Cornell students' infected computers?   No, they just use machine learning.

> domain."  They're completely unrelated.

SPF allows me to discard all messages that claim to be from domain X but come from IP addresses not listed for domain X, which means that I never have to write a Received: header for that message.   If there is no SPF for the domain that sent the message, I would like to just discard it as spam, but that's not safe to do because so many small sites don't implement SPF or get it wrong.   But in any case where there is no SPF record, the site is definitely not trustworthy: I cannot rely on the Received: header fields it included in the message.   And if the site _is_ trustworthy, then modulo a few small exceptions like Cornell, it's not originating anything that can be reasonably identified as spam, because if it could have been reasonably identified as spam, it would never have been forwarded.   So I can't see how the experiences you are describing would motivate the inclusion of something like an unobfuscated Received: header field across organizational boundaries if SMTP were being designed today.

> to blame, who to alert, and who to block.

I don't know who "we" is here.   Is this really how Google, et al., operate?   It sounds like something people like you and me used to do ten years ago, that some of us old-timers are still doing (I only stopped using spamassassin recently, for example).   I would be okay with being proven wrong, but the fact that you are still doing this isn't persuasive to me.

--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[email-masked]

---

From: Ned Freed <[email-masked]>
Date: Mon, 30 Nov 2015 07:19:09 -0800 (PST)

> I saw the recent message, and again, what you described is something that
> doesn't scale.

I'm afraid again you're demonstrating that you really have no idea how this
stuff works at scale.

It is of course the case that Received: field analysis that depends on
knowledge of the domains involved doesn't scale to handle huge numbers of
domains. But that's irrelevant: Nobody cares about smalldomain.com's Received:
fields in this context, for any value of smalldomain.com.

The present day reality is that a significant fraction of mail originates
from a small number of domains. Those are the ones you care about. They're
the ones you care about, and since you can verify that mail actually came
from one of them through other means, you now have Received: fields you
can trust for a significant fraction of the mail you receive.

> If that is your model for how header-field messages are used for validation, I
> think what I said is actually more generally accurate.

Where on earth did you get the idea that Received: fields are used for
validation?

> Do you seriously think that Google has special-case header parsing to deal with

I have no idea what Google does, but I can assure you Received: field analysis
does play a role in spam filtering elsewhere. And even if it didn't, there's
more to mail operations than spam filtering.

				Ned

---

From: Al Iverson <[email-masked]>
Date: Mon, 30 Nov 2015 10:11:17 -0600

> more to mail operations than spam filtering.

I work for an email service provider dealing with rather large volumes
of mail, yet Received header analysis still plays an important role
for us in troubleshooting operational issues. It would be a tremendous
loss to lose received headers or the IP address information in those
headers. You raise Google as an example; Gmail occasionally seems to
delay message delivery. The internal received hops in a Gmail header
tell you that it's Gmail, not the sender, delaying that delivery of
the message, because the message is likely queued on some internal
server for some period of time, after sender's initial handoff.

Some of us do actually process this data at some level of scale to
identify issues.

Not only do I seriously believe it, but I have also personally
observed AOL to use special-case header parsing to deal with certain
types of mail, to more directly answer your question.

Regards,
Al Iverson

--
Al Iverson - Minneapolis - (312) 275-0130
Simple DNS Tools since 2008: xnnd.com
www.spamresource.com & aliverson.com

---

From: Steve Atkins <[email-masked]>
Date: Mon, 30 Nov 2015 08:17:39 -0800

> On Nov 30, 2015, at 8:11 AM, Al Iverson <[email-masked]> wrote:
> 

I've heard AOL people discuss "second received header parsing", and
effective filtering of infected dorm computers behind University smarthosts
was one of the use cases. This isn't something speculative, it's something
that's been in use for years.

Cheers,
  Steve

---

From: "John Levine" <[email-masked]>
Date: 30 Nov 2015 16:21:24 -0000

>I'm unaware of such a mechanism existing at the moment --

That's more or less what Authentication-Results and the deprecated
Received-SPF headers do, recording a snapshot of what the SPF results
were at the time a message was relayed. The ARC proposal in
draft-andersen-arc-00 adds a signed chain of A-R trace headers.

But again, SPF and A-R and ARC don't do the same thing as Received, which
is why we have different headers for them.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Mon, 30 Nov 2015 08:46:42 -0800

> Received-SPF headers do, recording a snapshot of what the SPF results
...

Probably relevant, current work:

     Authenticated Received Chain (ARC)
     https://tools.ietf.org/html/draft-andersen-arc-00

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Ted Lemon <[email-masked]>
Date: Mon, 30 Nov 2015 22:51:28 +0000

> in distinguishing good from bad

Very true.   And it's likely that there is some percentage, P, of messages that are prevented from being delivered because that information is in the Received: header field.   If P is 99, obviously we'd better not take that information out of the Received: header field.   If it's .0001, then this isn't a good argument for retaining the information.

> the email (or do you think that DNS results are constant for all time?)

Anything that involves a volitional act on the part of some person is an exception.   The interesting question is, does SPF provide me any greater effectiveness in my system's automatic behaviors.   The answer is yes.

> known to those other people.

Why would I be relaying mail on?   Only for a mailing list.   For the mailing list, what I want SPF to validate is that the mail came from the mailing list.   I'm relying on the mailing list provider to filter spam in this case; any secondary spam filtering that I do on the tail end is going to be compromised by the loss of information when the mail goes through the list relay.   Still might be worth it, but what is the value of P in this case, for a general solution, as opposed to a one-off heuristic?

> that's a shame, I consider myself very trustworthy and I've never

Right, and because you don't use SPF, I can't whitelist you, because anybody can send me mail claiming to be from you, and I have no prima facie basis for rejecting it (unless you use DKIM, I suppose).  The problem isn't that _you_ aren't trustworthy: it's that your system for delivering mail to me isn't.  I could maybe study your headers over time and come up with some heuristic or machine-learning system that would distinguish between you and an imposter, but you're making me do a hell of a lot of work when setting up SPF is really quite easy.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 30 Nov 2015 15:02:37 -0800

On 11/30/2015 2:51 PM, Ted Lemon wrote:
> Why would I be relaying mail on?   Only for a mailing list.   For the mailing list, what I want SPF to validate is that the mail came from the mailing list. 

That's a common misconception.  It leaves out a variety of other, valid
scenarios.

The most obvious is mailbox aliasing, such as for vanity addresses such
as university alumni associations provide.

Email is store and forward, and this can and does mean transit across
/multiple/ independent administrative domains.  The fact that the vast
majority of mail goes directly (one hop) from origin AD to the
recipient's AD does not mean it is reasonable for anyone to make
systemic design decisions that constrain that fundamental flexibility.

IMO an essential design benefit in many/most aspects of Internet
technologies is avoiding making any more global assumptions (or
requirements) than essential.  "Deferring to the end systems" is a very
broad-based design requirement and it includes minimizing assumptions
about the transit infrastructure.

d/

---

From: Ted Lemon <[email-masked]>
Date: Mon, 30 Nov 2015 23:06:17 +0000

> as university alumni associations provide.

That's a mailing list with one subscriber.   Same scenario.

> systemic design decisions that constrain that fundamental flexibility.

Sure, but in order for this argument to apply to the present discussion, we would have to be able to show that what is being proposed would constrain that fundamental flexibility, and I don't think it does.

> about the transit infrastructure.

Sure, but in this case wouldn't deferring to the end systems argue in favor of allowing end systems to make the decision as to whether their private information should be exposed?

---

From: "John Levine" <[email-masked]>
Date: 1 Dec 2015 02:03:55 -0000

>Sure, but in this case wouldn't deferring to the end systems argue in favor of allowing end systems to
>make the decision as to whether their private information should be exposed?

You can expose whatever you want, but if you expect anyone to accept
your mail, you will follow the rules set by the recipient systems.
This has been the way that mail has worked for many decades, and it's
not going to change now.

For a lot of the changes proposed here, the practical effect would be
that anyone foolish enough to implement them would find that the set
of systems accepting their mail shrinks dramatically.  For anyone
planning self-righteous complaints that you HAVE to accept my mail
because I'm following some standard or other, don't bother.  See, for
example

https://jl.ly/Email/compliant.html

R's,
John