Subject: [ietf-smtp] FWD: IETF Policy on dogfood consumption or avoidance - SMTP version

From: John C Klensin <[email-masked]>
Date: Sun, 15 Dec 2019 16:24:32 -0500

In the likely event that not all readers of the ietf-smtp list
are following [email-masked], I am forwarding the following for
your information.  People will do what they decide they need to
do, but my hope is that we can separate the discussions of what
5321 (or 5321bis) says or ought to say, technical issues with
error responses, etc., from questions about why the Secretariat
was told to reject IP literals,whether whatever process produced
those instructions is appropriate going forward and, if not, how
the relationships should be adjusted.

>From my point of view, we should not ask the Secretariat to make
changes to the instructions they were given on the basis of
discussion on this list, even if a clear consensus emerges.
Instead, either whatever "leadership" decisions produced those
instructions need to change them or the IETF community needs to
fix those mechanisms.  Those issues are, IMO, a matter for the
IESG and the IETF list (or, if needed, late feedback to the
Nomcom).

   john

---------- Forwarded Message ----------
Date: Sunday, December 15, 2019 16:15 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], [email-masked]
Subject: IETF Policy on dogfood consumption or avoidance - SMTP
version

Hi.

It has long been my personal belief that, in its operation of
various of its own services on the Internet the IETF should
adhere closely to its own standards.  If we do not do so, we
lose all credibility in recommending to others that they follow
our standards.  This practice has been referred to in many
discussion threads over the years as "eating our own dog food".  

It has recently come to the attention of several of us, via an
extended discussion on the SMTP list, that the IETF email
servers are rejecting all SMTP connections whose EHLO commands
contain IP address literals.   While the text describing the
appropriateness of use of IP literal is RFC 5321 is more
complicated than it probably ought to be, the discussion in
Section 4.1.4 of that document seems quite clear that an SMTP
server MUST NOT reject a message simply because an IP address
literal (or a domain name that does not point to a host) is
used. Those interested in the niceties of that issue should
review the correspondence on the [email-masked] list and
comment there if appropriate.

A ticket ( [www.ietf.org/rt #282782] ) was generated early in
the month about the ietf.org mail servers apparently rejecting
messages with IP address literals in the EHLO field.  The
rejection is accompanied by a reply message that appears to be
inappropriate in multiple ways; again, those interested should
see the ietf-smtp list for an already-extensive discussion.  The
Secretariat responded by indicating that all such addresses were
being rejected and that the rejection was occurring under
instructions from IETF leadership, instructions that were
reaffirmed after the ticket was filed.  Whatever the problem is,
and indeed, whether there is a problem, the Secretariat is
therefore blameless.  I suggest that the IETF has a problem.

The purpose of this note is _not_ to evaluate the underlying
technical issues, what should be done about them, or whether the
text in RFC 5321 should be improved.  Those, it seems to me, are
topics for the ietf-smtp list.   They have been discussed there
at length and presumably will continue to be discussed there.
It is whether there is consensus among IETF participants that
"the leadership" (I presume whatever bodies, individuals, or
their designees are relevant) should have the authority to
instruct the Secretariat to violate an IETF standard without
consultation of appropriate experts within the community
(presumably on relevant mailing lists), evidence of IETF rough
consensus, and/or Internet Drafts that specify alterations to
the relevant standard(s).  I also don't want to cast blame about
decisions of the past, only to understand what the process is
for giving instructions to the Secretariat (or approving their
suggestions) is now and whether IETF conformance to IETF
standards is something we care about for the future.

  john

---------- End Forwarded Message ----------

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Dec 2019 07:31:54 -0500

> IESG and the IETF list (or, if needed, late feedback to the

Hi John,

I wanted to point out something.

While the mail.ietf.org is raising the bar by requiring a FQDN, it is 
also not checking the FQDN validity.  I did two test using an existing 
FQDN that does not match the connecting client IP and a non-existing 
FQDN from the same IP machine:

EHLO santronics.com
EHLO foo.santronics.com

Each were acceptable, especially foo.santronics.com

So it wants a FQDN and yet doesn't check it.  Why?

Because we have a history of this check being unreliable, hence why a 
MUST NOT reject on the basis.

But why the rejection of a IP-literal if it won't even check if the 
FQDN is even exist or if it matches any of the IPs or even the SPF.  I 
don't get it.  They need to explain this.  What I see is a poor 
implementation of some aggressive anti-spam filters.  Yet, from what I 
can see, it is not doing any SPF check before the DATA command.  I 
gave it a fake machine host name and a return path with a HARD FAIL 
SPF policy and it accepted the information:

S: 220 ietfa.amsl.com ESMTP
C: EHLO foo.santronics.com
S: 250-ietfa.amsl.com
S: 250-PIPELINING
S: 250-SIZE 67108864
S: 250-ETRN
S: 250-STARTTLS
S: 250-AUTH PLAIN LOGIN
S: 250-AUTH=PLAIN LOGIN
S: 250-ENHANCEDSTATUSCODES
S: 250 8BITMIME
C: MAIL FROM:<[email-masked]>
S: 250 2.1.0 Ok
C: RCPT TO:<[email-masked]>
S: 250 2.1.5 Ok

If it is going to be anal for ip-literals, it should also check if the 
FQDN even exist at the very least, and if it has SPF supports, it MUST 
NOT accept the given MAIL FROM, but its probably delaying this until 
DATA if its going an DMARC check.

-- 
HLS

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 16 Dec 2019 17:27:38 -0500

> On Dec 16, 2019, at 7:31 AM, Hector Santos <[email-masked]> wrote:
> 
> While the mail.ietf.org is raising the bar by requiring a FQDN, it is also not checking the FQDN validity.  I did two test using an existing FQDN that does not match the connecting client IP and a non-existing FQDN from the same IP machine:
> 
> EHLO santronics.com
> EHLO foo.santronics.com
> 
> Because we have a history of this check being unreliable, hence why a MUST NOT reject on the basis.

Correct, checking whether the FQDN exists (and even more so resolves to
the client IP) has too high a false positive rate.

> But why the rejection of a IP-literal if it won't even check if the FQDN
> is even exist or if it matches any of the IPs or even the SPF.  I don't get it.

That's easy, ... a sufficiently low false-positive rate allows this
check to weed some junk.  That's all there to it.

"Real" inter-domain relays use FQDNs (whether valid or not), almost
none use address literals.

Mind you there are also reasonably good curated lists of HELO name
regexp patterns, that also have low FP rates, and do a decent job
at stopping lots of spam.  These mostly match PTR records of various
dynamic IP ranges that haven't made into the SpamHaus PBL or similar.

Operators configure rules that work for their overall traffic mix.
The resulting FP rate is almost never zero, but given complaints
about the right kind of legitimate traffic many operators will
make accommodations.  With the largest operators, if you don't
have personal contacts getting through can be difficult...

-- 
	Viktor.

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 16 Dec 2019 14:53:06 -0800

> dynamic IP ranges that haven't made into the SpamHaus PBL or similar.

Frankly, it simply points to professionalism, is the operator of the 
sending platform informed enough to use a proper FQDN, and often that is 
enough to make some operators consider email arriving as less trust worthy.

In a day and age where operators already go far beyond simple 
configuration items (eg some refusing to consider email from a domain 
without SPF records as more spammy) we should 'up our game'.

Having said that, I don't think the RFC's are the right place to enforce 
that you NEED a FQDN yet.  Just expect it will probably end up in a spam 
folder or rejected, without a white list ;)

We all now pretty well universally reject all email unless the IP 
sending it has a PTR record, doesn't mean the RFC should say we HAVE to 
have a PTR to send email.  (That would probably be a better starting point).

However, once we go down that slippery slope, it will affect more and 
more legacy and legitimate senders.

Let the operators decide what they will and will not accept, still seems 
the logical conclusion.

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: Keith Moore <[email-masked]>
Date: Mon, 16 Dec 2019 18:15:51 -0500

On 12/16/19 5:53 PM, Michael Peddemors wrote:

> sending platform informed enough to use a proper FQDN, and often that 
> is enough to make some operators consider email arriving as less trust 
> worthy.

Yes, it does point to professionalism.  Is the operator of the mail 
service professional enough to use only valid criteria in filtering 
mail, or do they make arbitrary, uninformed, cargo-cult decisions about 
what filtering criteria to use?

I've had offers for contracts blocked by spam filters, only finding out 
later when it was too late to get the gig.   A single blocked message 
can easily cost me tens or hundreds of thousands of dollars.

I don't want to hear about "professsionalism" of people sending mail, 
from people who (IMO irresponsibly) promote arbitrary criteria for mail 
filtering.    It's not "professional" to block valid mail for arbitrary 
reasons.

If the sender is caught lying, that's a valid criterion for filtering 
mail.   If the content of the message is clearly inappropriate, that's a 
valid criterion.   If the SMTP is clearly a violation of the protocol 
specification, that might be a valid criterion.   But failure of the 
sender to conform to some arbitrary whim, is not a valid criterion.

> without SPF records as more spammy) we should 'up our game'.
No, we should call out the operators who employ bogus criteria (or at 
least call out the bogus criteria) because those operators are doing 
serious harm to the reliability of email.
>
> Having said that, I don't think the RFC's are the right place to 
> enforce that you NEED a FQDN yet.  Just expect it will probably end up 
> in a spam folder or rejected, without a white list ;)
>
> sending it has a PTR record, doesn't mean the RFC should say we HAVE 
> to have a PTR to send email.  (That would probably be a better 
> starting point).
>
> Let the operators decide what they will and will not accept, still 
> seems the logical conclusion.

I have yet to see a single operator that does this responsibly, except 
for operators who offer customers the option of no spam filtering at all.

Keith

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 19 Dec 2019 11:54:27 +0100

On 2019-12-16 18:15:51 -0500, Keith Moore wrote:
> Yes, it does point to professionalism.  Is the operator of the mail service
> professional enough to use only valid criteria in filtering mail, or do they
> make arbitrary, uninformed, cargo-cult decisions about what filtering
> criteria to use?

How do you know whether they did the first or the latter?

Sure, the wrong explanation ("RFC 2821 violation", when there was in
fact no such violation, and RFC 2821 isn't even the relevant RFC
anymore) strongly suggests that whoever implemented that filter wasn't
exactly firm in their knowledge about protocol details.

But it is still possible that they analyzed a large collection of SMTP
sessions and found that this specific test reduced spam by X % at a
false positive rate of Y % (with Y very close to 0). That would make the
decision very much informed and non-arbitrary and the criterion valid.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Keith Moore <[email-masked]>
Date: Thu, 19 Dec 2019 11:55:44 -0500

> How do you know whether they did the first or the latter?

In this specific instance, I don't know.  But I've seen the latter done 
so many times that I can't dismiss the possibility.

Another disturbing thing I've seen is people essentially making up their 
own rules based on their own prejudices about what "professional" 
operators "should" do when sending mail.

(As an aside, I've come to realize that use of the word "professional" 
and derivatives thereof is a lot like use of phrases like "it is clear 
that" or comparisons with "reality" - all of these are often indicators 
of prejudice and/or unexamined assumptions.   The speaker may be 
accepting his/her assumptions without question and expecting the 
audience to do the same.)

> decision very much informed and non-arbitrary and the criterion valid.

I might agree with you about that, IF the analysis were carefully 
designed, AND it were performed accurately on a statistically valid 
sample, AND the analysis were repeated at frequent intervals, AND the 
required FP rate were established in advance by policy and not merely on 
a whim.   But in my experience, usually none of these is true.

Note also in this case - where the presence of an IP address literal in 
EHLO is used as the sole criterion for rejecting a message before the 
message is actually transferred to the server - there's no opportunity 
to consider other criteria such as sender address validity, DKIM or 
other indications of authenticity, or the content of the message 
itself.   To justify rejecting a message on a single test that is 
entirely unrelated to the content, I'd expect that test to have an 
extremely low FP rate, much better than one considered "good enough" 
when used in conjunction with other tests.

Keith

p.s. I don't accept presence of an IP address literal in EHLO as a valid 
test for spam simply because I've seen too many servers reject mail 
because they didn't like the DNS name presented in EHLO.   While "real" 
servers on the public Internet probably do have DNS names, there are 
operational reasons to justify not using them in EHLO.   Who is to say 
which choices are "professional"? I think it's a silly game, and we 
shouldn't pretend like we can build a reliable email service by 
encouraging such games.

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 19 Dec 2019 13:32:41 -0500

> \On Dec 19, 2019, at 11:55 AM, Keith Moore <[email-masked]> wrote:
> 
> Note also in this case - where the presence of an IP address literal in EHLO is used as the sole criterion for rejecting a message before the message is actually transferred to the server - there's no opportunity to consider other criteria such as sender address validity, DKIM or other indications of authenticity, or the content of the message itself.   To justify rejecting a message on a single test that is entirely unrelated to the content, I'd expect that test to have an extremely low FP rate, much better than one considered "good enough" when used in conjunction with other tests.

Performing multi-factor analysis on messages at "." time, while the remote
client is waiting for "250" is expensive and risky.  Under high load there
may not be sufficient CPU to evaluate all pending messages before some of
the clients time out.  That can lead to multiple deliveries of the same
message, and general exhaustion of server resources.  Faced with botnets
and SASL password brute-forcing attacks, ... MTA operators often strive
to turn away as much junk as possible with comparatively cheap tests before
the DATA command.

This includes RBL checks, presence of PTR, PTR name regexp checks, ...
Rejecting address literal in EHLO is a filter in this class, that can
be justified entirely on its efficacy.  If the reported one complaint
in 10 years is about right, then it is working well enough in practice.

Yes, likely the efficacy of the local rules is not reassessed as often
as one might wish, and may be largely complaint driven.  That's the
nature of how limited devops cycles are spent.  This is not shocking
or even surprising.

But I'd rather deal with a responsive operator with a few odd rules
that reject traffic outright (I get a bounce), than operators that
strictly follow the wire protocol, but then silently deliver to the
Junk folder, and are too big to respond to reports from most senders.

(Your example of losing business because of undelivered proposals
likely falls into the Junk folder delivery category, not SMTP-time
rejection, which you'd have addressed on receipt of the bounce).

-- 
	Viktor.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 19 Dec 2019 11:01:59 -0800

Viktor Dukhovni <[email-masked]> writes:

> nature of how limited devops cycles are spent.  This is not shocking or
> even surprising.

I think there is a root disagreement of motives beneath this discussion.
Keith, as he mentions explicitly, is advocating for a reliable email
service, by which I think he means accepting and properly delivering
well-formed mail according to the protocol.  It's not clear to me,
however, that reliable email service in that sense is the top priority for
a lot of SMTP server operators.  Some are prioritizing filtering out junk
mail over providing reliable email service.

I'm not sure this is the sort of conflict about which operators are going
to reach a universal consensus.  It's a disagreement about motivations and
incentives.  There are email users who would strongly prioritize filtering
out junk over reliable mail delivery, and will choose their providers
accordingly.  (And, of course, there are email users with the opposite
position.)

What role standards should play in that sort of conflict over priorities
is murky.  There is a lot of value in describing the properties of a
reliable email service.  But there's a bit of a "you can lead a horse to
water but you can't make them drink" problem in trying to get operators
who are prioritizing things other than reliable email service to then
follow those rules.

-- 
Russ Allbery ([email-masked])             <https://www.eyrie.org/~eagle/>

---

From: Brandon Long <[email-masked]>
Date: Thu, 19 Dec 2019 11:28:33 -0800

On Thu, Dec 19, 2019 at 10:32 AM Viktor Dukhovni <[email-masked]>
wrote:

> message is actually transferred to the server - there's no opportunity to
> consider other criteria such as sender address validity, DKIM or other
> indications of authenticity, or the content of the message itself.   To
> justify rejecting a message on a single test that is entirely unrelated to
> the content, I'd expect that test to have an extremely low FP rate, much
> better than one considered "good enough" when used in conjunction with
> other tests.
>

Perhaps the software should add the ability to enforce the rules only on a
sample and
provide a mechanism for feedback so that it would be as easy for the
operator to know
whether a rule is working well as it is to put it in place in the first
place

> Junk folder, and are too big to respond to reports from most senders.

Yes, it is true that with an operator dealing with metrics and measurements
and is therefore
generally more accurate will, false positives will be harder to deal with.
It's the nature of
that game.

Brandon

---

From: Keith Moore <[email-masked]>
Date: Thu, 19 Dec 2019 14:53:12 -0500

On 12/19/19 1:32 PM, Viktor Dukhovni wrote:

>> \On Dec 19, 2019, at 11:55 AM, Keith Moore<[email-masked]>  wrote:
>>
> message, and general exhaustion of server resources.
Yes.   Some version of this has been understood since at least RFC 1047.
> Faced with botnets

Not surprising.   I wonder if there's any way we can improve the ability 
of MTAs to reliably filter spam based on cheap tests?

> nature of how limited devops cycles are spent.  This is not shocking

It is not surprising, but it should be.   I don't see why maintenance of 
networks should employ any less discipline than maintenance of ships or 
aircraft.

(To be fair, the overall state of software used to communicate over a 
network is equally sad.   I can't help but wonder if typically poor 
practice in software development has leaked into "devops" practices and 
done further harm to operations.)

> Junk folder, and are too big to respond to reports from most senders.
I certainly agree about that.   Delivery to the Junk folder is pretty 
much tantamount to sending the message to /dev/null, except in certain 
cases like the initial exchange of email between two people who have 
agreed to exchange email via other means.   In those cases the recipient 
knows to look in the Junk folder to see if the message was mislabeled, 
and may even be able to add the sender to a white list or address book.
> rejection, which you'd have addressed on receipt of the bounce).

I wouldn't have received the bounces in such cases.   The senders of 
those messages would have received them, and likely concluded that my 
email address no longer worked at all.

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 19 Dec 2019 15:05:04 -0500

On 12/19/19 2:01 PM, Russ Allbery wrote:

> There are email users who would strongly prioritize filtering

Full disclosure: I have multiple mailboxes with multiple email providers 
specifically because I want different levels of spam filtering for 
different kinds of message traffic.  Though it is kind of expensive.

> who are prioritizing things other than reliable email service to then

I think it would be useful if we can improve the protocols, and/or 
provide sound advice, so that those trying to provide reliable delivery 
could more easily do so.   For those that don't want to try, I agree 
that we really can't do much.

Keith

---

From: Hector Santos <[email-masked]>
Date: Thu, 19 Dec 2019 17:52:53 -0500

> in 10 years is about right, then it is working well enough in practice.

I don't agree this is a justification to bring break a base SMTP 
protocol feature.

This subjective justification talk is equivalent to someone using 
Pareto's Principle to create a RISC SMTP server with low used commands 
like VRFY, EXPN, HELP, NOOP, TURN eliminated.  Odds are very high the 
impact will be low and they will probably get away.  We can ignore 
those one or two clients still using the above low occurrence commands.

For the record, the "complaint" would of been much more and earlier if 
the mail.ietf.org site wasn't whitelisted.  It hid the problem.  When 
it was changed recently, the CBV kicked in.

I also have some doubt this was instituted for "10 years." I can 
easily verify that.  If the IP was constant for 10 years until the 
recent IP change,  the CBV check was preempted.  I can verify this by 
checking my 16 years WCSAP configuration of wcSAP daily statistics and 
session trace logs archives. All saved.  I might do that soon.

> Junk folder, and are too big to respond to reports from most senders.

We have a different view in what is a subjective consideration vs a 
robust standard SMTP protocol feature which is deterministic in nature 
when filtering is applied.  Subjective filters have a higher rate of 
false/true positives/negatives. Its one reason in the wcSAP, we have a 
disclaimer:

http://www.winserver.com/public/aup/antispamsupport.htm#Disclaimer

Wildcat! is one of more aggressive mail filters package around.  But I 
look for deterministic methods, not subjective, and leave the latter 
to admins.

The fact is, there is a SMTP protocol rule of using a HELO/EHLO 
[ip-literal] that MUST match the connection IP.  If not, it is a 
strong reason for rejection.  I have wcSAP stats for that.

The difference with the SDO is a flat rejection of ip-literal based on 
some subjective notion that ALL senders using ip literals are bad.

If it rejected on the rDNS detection that a FQDN was available and 
enforces its usage, that may be a easier argument to sell because it 
is a deterministic rule.

But unfortunately, that isn't the case here.

Finally, depending on one's view of the controversial CBV method, 
others will have a different view on this than me,  but I and my 
customers have 16+ years of a high efficacy system to detect 
fraudulent senders, especially reverse paths. By far, CBV is the 
highest payoff filter right now, today, bar none, than any other known 
method. The CBV is a compliant SMTP client.  It needs to be low 
overhead and fast. It uses the [ip-literal] to reduce PTR overhead and 
to allow it to work ALL the time out of the install box.  Since it is 
a standard feature, it is always expected to work. If sites who are 
beginning to use ip-literal as a subjective trigger to flat out reject 
- they are all in violation of the specification.

BTW, the system does offer the required options that covers the CBV 
option:

; CBV Mail Host Domain

;SapHost   mail.yourdomain.com ; Use a special mail host domain, 
should be MX host.
;SapHost   [serverdomain]      ; Use Primary Host Domain
;SapHost   [mailhost]          ; Use PTR lookup of WCSMTP server IP
SapHost    [serverip]          ; Use WCSMTP server bracketed IP address

So its not something that we are stuck with but no doubt, it adds to 
the efficiency of the method. To eliminate it for no good reason at 
all is pretty hard to swallow.

I have to stop, sorry. Time for a drink. :)

-- 
HLS

---

From: Keith Moore <[email-masked]>
Date: Thu, 19 Dec 2019 18:04:05 -0500

On 12/19/19 5:52 PM, Hector Santos wrote:

> [ip-literal] that MUST match the connection IP.

The more the IPv4 network is polluted with NATs, the less viable this 
rule will be.   But we can hope that the IPv4 public network will be 
decommissioned soon.  :)

Keith

---

From: Paul Smith <[email-masked]>
Date: Fri, 20 Dec 2019 09:11:59 +0000

> rule will be.   But we can hope that the IPv4 public network will be 

I wish - we have a leased line here in the UK - the provider STILL can't 
give us IPv6 on it, and can't give any idea of when it will be available.

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: Paul Smith <[email-masked]>
Date: Fri, 20 Dec 2019 09:22:55 +0000

> strong reason for rejection.  I have wcSAP stats for that. 

The thing is that RFC 5321 says you MUST NOT reject mail if the EHLO 
parameter isn't "right".

"An SMTP server MAY verify that the domain name argument in the EHLO
    command actually corresponds to the IP address of the client.
    However, if the verification fails, the server MUST NOT refuse to
    accept a message on that basis."

For the 'address literal' option, all it says is "an address literal 
SHOULD be substituted for the
    domain name" (note the word 'an' - no mention is made (AFAICS) of 
any correctness requirement).

Note that I'm not arguing against tests which require that, just that 
saying that the address literal MUST match the connection IP is going 
beyond the standards (IMHO).

It's actually quite hard to automatically get a correct address literal 
if you're behind NAT, and if you're also on a dynamic IP, then it's 
approaching the impossible. Now, you may say that 'if the sender is on a 
dynamic IP we don't want to receive mail from them', and that's a valid 
viewpoint, but I'm not sure the SMTP standard should prohibit SMTP 
senders being on dynamic IP addresses.

--

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 20 Dec 2019 14:47:11 +0100

On 2019-12-19 11:01:59 -0800, Russ Allbery wrote:
> a lot of SMTP server operators.

Right. The top priority for me (currently only operating my personal
mail server) isn't to reliably deliver every spam into my mailbox. It is
to reliably deliver non-spam to my eye-balls. Which requires me to weed
out most of the spam, even though this is perfectly well-formed from a
protocol POV. Because reliable delivery to my eye-balls doesn't work if
there are a few legitimate messages among a deluge of spam.

That was also true when I was running the mail service for a few hundred
users (although the tradeoffs were a bit different then) and I assume it
is very similar for large email providers (although again the tradeoffs
are different).

        hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 20 Dec 2019 14:55:44 +0100

On 2019-12-19 14:53:12 -0500, Keith Moore wrote:
> I wouldn't have received the bounces in such cases.   The senders of those
> messages would have received them, and likely concluded that my email address
> no longer worked at all.

I assume that you have the technical abilities to run your own mail
server. So if you think commercial (or free) providers do a lousy job,
why don't you do it yourself? (I wanted to ask that in my previous
reply, but I wasn't sure whether you were the sender or receiver of
those lost mails) If the answer is that it isn't worth your time, then
I'm afraid you are getting the quality of service that you are willing
to pay for.

        hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 20 Dec 2019 15:11:25 +0100

On 2019-12-19 17:52:53 -0500, Hector Santos wrote:
> I don't agree this is a justification to bring break a base SMTP protocol
> feature.
[...]

> The fact is, there is a SMTP protocol rule of using a HELO/EHLO [ip-literal]
> that MUST match the connection IP.  If not, it is a strong reason for
> rejection.  I have wcSAP stats for that.

You are aware that you are directly violating a MUST NOT constraint in
the RFC by doing this, while there is no such constraint against
rejecting all IP literals (or just IP addresses equal to 7 modulo 13,
FWIW). I don't think you can argue with protocol compliance on this.

> The difference with the SDO is a flat rejection of ip-literal based on some
> subjective notion that ALL senders using ip literals are bad.
> 
> If it rejected on the rDNS detection that a FQDN was available and enforces
> its usage, that may be a easier argument to sell because it is a
> deterministic rule.

I am very unclear on what you understand to be "deterministic". 

You seem to be making assumptions on what a "good operator" should do,
use statistics to verify that applying these assumptions as rules have a
sufficiently high true positive and false negative rate and then call
them "deterministic".

If other people do the same (but with assumptions that don't match
yours) you call it "subjective".

> By far, CBV is the highest payoff filter right now, today,
> bar none, than any other known method.

There are some people (not me) who consider CBV abuse.

        hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 20 Dec 2019 15:28:41 +0100

> I am very unclear on what you understand to be "deterministic". 

Which reminds me of an argument I had back in the time when Spamcop was
a thing. For those of you don't remember them, they ran a blacklist
directly driven by user feedback. You could report spam and they would
analyze the Received headers to identify the real sender and add that to
their list. They would also notify the purported sender to give them a
chance to object.

This didn't always work perfectly, but the argument wasn't about that.

The argument was that using the recipient's opinion on whether a given
mail was spam or not spam was "subjective" and you shouldn't base
filters on that.

Instead you should use "objective" criteria like whether the sender had
a PTR record.

Well, that might be objective, but objective for what? There may or may
not be correlation between spam senders and missing PTR records. There
might even be a causal relationship. But that's hard to prove, and in
any case it's just a proxy. Because the end goal is reduce spam, not to
improve the coverage of PTR records.

        hp

---

From: Keith Moore <[email-masked]>
Date: Fri, 20 Dec 2019 10:00:01 -0500

> there are a few legitimate messages among a deluge of spam.

Just for clarification, I've tried to be careful to say that I want 
_legitimate_ mail (maybe even if it's not perfectly well-formed) to be 
delivered reliably.    I have no problem with filtering mail that's 
actually spam.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 20 Dec 2019 10:03:12 -0500

On 12/20/19 8:55 AM, Peter J. Holzer wrote:

> why don't you do it yourself?

I currently have an arrangement for delivery of my mail that is 
satisfactory to me, if a bit expensive.   But it's clear that for the 
public Internet in general, spam filtering is still a problem that has a 
significant negative effect on reliability of email. I think it's worth 
considering whether IETF could improve this situation via some 
combination of improved protocol and sound, carefully-reviewed advice 
for implementors and/or operators.

Keith

---

From: John Leslie <[email-masked]>
Date: Fri, 20 Dec 2019 10:36:41 -0500

On Fri, Dec 20, 2019 at 10:03:12AM -0500, Keith Moore wrote:
> satisfactory to me, if a bit expensive.?? But it's clear that for the 

   I too would like to consider whethr IETF could generate
" improved protocol and
" sound, carefully-reviewed advice for implementors and/or operators.

   Alas, I've come to doubt this is possible... :^( :^(

--
John Leslie <[email-masked]>

---

From: Hector Santos <[email-masked]>
Date: Fri, 20 Dec 2019 12:07:58 -0500

> FWIW). I don't think you can argue with protocol compliance on this.

Hi Peter,

I believe I have explained this. Maybe not.

I admitted the EHLO/HELO [ip-literal] match test against the technical 
requirement it must represent the connecting IP address, was, in fact, 
a violation of the MUST NOT reject when enforced.  There is ambiguity 
whether it just applied to FQDN validity checking due to strong 
history of configuration problems with FQDN. But in principle,  the 
MUST NOT reject HELO/EHLO regardless of input is well respected.

This was why I asked Kkensin during RFC2821bis to consider the pending 
era of new "deterinistic" protocol methods because some may begin 
conflict with the spirit of SMTP always accept mail methodology. These 
were on their way.

- Enforcement pure 100% SMTP compliancy
- CBV checking the return path
- SPF checking the return path domain policy
- ADSP checking the author domain policy

The difference with "deterministic" concepts as oppose to 
"nondeterministic" or "subjective' concepts where it is more about 
"reputation," these are not immediate, but learned.

> I am very unclear on what you understand to be "deterministic".

 From google:  Deterministic functions always return the same result 
any time they are called with a specific set of input values and given 
the same state of the database.

Nondeterministic is basically when the function need more help beyond 
the basic protocol.

 From google: In computer science, a nondeterministic algorithm is an 
algorithm that, even for the same input, can exhibit different 
behaviors on different runs, as opposed to a deterministic algorithm. 
There are several ways an algorithm may behave differently from run to 
run.

Nondeterministic systems contain fuzziness in results. The key is 
that, your results MAY be different from mine.  Not so with 
deterministic functions, protocols and methods.

DKIM POLICY model is deterministic
DKIM TRUST model is nondeterministic

Every wonder why we can't get a DKIM TRUST going as a standard? 
Because it suffered as a "Battery Required" protocol.  It doesn't work 
without a 3rd party trusty party service.

DKIM Policy got traction with its 1st party only deterministic rules. 
  The reason why we had such a terrible problem with DKIM+ADSP and now 
DKIM+DMARC is because we can't resolve the subjective, 
nondeterministic "Batteries Required" 3rd party signer authorization 
problem.  Its going on almost 15+ years, Peter. :(  And because we 
can't resolve it, we now begin break RFC5322 protocol and long time 
mail engineering with strong persistent Author fields by rewriting 
5322.From.

Go figure.

> sufficiently high true positive and false negative rate and then call

Wow, really, is that how you read my text?   It conflicts with my 
philosophy.

Sorry, no. In fact, deterministic concepts really do not need the 
operator to be involved to make a decision. He just enabled it or not. 
  It is the protocol negotiation violation that is 100% repeatable, 
everywhere, that makes it popular and workable.

With domain policy protocols, like SPF, we are fundamentally accepting 
the domain is telling us what to expect with no fuzziness.  A SPF 
domain may use a softfail or neutral result, and we are asked not 
reject on that.  So we don't.  But SPF says to reject on a hardfail, 
so we do.   Of course, a local policy can do what it wants.  But does 
not change the deterministic nature of protocol.

> yours) you call it "subjective".

Yes, when you have different yields, yes, its nondeterministic, and I 
would say subjective and also presents a security problem.

Every site that implements SPF, and they honor the domain policy, it 
is expected that all sites will have exact yields.

With this SDO being the only site I am aware of that flat out rejects 
ip-literal usage regardless of validity, right not, it is subjective 
rule on that part only.  You will not find that to be that case with 
99.99% if the sites.

When everyone begin to do it, and we decide to clarify it in 
RFC5321bis that ip-literal are obsolete and we loosen the MUST NOT, 
then it becomes part of the protocol and that point, i would consider 
it to be deterministic and no longer subjective, even thought the 
decision to do this, probably with an extremely rough WG concensus 
(BAD) was subjective decision.

> There are some people (not me) who consider CBV abuse.

I get it.  That is why I would never attempted to recommend it to be 
IETF standard or seek IETF approval. But it works extremely well. Its 
a black box to me:

    result =  CBV(reverse-path)

Today, CBV is a fast small footprint, single function SMTP client (see 
below). Tomorrow, maybe the function can be replaced using advanced 
'reverse-path' ESMTP extension.  I remember trying VRFY and I don't 
think many sites have opened themselves to VRFY despite it is a low 
usage SMTP command MUST requirement.

Finally along those lines, once upon a time, a set of IETF cogs 
vehemently preached against GreyListing, SPF and ADSP for the same 
basic reasons and today, they are on board, completely or partially. 
With ADSP, the same folks who had it erroneously abandoned, brought it 
back as DMARC without resolving the same problems ADSP was abandoned 
for -- 3rd party authorization concepts.

-- 
HLS

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 20 Dec 2019 19:55:14 +0100

On 2019-12-20 12:07:58 -0500, Hector Santos wrote:
> From google:  Deterministic functions always return the same result any time
> they are called with a specific set of input values and given the same state
> of the database.
> 
> Nondeterministic is basically when the function need more help beyond the
> basic protocol.
> 
> algorithm that, even for the same input, can exhibit different behaviors on
> different runs, as opposed to a deterministic algorithm. There are several
> ways an algorithm may behave differently from run to run.
> 
> Nondeterministic systems contain fuzziness in results. The key is that, your
> results MAY be different from mine.  Not so with deterministic functions,
> protocols and methods.

So how is 

    if re.match(r'^\[\d+\.\d+\.\d+\.\d+\]$', ehlo_string):
        return REJECT

non-deterministic? This depends on only a single input parameter which
is completely under the control of the client. Send the same EHLO
command and you get the same response - always.

OTOH, you suggested "checking the remote IP address of the connection
against the address literal in the EHLO command" and "checking whether a
PTR record exists" as deterministic rules.

These not only have more input parameters (the remote ip address and the
result of a DNS query respectively), those are subject to temporary
changes: The IP address may change because of routing changes and NAT
gateways, DNS responses may be lost or outdated. So while the algorithm
itself may be deterministic, the result is not, because some input
parameters are prone to random fluctuations.

> Sorry, no. In fact, deterministic concepts really do not need the operator
> to be involved to make a decision. He just enabled it or not.

Well, enabling a specific check or not is a decision. Nobody expects a
human to be directly in the loop during an SMTP transaction.

> everywhere, that makes it popular and workable.

There is no protocol violation. There is your (the developer's) personal
opinion on how a sending MTA on the public internet should be
configured. I may personally agree with them, and you might even do a
survey and find that "97 out of 100 experts agree", but it's still a
personal opinion, not part of SMTP as specified in RFC 5321.

You apparently keep stats and check them periodically, which is good and
- as Keith so forcefully pointed out - is more than many ESPs do. I
don't say that what you are doing is wrong - in fact it seems like a
quite sensible thing to do - but it certainly doesn't match your claims.

> Yes, when you have different yields, yes, its nondeterministic, and I would
> say subjective and also presents a security problem.

You seem to have missed my point. My point is that whether you call
something "deterministic" or "subjective" seems to depend on whether you
came up with the idea or someone else instead of any actual determinism
of the rule. (See example above).

> ip-literal usage regardless of validity, right not, it is subjective rule on
> that part only.  You will not find that to be that case with 99.99% if the
> sites.

Just as subjective as your rule that the peer address must match the IP
literal, or your suggestion that an IP literal must onlybe used if there
is no PTR record. Both may be good practice, but they are not mandated
by the RFC.

And even for outright syntax errors you might choose to tolerate them.
For example, the RFC is quite clear that there is no space between ":"
and "<" in "MAIL FROM:<[email-masked]>". Yet enforcing that rejects
quite a lot of legitimate mail (or did, when I last tried it). So there
is an assessment of the pros and cons of being conservative or liberal
involved.

        hp

---

From: Hector Santos <[email-masked]>
Date: Fri, 20 Dec 2019 16:14:01 -0500

Peter,

Thanks for your comments and wisdom.

I agree, we have to judge on how a strict or relax a system is.   I 
follow the Postel's principle and did so before it existed.

You mentioned the space issue in MAIL FROM.  Yes, an old biggie with 
some MUAs, like OE.  Some servers enforced the syntax, others, like 
wcSMTP, does not.

There are others items, like how the a SMTP server handles DATA that 
doesn't include the minimum  RFC 822 or 2822/5322 headers where the 
minimum requirements differ.  There are, of course, applications where 
headers do not exist or not those related to email, i.e. print, fax, etc.

The issue with minimum TIMEOUT values, scaling and load management and 
the whole idea that the server MUST wait until the client issues a 
QUIT, never close the socket, especially after a transaction is 
completed and the MTA waits for more mail to send. wcSMTP gives you 
far less than 5 mins to start a new transaction, otherwise <click>.

So there are certainly many areas of mail system that we fine tune, 
But I think in principle no one is hurt by the fine tuning.  If it 
did, rest assured, I will nix the idea even if it just hurt 1 system. 
  1 system too many.

But in the regards with the specific HELO/EHLO [IP-literal], I firmly 
believe the flat out usage rejection in horribly wrong for one simple 
reason - you MUST change your otherwise valid code to comply.

I don't see any justification for the SDO to flat out reject it.

I can see justification if the SDO checked for a client IP match, like 
wcSMTP does, but it did not.

I can see justification if the SDO did a PTR lookup, found records and 
rejected for that reason, but it did not.  Thus reason would be more 
acceptable and would probably pressure me to adjust because the specs 
does use MUST use FQDN when it is available.

So Peter, I don't know if we are not agreeing, in my opinion, this 
specific issue is much different that any other.  It actually forces 
changes in MTA code to make sure it no longer uses ip literals.  IMO, 
that promotes changing the specs.

Finally, I am not a regex guru, my wcSMTP C/C++ CheckHello() function 
does a complete syntax check:

//-----------------------------------------------------------------
// 451.2 05/14/04 01:51 pm
// return TRUE if helo is a domain name or if a dot address is
// equal to the connection address

BOOL isGarbage(const char *sz)
{
     if (sz) {
        for (DWORD i = 0; i < strlen(sz); i++) {
          if (sz[i] < 32 || sz[i] > 'z') return TRUE;
        }
     }
     return FALSE;
}

BOOL IsNumber(const char *psz)
{
     if (!psz || !psz[0]) return FALSE; // zero length string
     char *pOut;
     strtol(psz,&pOut,10);
     return psz!=pOut;
}

//------------------------------------------------------
// smtpFilterCheckHello
//
// Check the HELO/EHLO (szHelo) hostname for IP literals, [#.#.#.#], 
against
// the connecting client ip (cip) address.
//
// return FALSE if syntax or match fails, TRUE otherwise.

BOOL APIENTRY smtpFilterCheckHello(const char *szHelo, const DWORD cip)
{
     DWORD addr = INADDR_NONE;
     if (szHelo && szHelo[0]) {

        //----------------------------------------------
        // check for space
        const char *p = strchr(szHelo,' ');
        if (p) return FALSE;
        if (isGarbage(szHelo)) return FALSE;
        //----------------------------------------------

        // 454.2F9 check for illegal [] formats
        if ((szHelo[0] == '[') && (szHelo[strlen(szHelo)-1] != ']')) {
            // illegal format, left, but no right bracket
            return FALSE;
        }
        if ((szHelo[0] != '[') && (szHelo[strlen(szHelo)-1] == ']')) {
            // illegal format, right, but no left bracket
            return FALSE;
        }

        if (szHelo[0] == '[') {
            char sz[MAX_PATH] = {0};
            strncpy(sz,&szHelo[1],sizeof(sz)-1);
            char *p = strchr(sz,']');
            if (!p) {
              // illegal format, no right bracket
              return FALSE;
            }
            *p = 0;
            addr = inet_addr(sz);
            if (addr == INADDR_NONE) {
                // illegal dotted-decimal address
                return FALSE;
            }
        } else {
           const char *p = strrchr(szHelo,'.');
           if (p && IsNumber(p+1)) {
               // illegal TLD
               return FALSE;
           }

           addr = inet_addr(szHelo);
           if (addr != INADDR_NONE) {
               // illegal format, no brackets
               return FALSE;
           }
        }
     }
     return (addr == INADDR_NONE) || (addr == cip);
}

Yes, I agree this check is deterministic because it would follow the 
specification.  The handling part is probably were the disagreement 
might be.   I choose to use a spec-related rule. The SDO choose to use 
some non-SMTP subjective rule to reject ip-literal usage entirely. 
That is not SMTP, imo.

-- 
HLS

On 12/20/2019 1:55 PM, Peter J. Holzer wrote:
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

---

From: Hector Santos <[email-masked]>
Date: Sat, 21 Dec 2019 10:00:43 -0500

> It's actually quite hard to automatically get a correct address
> literal if you're behind NAT, and if you're also on a dynamic IP, then
> it's approaching the impossible.

For auto preparing the correct MUA's MTA setup, I agree, it is a quite 
hard to get it coded right.  I believe the SMTP "ideal" MTA auto setup is:

(1)  Set Client.IP from Sender Session Connection IP

(2)  Set EHLO.field from manual configuration/definition

(3.1)  if EHLO.FIELD is empty then EHLO.FIELD = using socket gethost() 
for client.ip
(3.2)  if EHLO.FIELD is empty then EHLO.FIELD = get first PTR for 
client.ip

(4)  if EHLO.FIELD is empty, then EHLO.FIELD = "[" + Client.IP= "]"

Not every MTA does #2.  Not every MTA has a dns resolver, so it may 
not do 3.2, but it may do 3.1 and the behavior depends on the OS 
socket stack.  But going direct to DNS (3.1) is generally the right way.

Either way, EHLO.FIELD can end up being validation incorrect.

It required two things, either a relaxation at wcSMTP or the MUA has a 
manual way to set the ip-literal for the NAT.

I don't fully recall the widely tested MUAs, but the popular ones like 
OE used the machine's netbios name. It probably only did 3.1 and I 
never saw it used #4 to set an ip-literal.  Since I using Thunderbird, 
it did use #4 and I issued a report to the Thunderbird guys to add an 
option to offer #2.  I assisted with the suggested C/C++ changes to 
the EHLO command function.

But that would be a manual configuration the MUA user had to be 
familiar with because with TBIRD there was no "first time setup, What 
is your NAT address?" popup configuration.  With the change, you had 
to use the "config://" url.

The impact was seen mostly with the submit protocol where the MUA's 
MTA issued a private-ip-literal, and of course, it would not match the 
NAT's public ip and wcSMTP would filtered it.

So the way I approached the  problem was:

Overall, in wcSMTP, client authentication ruled to bypass these new 
extended wcSMTP checks.

- With Public Port 25 access, the helo/ehlo ip-literal check is done. 
Admin has option to disable it.  If the connected IP was 
authenticated, via a IP relay table or via POP3 B4 SMTP method, it was 
skipped.

- With Private Port access (port 587), the EHLO ip-literal check was 
skipped because Authentication was expected after EHLO.

I recall sending an implementation note to Klensin and Gellens 
explaining the "MUA behind the NAT" problem and the suggestion to 
relax any EHLO validation now authorized by RFC4409 because ESMTP AUTH 
was now expected and required.

> Now, you may say that 'if the sender
> is on a dynamic IP we don't want to receive mail from them', and
> that's a valid viewpoint, but I'm not sure the SMTP standard should
> prohibit SMTP senders being on dynamic IP addresses.

I agree.

-- 
HLS

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 21 Dec 2019 15:40:29 -0500

> On Dec 21, 2019, at 10:00 AM, Hector Santos <[email-masked]> wrote:
> 
> I believe the SMTP "ideal" MTA auto setup is:
> 

This is what Postfix does by default:

	myhostname = <system hostname, with .$mydomain if not fqdn>
	smtp_helo_name = $myhostname

Either or both can be set explicitly.

> (3.1)  if EHLO.FIELD is empty then EHLO.FIELD = using socket gethost() for client.ip

This is not robust in the face of temporary lookup failures, so we don't do
that.

> (3.2)  if EHLO.FIELD is empty then EHLO.FIELD = get first PTR for client.ip

Ditto.

> (4)  if EHLO.FIELD is empty, then EHLO.FIELD = "[" + Client.IP= "]"

This is frowned upon by receiving systems, we don't do that.

> Not every MTA does #2.  Not every MTA has a dns resolver,

Those that don't, do send email to across the public Internet, so
are out of scope for this thread.

> so it may not do 3.2, but it may do 3.1 and the behavior depends on the OS socket stack.  But going direct to DNS (3.1) is generally the right way.

Well, my take is that #2 is the preferred (robust, dare I say "deterministic")
approach.

-- 
	Viktor.

---

From: Keith Moore <[email-masked]>
Date: Sat, 21 Dec 2019 15:47:09 -0500

> This is frowned upon by receiving systems, we don't do that.

So if you don't have an explicitly configured hostname, do you simply 
refuse to send messages?   Or send EHLO example.com?  or what?

DNS is nowhere nearly universal, and cannot be.   Using an IP address 
literal in EHLO makes perfect sense, at least for systems forwarding to 
a private submission server.   And there are so many of those systems 
out there that requiring explicit configuration for all of them might 
not be the best choice.

(As for the notion that "real MTAs have DNS names", I think that even 
that's a stretch.   People are constantly making dubious assertions 
about what "real systems" do or should do.   But the Internet is 
necessarily a tremendously diverse place and it's easy to overlook some 
important corner case when making such sweeping assertions.)

Keith

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 21 Dec 2019 17:20:55 -0500

> On Dec 21, 2019, at 3:47 PM, Keith Moore <[email-masked]> wrote:
> 
> So if you don't have an explicitly configured hostname, do you simply refuse to send messages?   Or send EHLO example.com?  or what?

There's always a hostname.  It defaults to the return value of gethostname().
And if that contains no dots, and "mydomain" is not set, then the domain
defaults to "localdomain".  So worst case you get "shortname.localdomain".

Internal documentation:

      get_hostname() returns the local hostname as obtained
      via gethostname() or its moral equivalent. This routine
      goes to great length to avoid dependencies on any network
      services.
      Fatal errors: no hostname, invalid hostname.

> DNS is nowhere nearly universal, and cannot be.

There is deliberately no name resolution dependency (DNS or
otherwise) in the code that determines the system hostname.
Postfix is documented to work on systems that are not connected
to the Internet.  All that's required is:

	smtp_host_lookup = native

-- 
	Viktor.

---

From: Paul Smith <[email-masked]>
Date: Sat, 21 Dec 2019 22:28:50 +0000

> defaults to "localdomain".  So worst case you get "shortname.localdomain".
But RFC 5321 says:

The domain name given in the EHLO command MUST be either a primary
       host name (a domain name that resolves to an address RR) or, if
       the host has no name, an address literal, as described in
       Section 4.1.3  <https://tools.ietf.org/html/rfc5321#section-4.1.3>  and discussed further in the EHLO discussion of
       Section 4.1.4  <https://tools.ietf.org/html/rfc5321#section-4.1.4>.

So, if it's a host name, it has to resolve. So, 'shortname.localdomain' wouldn't be valid.

--

---

From: Keith Moore <[email-masked]>
Date: Sat, 21 Dec 2019 17:50:45 -0500

>> defaults to "localdomain".  So worst case you get "shortname.localdomain".

I expect that there are a lot of MTAs advertising EHLO 
raspberry.localdomain or debian.localdomain or some such.

> So, if it's a host name, it has to resolve. So, 'shortname.localdomain' wouldn't be valid.

The interesting thing to me is that if you don't have a valid DNS name, 
an IP address literal seems like the right thing to use. Even if behind 
a NAT at least it's some identifier that is associated with the sending 
host that can go in a Received header, still potentially useful for 
tracing if the server has logs showing the source IP address.   An EHLO 
tag based on a boilerplate hostname and boilerplate suffix is 
essentially useless for that purpose.

Seen in that light, the IETF operators' filtering policy is looking less 
valid all the time.

Keith

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 21 Dec 2019 18:44:17 -0500

On Sat, Dec 21, 2019 at 10:28:50PM +0000, Paul Smith wrote:
>
> > There's always a hostname.  It defaults to the return value of
> > gethostname().  And if that contains no dots, and "mydomain" is not
> > set, then the domain defaults to "localdomain".  So worst case you
> > get "shortname.localdomain".
> >
> But RFC 5321 says:
> 
>   host name (a domain name that resolves to an address RR) or, if the
>   host has no name, an address literal, as described in Section 4.1.3
>   <https://tools.ietf.org/html/rfc5321#section-4.1.3>  and discussed
>   further in the EHLO discussion of Section 4.1.4
>   <https://tools.ietf.org/html/rfc5321#section-4.1.4>.

On the public Internet, a Postfix MTA that expects to deliver mail to
others will often have an explicitly configured name, an explicitly
configured domain suffix, or both.

> So, if it's a host name, it has to resolve. So,
> 'shortname.localdomain' wouldn't be valid.

It is valid enough for private networks.

> raspberry.localdomain or debian.localdomain or some such.

That's fine, they generally relay via a smarthost, which won't care
about their EHLO name.

> > So, if it's a host name, it has to resolve. So,
> > 'shortname.localdomain' wouldn't be valid.
> 
> an IP address literal seems like the right thing to use.

Only in theory.  In practice, you use an FQDN.

> Even if behind a NAT at least it's some identifier that is associated
> with the sending host that can go in a Received header, still
> potentially useful for tracing if the server has logs showing the
> source IP address.

In practice MTAs already record the IP address in Received headers (in a
comment after "from <heloname>").  So the IP-literal EHLO is largely
redundant, while the EHLO name caries useful additional information.

> An EHLO tag based on a boilerplate hostname and
> boilerplate suffix is essentially useless for that purpose.

Perhaps, in theory, but not in practice.

> Seen in that light, the IETF operators' filtering policy is looking
> less valid all the time.

Except for the part where it works well enough.

-- 
    Viktor.

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 22 Dec 2019 11:44:07 +0100

On Sat 21/Dec/2019 23:20:55 +0100 Viktor Dukhovni wrote:
>> On Dec 21, 2019, at 3:47 PM, Keith Moore wrote:
>> 

I guess recipient email addresses have to be given in dotted quad too...?

Best
Ale
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 22 Dec 2019 14:59:10 -0500

> On Dec 22, 2019, at 5:44 AM, Alessandro Vesely <[email-masked]> wrote:
> 

No, that conclusion does not follow.  While the system hostname is
determined "statically", with no name service lookups, the nexthop
domain is subject to name resolution, which defaults to "dns", but
can be set to "dns, native" on mixed networks, or just "native" on
networks where DNS is not available.

Of course if one wants to send mail to "luser@[192.0.2.1]" that'll
work too (but that address form isn't by default accepted via SMTP
from clients that are neither on a local network nor by some means
authenticated, an open-relay configuration als possible, though it
is not likely to be enabled just by accident.

-- 
	Viktor.

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Dec 2019 04:33:44 -0500

> raspberry.localdomain or debian.localdomain or some such.

I did a grep in December logs, I see 4 instances with all 
failed/rejected with 550.  The last traced session was on the 22th:

**************************************************************************
Wildcat! ESMTP Server v8.0.454.9
SMTP log started at Sun, 22 Dec 2019  07:18:48
Connection Time: 20191222 07:18:48  cid: 0000A55B tid: 00001488
SSL-Enabled=YES No-Quit-Cancel=OFF Receiver-Bin=ON
Client IP: 206.81.8.171:36566 (unknown) Host IP: 76.245.57.69:25
07:18:48.959 ** WCX Process: smtpcmd-connect  ret: -1
07:18:48.960 S: 220-winserver.com Wildcat! ESMTP Server v8.0.454.9 ready
07:18:48.960 S: 220-************** WARNING:  FOR AUTHORIZED USE ONLY! 
**********************
07:18:48.960 S: 220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS 
PROPRIETARY COMPUTERS    *
07:18:48.960 S: 220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR 
DISTRIBUTE UNSOLICITED *
07:18:48.960 S: 220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM 
WILL RESTRICT ACCESS *
07:18:48.960 S: 220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY. 
                      *
07:18:48.960 S: 220 
************************************************************************
07:18:49.047 C: EHLO varon.localdomain
07:18:49.053 ** WCX Process: smtpcmd-check-ehlo  ret: -1
07:18:49.053 S: 250-winserver.com, Pleased to meet you.
07:18:49.053 S: 250-SIZE 102400000
07:18:49.053 S: 250-8BITMIME
07:18:49.053 S: 250-SUBMITTER
07:18:49.054 S: 250-ETRN
07:18:49.054 S: 250-AUTH CRAM-MD5 DIGEST-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
07:18:49.054 S: 250-AUTH=LOGIN
07:18:49.054 S: 250-HELP
07:18:49.054 S: 250 STARTTLS
07:18:49.181 C: MAIL FROM:<[email-masked]> SIZE=1926
07:18:49.193 S: 250 <[email-masked]>... Sender validation 
pending. Continue.
07:18:49.237 C: RCPT TO:<[email-masked]>
07:18:49.378 ** WCX Process: wcsap ret: 550 (140 msecs) (Rejected by 
WCSAP RBL Host bl.spamcop.net)
07:18:49.378 S: 550 Return Path not verifiable.
07:18:49.423 C: RSET
07:18:49.423 S: 250 Reset State #1
07:18:49.466 C: QUIT
07:18:49.466 S: 221 closing connection
07:18:49.466 ** Completed. Elapsed Time: 531 msecs

IP Rejected by spamcop.net.

I suspect when I put my logs thru a Deep NN engine, its going to tell 
me '*.localdomain' usages are mostly bad and I can save 1/2 second by 
instantly rejecting the .localdomain usage and it will probably be 
Pareto (>79%) accurate.

> Seen in that light, the IETF operators' filtering policy is looking

+1.

Especially when no validity check is done on the non-ip-literal input.

Was the original intent just to describe the machine identifier with 
no validation necessary?

Also EHLO was used for a capabilities "Link Establishment" like 
phrase.  We probably should not lump the machine identifier with a 
capabilities negotiation concept.

Mostly joking,  a new ESMTP command  "CAPA" that can replace EHLO?

-- 
HLS

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 23 Dec 2019 14:46:42 +0100

On 2019-12-20 10:03:12 -0500, Keith Moore wrote:
> But it's clear that for the public Internet in general, spam filtering
> is still a problem that has a significant negative effect on
> reliability of email.

I would phrase that differently: Spam has a significant negative effect
on reliability of email. Spam filtering mitigates some of that, but is
not able to achieve 100% reliability.

> I think it's worth considering whether IETF could improve this
> situation via some combination of improved protocol and sound,
> carefully-reviewed advice for implementors and/or operators.

I'm not optimistic there.

        hp

---

From: Dave Crocker <[email-masked]>
Date: Mon, 23 Dec 2019 06:00:30 -0800

> not able to achieve 100% reliability.

Exactly right. 90-95% of the mail crossing the open Internet is spam. 
The better receive-side filtering engines catch all but a very, very 
small percentage of that.

> I'm not optimistic there.

+10

It's worth considering if and only if there are fresh and highly 
pragmatic ideas driving the effort.

Given the number of years of anti-spam effort by intelligent, 
experienced, and motivated people, and the number of times there have 
been similar efforts to offer 'solutions', the likelihood of crossing 
the threshhold of 'fresh and highly pragmatic' is low.  Very low.  But 
low isn't (quite) zero.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 23 Dec 2019 15:48:01 +0100

On 2019-12-20 16:14:01 -0500, Hector Santos wrote:
> So Peter, I don't know if we are not agreeing, in my opinion, this specific
> issue is much different that any other.

This is where we very much disagree. For me, what the filter deployed by
the IETF admin and the one you are using are morally exactly equivalent
(obviously they are not quantitatively equivalent, since on affects a
strict subset of the other): Both are heuristics which are based on a
characteristic which is relatively common in spam and very rare in
legitimate mail. As such, both must be exclusively evaluated on their
true positive and false positive rates and the context in which they are
deployed. Neither of you can claim the moral high ground.

> It actually forces changes in MTA code to make sure it no longer uses
> ip literals.

It seems to me that your CBV agent violates at least one MUST
constraint, so that might not be a bad idea.

> IMO, that promotes changing the specs.

Not at all. The admin of one specific MTA implemented a heuristic which
happened to block one specific CBV probe which caused some mails to be
rejected. Such stuff happens. Changing the configuration of one or both
systems can fix the problem. No spec change necessary.

The discussion did however strengthen my belief that the paragraph from
section 4.1.4:

|   An SMTP server MAY verify that the domain name argument in the EHLO
|   command actually corresponds to the IP address of the client.
|   However, if the verification fails, the server MUST NOT refuse to
|   accept a message on that basis.  Information captured in the
|   verification attempt is for logging and tracing purposes.  Note that
|   this prohibition applies to the matching of the parameter to its IP
|   address only; see Section 7.9 for a more extensive discussion of
|   rejecting incoming connections or mail messages.

ought to be deleted. Not to legalize what the IETF admin did - that's
already fine - but what you did. Because you are directly violating this
MUST NOT and one might read that paragraph as having priority over
section 7.9. So that would mean that the more specific test is
prohibited but the wider test is allowed - which I find perverse.

> Finally, I am not a regex guru,

The code snippet I posted was just pseudo-code which rejected any
sequence of 4 decimal numbers separated by dots enclosed in square
brackets. I have no idea whether this is the actual test used by the
ietf server (probably not). It was just intended to show that the
observed effect can be achieved by an entirely deterministic (and very
simple) algorithm.

        hp

---

From: Keith Moore <[email-masked]>
Date: Mon, 23 Dec 2019 10:33:44 -0500

> not able to achieve 100% reliability.

Yeah, I see so much evidence of poorly-chosen spam filtering criteria 
that I'm not willing to give the spam filters a pass just because the 
volume of spam is so great.

I think we need to recognize that spam filters are part of the 
problem.   Just because a problem exists does not mean that any supposed 
fix is good.

Keith

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Dec 2019 11:34:16 -0500

> prohibited but the wider test is allowed - which I find perverse.

I agreed and I have admitted I have a single rule for comparing the 
ip-literal with the connection IP. Mismatches are rejected.  There are 
some cases (MUA behind the NAT) where there are a FP, resolved with 
authentication requirements. I believe this test is a correct 
technical SMTP check compared to a subjective, nondeterministic 
conclusion  to block all ip-literals that technically violates SMTP.

That say, in the spirit of RFC2821, again, the CBV probe predated 
RFC5321 and I was the one who asked Klensin for some guidance and 
verbosity on the 2000s growth of new mail filters, including new EHLO 
filters.  Section 7.9 was a result.

Keep in mind CBV and SPF were developed and implemented around the 
same time.  SPF also has HELO/EHLO checking. A rejection was now 
possible for any filtering reason on HELO/EHLO. It was a black box now:

   result = CHECK_HELO(client.hostname, client.ip)

where CHECK_HELO() can consist of new considerations:

   - ip-literal test
   - SPF test

SPF needed a delay to see what reverse-path was provided. So it was 
not just about a CBV probe but EHLO [ip-literal] correctness for my 
SMTP receiver.

The EHLO [ip-literal] filter I implemented was based on original Modem 
Hosting Caller ID methods and logic that we offered in our 1980s 
Online BBS hosting package which many 3rd party developers implemented 
into create "New Sign up" add-on scripts.  Today, while all that is 
legacy, Wildcat! still offers modem based hosting for federal, major 
enterprise, banking, inventory, payroll and billing private file/mail 
exchange networking.

So I believe the EHLO [ip-literal] test is a technically correct 
justified one that is not based on some "feeling" that all users 
should be lumped under the same category, in particular when it 
doesn't validate what it is enforcing MTA to do.

> observed effect can be achieved by an entirely deterministic (and very

ok, I wonder if I can do this:

EHLO ip-literal   ; no squared brackets

and see if the SDO site allows it.  I suspect it will.  I could just 
do the right thing and provide a valid FQDN, if available. But who are 
we really addressing when anyone can just make it a value known to 
skip & preempt some delayed check?

My philosophy is simple; where ever possible strong SMTP compliancy, 
without breaking it, can applied, in order to raise the bar to move 
the legacy MTA into current levels of SMTP compliancy, is a good 
thing.  I don't believe bad guys complain, only good guys when it 
comes to impact.  Bad guys will not bother with it. Smart ones will 
adapt - a good thing but bad for detecting bad content. Good guys will 
always adapt. But in this case, again, imo, it is based on a bad 
reason and it will cost developers time and money to changed their 
code -- painfully unnecessary but ....

If we want change, then go for the big bang. Naybe the real desire 
here is to completed eliminate the EHLO machine identifier idea and 
only use EHLO for what is really for -- to negotiate client/server 
capabilities.

I already whitelist the SDO. What I hope is that others do not follow 
the SDO lead here. If they do, then I will forced to consider change 
code and we will have no choice to update the specs.   I can just set 
it now and forget about it. I would have to test to see what impact it 
will have, short residence times vs longer residence time due to PTR 
lookups but I can't manually set a host name because everyone is 
different. The more I can make it plug and play, the better it is, one 
less item to configure, the less support issues I have, and all that 
is a lesser cost of development and adds to a lesser TCO.  This is a 
legit mail engineering consideration.

Thanks

-- 
HLS

---

From: Keith Moore <[email-masked]>
Date: Mon, 23 Dec 2019 11:50:55 -0500

> On Dec 23, 2019, at 11:34 AM, Hector Santos <hsantos=[email-masked]> wrote:
> 
> I agreed and I have admitted I have a single rule for comparing the ip-literal with the connection IP. Mismatches are rejected.  There are some cases (MUA behind the NAT) where there are a FP, resolved with authentication requirements. I believe this test is a correct technical SMTP check compared to a subjective, nondeterministic conclusion  to block all ip-literals that technically violates SMTP.

I believe that NATs within the ipv4 network (not merely on the periphery) will become increasingly commonplace, for as long as there is a public ipv4 Internet.  I also believe that the HELO/EHLO tag is really only usable for inclusion in Received fields and to identify the client system within its own environment.  So any checks on validity with respect to the server environment are dubious at best and may do harm to the protocol.  This applies equally to IP address literals and names.  

Keith

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Dec 2019 12:59:59 -0500

> I believe that NATs within the ipv4 network (not merely on the periphery) will become increasingly commonplace, for as long as there is a public ipv4 Internet.  I also believe that the HELO/EHLO ta…

I believe I can support 100% the elimination of EHLO machine 
identification semantics and the need to validate it and let the 
receiver add a trace header that records the connection IP and 
optional a matching PTR, I believe many, if not all, are already doing 
this.

But there are senders using MTAs that well known to be bad guys with 
constant usage of the same machine identifiers.  Most don't seem to 
learn nor adapt because its constantly used. These will also be part 
of a local policy filter.

Do you believe a backward compatible ESMTP solution is possible along 
the lines of a connection response preempting the need to use EHLO? 
Afaik, ESMTP AUTH is the only protocol that required a double EHLO 
sequence with the idea that the 2nd one may show additional 
capabilities after an successful AUTH takes place.

Ok, let me throw this into the new mail filtering mix.  I am sure some 
here will not like it, and neither did I but I had no choice.  Like I 
said, wcSMTP is one of more aggressive filtering systems. In December 
2018, I released an major update that included Geo IP Location filtering.

http://www.winserver.com/public/aup/wcGeoIp-Location-Filters.htm

The idea is simple: IF your operation, business, etc, absolutely has 
no expectation to be doing business or communications with particular 
countries,  nations or regions of the world, wcSMTP can immediately 
filter by GeoIP Location at the connect level.

I was totally against such concepts until last year. With the massive 
world wide attacks we ALL are currently feeling and seeing, again, 
something had to be done.  It was a huge gain for many of our 
operators.  Its has a year and the only adjustments were a few 
adjustments to the filter database now shared with our network of 
sysops and a few of our FTP-only sysops turned it off for scaling 
reasons. A major plus in our constant battle against foreign attacks.

Thanks

-- 
HLS

---

From: Keith Moore <[email-masked]>
Date: Mon, 23 Dec 2019 13:44:24 -0500

> the lines of a connection response preempting the need to use EHLO? 

I believe that any SMTP solution has to be backward compatible, so 
incompatible solutions are non-starters.   At the same time I think 
there may be room to "pessimize" processing of certain kinds of traffic 
in order to encourage senders to upgrade.  (e.g. return "4xx try again 
later" in response to MAIL or DATA, at random, with probability < 1 but 
possibly increasing over time.)

I'm still not convinced that there is any "need" to use EHLO tags in 
spam filtering because the EHLO tag says absolutely nothing about the 
message content or even about whether the sender is misrepresenting 
itself in order to hide something.   (Or at least I don't see a reliable 
way of testing the EHLO tag to determine from the server's environment, 
whether it's valid in the client's environment.)   From a standards 
perspective I still think it makes sense to discourage use of the EHLO 
tag for spam filtering (maybe SHOULD NOT rather than MUST NOT), but also 
to explain why use of EHLO tags for that purpose is a Dubious Idea.

> In December 2018, I released an major update that included Geo IP 
> Location filtering.
>

I think it's useful to think of spam filters in two categories:

(a) stopgap countermeasures which may make sense in the short term and 
for specific conditions,  vs.
(b) countermeasures that make sense in the longer term and are broadly 
applicable.

I agree that sometimes the stopgap countermeasures are necessary, but I 
think it's important to distinguish the two and periodically review the 
stopgap measures to see if they're still applicable. Today, perhaps, a 
business may have no expectation of doing business with clients in 
particular countries.  But that may change in a year or two, at which 
point a filter that blocks inbound mail from Elbonia becomes harmful to 
the business.

Also, a country-specific block may make sense for some sites, or some 
users, but not for others.    Different users have different needs.   
Different sites also get different amounts and kinds of spam.

Another consideration is that some kinds of filters are very easy for 
spammers to circumvent, and spammers are sometimes more nimble than 
operators.   So if you're choosing how to filter a particular kind of 
spam, it might be prudent to pick a filter that is harder to circumvent 
over one that is easy - that may mean you have fewer fires to fight.   
If spammers can easily hide their tracks using botnets or other traffic 
laundering measures, country-specific blocks on source IP address may 
not be useful for very long.

The very nature of the IETF consensus process and the RFC publication 
process, with their inherent delays, means that IETF will have a much 
harder job documenting long-term countermeasures than stopgap 
countermeasures.

Keith

---

From: Russ Allbery <[email-masked]>
Date: Mon, 23 Dec 2019 10:51:19 -0800

> I think we need to recognize that spam filters are part of the problem.
> Just because a problem exists does not mean that any supposed fix is
> good.

I completely agree with this, and I also have seen a lot of bogus spam
filtering criteria.  But I think the question is harder: Is there any fix
that is good?  Or are there only least-bad fixes?

The basic problem is stark and difficult: 90-95% of email is spam,
malware, phishing, or other types of unwanted junk, and this mail is being
sent by intelligent (sometimes) and adaptive (occasionally) adversaries.
Meanwhile, in my experience the tolerance of the average person for junk
delivery into their inbox is *well* below 50%.  I haven't measured this,
so take with substantial salt, but anecdotally I would say that the
usability of email for a lot of people drops significantly if more than
10-20% of their inbox is junk, and many people want it lower than that.

Even harder, letting through even one or two malware or phishing messages
can be very dangerous, regardless of how good the percentages look.

Gmail in my experience does way better than that, getting down to about
1%, at the cost of some (but not that many) false positives.  For my
personal email, I use a Bayesian filter tuned solely for me and almost no
other filtering criteria and get more like 5% spam in my inbox, at a cost
of more false positives than Gmail.

The most effective standardized spam filtering techniques to date that
have retained effectiveness over time and not been fairly trivially
bypassed by spammers have been authentication approaches (SPF, DKIM,
etc.), which only handle certain classes of junk (but are particularly
helpful against phishing, which for most people is the most dangerous form
of junk).  Those now seem to be clearly good ideas, but by themselves seem
unlikely to achieve the necessary outcome.

What else is proven to a level that the IETF can standardize and recommend
it as a replacement to the ad hoc techniques that operators are using to
try to keep the ship from sinking?

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 23 Dec 2019 19:22:12 +0000

On 23/12/2019 17:59, Hector Santos wrote:
> sequence with the idea that the 2nd one may show additional capabilities
> after an successful AUTH takes place.

STARTTLS is used that way also - eg. to only advertise non-encrypted
AUTH methods once wrapped in the encrption.
-- 
Cheers,
  Jeremy

---

From: Keith Moore <[email-masked]>
Date: Mon, 23 Dec 2019 14:49:35 -0500

> that is good?  Or are there only least-bad fixes?

See my other message to Hector about different categories of spam filtering.

Clearly there are different degrees of "good" or "bad".   Part of what 
we might be able to do is define some criteria for goodness, for 
measurement of goodness, and for how often to re-evaluate goodness.    
(For example whenever someone quotes an FP rate without telling me how 
they measured it, I take it with a grain of salt.)

> can be very dangerous, regardless of how good the percentages look.
Agree.
> of more false positives than Gmail.

It's hard to meaningfully compare filter effectiveness at different 
domains, because different domains (and different users) get different 
amounts and kinds of spam.

My business email account has no spam filter.   It gets a decent amount 
of spam, but I also don't get a huge volume of mail at that account so 
manually filtering out the spam is still fairly easy. OTOH I have other 
accounts that have fairly aggressive spam filters, but 99% of the 
messages that make it through are still spam.

I also find that the user interface makes a huge difference in how 
tolerable spam is.   If I'm using a webmail interface that requires 
several clicks and page loads and other delays to deal with spam, the 
spam is more annoying than with my normal UA where I can fairly easily 
select a set of 40-50 messages, unselect the 2-3 messages that appear 
legitimate, and hit delete.

> unlikely to achieve the necessary outcome.

Another thing I'm wondering is whether it would help for IETF to 
recommend that MSAs that forward mail to arbitrary SMTP servers on the 
public Internet, sign their outgoing traffic (say with DKIM) and perhaps 
to define a profile for doing so.

More broadly speaking, can we effectively raise the bar for "legitimate" 
email in a well-defined way so that everybody knows what hoops they need 
to jump through rather than having to guess?   And can we raise the bar 
in such a way that favors legitimate mail over spam?

For instance, now that we have a spec for relaying using SMTP over TLS, 
if we specified that relay of mail from the MSA to the MX of the 
destination domain SHOULD use TLS with client certificates, then servers 
would have a more reliable and quicker way to identify and classify 
senders.

> try to keep the ship from sinking?

I don't know that we need to restrict ourselves to "proven" 
techniques.   I do think that some techniques have been around long 
enough that it might make sense to evaluate their effectiveness, but I 
don't think new ideas should be out-of-scope.

Keith

p.s. also I don't think this is likely to be "exciting" work, or work 
that we should crow about.  If successful, it's more likely to promote 
gradual improvement over time than any earth-shattering immediate 
change.  I think it's work that needs to be approached with very sober, 
level-headed judgment and analysis.

---

From: Sam Varshavchik <[email-masked]>
Date: Mon, 23 Dec 2019 18:24:25 -0500

Peter J. Holzer writes:

> already fine - but what you did.

No, it should be deleted because it will be ignored, in practice. It is very  
easy to understand why.

The reason you see various places doing this is because it works for them.  
As simple that. They've reached the conclusion that this check blocks a  
bunch of crap with negligible, if any at all, impact on their non-spam mail  
traffic.

However, no SMTP server I'm aware of does this by default, in its stock  
configuration.

If you see someone rejecting a HELO/EHLO with a non-matching IP address  
literal (or even any IP address literal), it's because two things must've  
happened earlier:

1) Their mail server turned out to be capable of doing this.

2) The organization that runs this mail server affirmatively decided to make  
their mail server do that.

So, in light of that, let's consider this statement:

> |   accept a message on that basis.

There are only two audiences for this statement:

1) Someone who uses an SMTP server.

2) Someone who implements an SMTP server.

Those are the only two possibibilities. I can't think of anyone else who  
will hear this, in their daily life.

So, we can now ask a few more questions. We already know that this already  
happens in ractice. How common it is, I don't know, but it is already in  
noticeable use. Therefore, we can break down the first kind of the audience  
for this RFC as follows:

3) What are the chances that someone who simply uses an SMTP server will  
read this particular RFC, and even if they do:

4) How likely are they to comply, and turn off this configuration setting,  
which they believe to be useful for them, based on what they just read.

I think that the answers to 3 and 4 are pretty easy. This is very unlikely  
to happen. So, what's left are SMTP server implementations, the second half  
of the potential audience for this RFC. Ok, so let's consider the impact on  
SMTP server implementation.

Now, keep in mind that this is going to be just an optional feature which is  
very unlikely to be enabled by default. So, how likely is that an SMTP  
server implementation will either:

5) Remove the feature which, apparently, is popular with those who use this  
SMTP SMTP implementation, or at least some non-trivial portion of the users,  
or:

6) A new SMTP server implementation will decline to implement such a (semi)- 
popular feature in the first place.

I don't think that this is very likely to happen. Therefore, I expect that  
"MUST NOT" provision, if it exists already or will exist, to be mostly  
ignored.

---

From: Keith Moore <[email-masked]>
Date: Mon, 23 Dec 2019 23:37:39 -0500

On 12/23/19 6:24 PM, Sam Varshavchik wrote:

> The reason you see various places doing this is because it works for 
> them. As simple that. They've reached the conclusion that this check 
> blocks a bunch of crap with negligible, if any at all, impact on their 
> non-spam mail traffic.

I would not draw that conclusion, at least without additional 
information.   My conclusion would be that you see such practices 
because a site believed that it worked for them at sometime in the 
(perhaps distant) past.   But something I've seen across many different 
organizations is that fixes tend to be seen as necessary indefinitely, 
even long after they've outlived their usefulness or caused more 
problems than they solved.

Another thing I've often seen is that measurement of the effectiveness 
of such measures is poor to nonexistent.

Another thing I've often seen is that "running code", however badly 
running, is often taken as the ultimate authority for how things 
"should" work, no matter what the rules or policies or business 
interests are, because the people who are maintaining such code don't 
want to take the risk of disrupting operations. (often because 
operations is understaffed)

But perhaps the fundamental problem is that too many people actually 
don't consider reliable email delivery as an essential service, and 
that's why they justify throwing away messages for dubious reasons.

Keith

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 24 Dec 2019 12:30:44 +0100

On Mon 23/Dec/2019 20:49:35 +0100 Keith Moore wrote:
> Another thing I'm wondering is whether it would help for IETF to recommend that
> MSAs that forward mail to arbitrary SMTP servers on the public Internet, sign
> their outgoing traffic (say with DKIM) and perhaps to define a profile for
> doing so.
> 
> More broadly speaking, can we effectively raise the bar for "legitimate" email
> in a well-defined way so that everybody knows what hoops they need to jump
> through rather than having to guess?   And can we raise the bar in such a way
> that favors legitimate mail over spam?
> 
> For instance, now that we have a spec for relaying using SMTP over TLS, if we
> specified that relay of mail from the MSA to the MX of the destination domain
> SHOULD use TLS with client certificates, then servers would have a more
> reliable and quicker way to identify and classify senders.

There seems to be a common path, shared among SPF, ADSP, and DMARC.  At some
times, it was common belief that universal adoption of the token protocol would
stop spam.[*]  The protocol provides for soft or hard settings, so spam
fighters push for hard settings, while experts warn that the intended use of
the protocol differs.  Eventually, MTAs settle for soft settings.  Next.

The latter stage hasn't yet come for DMARC.  For example, SM found an a web
tool which diagnoses the IETF "Not [having] all authenticity marks against
email phishing (DMARC, DKIM and SPF)".[†]  That's not the only web tool doing so.

Spam comes in different shades.  The best we can hope for email authentication
is to eliminate just the anonymous spam, that which provides false identities.
 With a little additional effort, trash-domain authentications can be
recognized as well.  However, hard settings for everybody entails some change
in SMTP semantics, for SPF it was forwarding, for ADSP and DMARC mailing lists.

I don't think rfc5321bis should address email authentication directly, but
reshaping Section 3.9 might help.

Best
Ale
-- 

[*] The FUSSP won't be effective until it has been deployed at more than 60% of
SMTP servers and that's not a problem.
https://www.rhyolite.com/anti-spam/you-might-be.html#senior-IETF-member-5

[†] https://internet.nl/mail/ietf.org/294467/
cited by SM as http://r.elandsys.com/r/94862 in
https://mailarchive.ietf.org/arch/msg/ietf/6PxMGowo_74RuXNWxfZf-8lfdO8

---

From: Keith Moore <[email-masked]>
Date: Tue, 24 Dec 2019 10:11:14 -0500

> the protocol differs.  Eventually, MTAs settle for soft settings.  Next.
Well, maybe it's time to raise the bar a bit, and maybe IETF can help.
> is to eliminate just the anonymous spam, that which provides false identities.
It might also help if people can be held accountable for the mail that 
they send.   If the authentication works well enough, it basically 
forces spam and malware to go anonymous, to keep the senders from being 
held legally accountable.   Of course, advertisers will continue to 
believe that what they are doing is legitimate, but if they're forced to 
authenticate themselves using real names, it will be easier to 
distinguish their mail from that of reasonable people.
> reshaping Section 3.9 might help.

I agree that 5321bis is not the place to address most of these things, 
though there might be slight changes to the language.  But I've been 
assuming throughout this discussion that efforts to improve spam 
filtering would be separate from the revision of 5321.   Sorry if I 
haven't made that clear.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Tue, 24 Dec 2019 07:24:54 -0800

> Well, maybe it's time to raise the bar a bit, and maybe IETF can help.

For this to be a useful exercise, there are two, foundational actions 
that need to be taken:

1. Recruit significant participation from the folk in the anti-abuse 
industry who have been working on this topic for 25 years.

2. Get everyone else who participates to be far more familiar with that 
25 year history and the pragmatics and real challenges of current 
anti-abuse efforts.

Only then will beginning such a conversation be anchored in messy 
practice rather than appealing theory.

d/

---

From: John C Klensin <[email-masked]>
Date: Wed, 25 Dec 2019 00:11:00 -0500

--On Monday, December 23, 2019 18:24 -0500 Sam Varshavchik
<[email-masked]> wrote:

>...
> Peter J. Holzer writes:
> 
>> The discussion did however strengthen my belief that the
>> paragraph from section 4.1.4:
>...
>> ought to be deleted. Not to legalize what the IETF admin did
>> - that's already fine - but what you did.
> 
> No, it should be deleted because it will be ignored, in
> practice. It is very easy to understand why.
> 
> The reason you see various places doing this is because it
> works for them. As simple that. They've reached the conclusion
> that this check blocks a bunch of crap with negligible, if any
> at all, impact on their non-spam mail traffic.
> 
> However, no SMTP server I'm aware of does this by default, in
> its stock configuration.

Sam,

I want to get an oar in the water about an alternate point of
view, not because I'm convinced it is the right one but because
the conclusion you describe above may be problematic.

_Any_ envelope-level spam filtering technique other than, maybe,
specifically identifying a particular bad actor and rejecting
mail from them, is going to be subject to false positives --
legitimate messages that are incorrectly rejected.   Anyone who
reaches the conclusion you describe, whether they understand it
or not, is making the decision that losing some number (a number
that is actually very hard to estimate accurately) of legitimate
messages is ok if it makes a big dent in the spam.   That is ok,
but it doesn't work well in environments in which reliable
delivery, i.e., not losing legitimate messages, is important.
And, in situations where it counts, the task of a mail
administrator who has to explain to her boss's boss why a
message from an important customer was rejected and didn't get
through is, well, not enviable.

Of course, if said boss's boss gets a spam-delivered phishing
message and stupidly responds to it in a way that puts the
organization at risk, having to explain why that was allowed to
happen is not likely to be much fun either (see "disclaimer" at
the end).

If one decides to block attempts to open SMTP sessions by
rejecting IP literals at EHLO time (a singularly blunt
instrument, even more blunt, IMO, than rejection based on IP
address ranges, there are also two ways of doing it.  One is to
return a 5yz code in response to the EHLO, thereby rejecting all
messages using such syntax regardless of origin or destination.
The second is to wait until the MAIL command, or maybe even one
or more RCPT commands, are received, thereby allowing
whitelisting if there are particular cases one wants to allow.
In the particular case of the IETF servers and posted anti-spam
policy, that would allow an IETF participant with special
circumstances to argue that their mail traffic should be
accepted even if they are somehow restricted to IP address
literals in EHLO.  I read the policy statement as requiring that
there be a possibility for such exceptions; others may not.  For
other lists and operators, if we were going to drop IP address
literals (a feature that has been allowed and supported for more
than 37 years, much longer if one counts email over FTP), I
think we have some obligation to explain why we are doing that
and just how to do it, including the tradeoffs.

I have quibbles with the analysis in the rest of your note, but
prefer to see them worked out in more extensive discussion than
to write a longer message now.

   john

Disclaimer: At the risk of saying something that some will
construe as political, while I understand the need for spam
filtering at or near the recipient end of the mail flow, I fear
that we are not doing ourselves or the Internet any favors by
getting incrementally better at it.  In the current model, the
higher a percentage of spam we intercept at the receiver end and
prevent from reaching the user, the more the spammers have
incentive to increase their output volume as well as their
technology for bypassing our filters.  The combination drives up
costs to mail service providers and creates more incentives for
concentration.   If there is a solution to spam, it is almost
certainly lies in general recognition of spamming as an
antisocial act and the creation of social, legal, and financial
disincentives to spamming.  That recognition will probably never
come as long as we are successful in preventing enough of it
from reaching legislators and their active constituents so that
they don't feel significant pain.  I'm not quite serious about
this, but, if one wanted to really stop spam by making life
painful for the spammers, the most useful possible step might be
to declare a "let all of it go through to the end users" week
every once in a while.

---

From: Keith Moore <[email-masked]>
Date: Wed, 25 Dec 2019 01:14:55 -0500

On 12/25/19 12:11 AM, John C Klensin wrote:

> At the risk of saying something that some will
> technology for bypassing our filters.

I certainly agree with this much.

Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 25 Dec 2019 08:57:17 -0500

--On Tuesday, December 24, 2019 07:24 -0800 Dave Crocker
<[email-masked]> wrote:

> For this to be a useful exercise, there are two, foundational
> actions that need to be taken:
> 
> 1. Recruit significant participation from the folk in the
> anti-abuse industry who have been working on this topic for 25
> years.
> 
> 2. Get everyone else who participates to be far more familiar
> with that 25 year history and the pragmatics and real
> challenges of current anti-abuse efforts.
> 
> Only then will beginning such a conversation be anchored in
> messy practice rather than appealing theory.

Dave,

Yes, but...

Pursuant to my disclaimer/rant at the end of my last note, we
should also not lose track of what you have quite properly
called an industry and others have characterized as an arm race.
Everyone I have met and talked with in that industry has been
absolutely sincere about reducing or stopping spam.  But
business models do depend on it and, like other types of arms
races, the pattern is one of escalation, increasingly complex
(and expensive) measures, and rarely, if ever, long-term
reductions in the problem (at least by many measures).  

If we want to see mechanisms that focus on the spam originators
and spammers, rather than the receiving end, focusing on people,
and especially mechanisms, that have characterized efforts for
most of the last 25 years of effort may not be the best answer.
I agree that trying to develop new approaches without
understanding the old ones would be unwise but, if one wants new
approaches, getting over-focused on the older ones may be unwise
too.   That is, IMO, completely consistent with your suggestion
about familiarity with pragmatics and real challenges of the
efforts, but also with the pragmatics and real challenges of the
underlying problem.

best,
   john

---

From: Sam Varshavchik <[email-masked]>
Date: Wed, 25 Dec 2019 10:44:14 -0500

John C Klensin writes:

> messages is ok if it makes a big dent in the spam.

Correct.

I'm sure there are occasional "anyones" who believe that their subjective  
mail filtering criteria is 100% perfect; I do believe that the overwhelming  
majority who do this already understand this, and they accept it.

>                                                      That is ok,

And those environments will simply choose not to use subjective mail  
filtering rules. Noone's forcing them to do this. Furthermore, this is the  
default state of affairs. I don't know out of any SMTP servers that, out of  
the box, has a bunch of subjecting mail filtering rules enabled. Everyone  
who uses subjective mail filtering policies, today, made their own decision  
to do so; and in my last message I explained why making this a "MUST NOT" is  
unlikely to make any difference. That's just what I think; perhaps  
eventually it turns out that my reasoning didn't held, and everyone ends up  
stopping filtering out various things in HELOs, or whatnot, because the RFC  
says they "MUST NOT" do this.

> through is, well, not enviable.

That depends on what happened in the first place – whether the mail  
administrator did that on their own volition, or because the same boss  
complained "I'm getting too much spam, can you do anything about it", the  
mail administrator explained the options, but the boss's eyes started  
glazing over hearing all the technical mumbo-jumbo, and the boss just waved  
their hand "just do it". So now the mail administrator explained that the  
mail was rejected because the boss approved the change, and the boss will  
simply tell the mail administrator to undo it, then, and everyone lives  
happily ever after.

> whitelisting if there are particular cases one wants to allow.

I agree, and that's what I do. But I also recognize that other  
implementations don't, and I'm somewhat skeptical that they'll find this  
argument persuasive.

> from reaching legislators and their active constituents so that

I do not believe that any social, legal, or financial disincentive will have  
any measurable difference whatsoever.

Unless something is adopted worldwide, spamming will simply move its  
origination point. Some of the spammers will not be able cover their tracks,  
very well, and a few dumb ones will get caught. But most will continue on,  
mostly unimpeded.

> to declare a "let all of it go through to the end users" week

There have been similar kinds of boycotts done in the past, in other  
contexts. I can't think of even one that succeeded.

And for this particular one: the target audience for this declaration is,  
basically, a herd of cats. And without a buy-in from most of the herd, there  
won't be much of an impact to speak of.

In the US, I think this will require participation from at least gmail and  
hotmail/outlook. I can't imagine why they will want to. Their existing  
customer-facing mail filtering is good enough already, and I doubt that any  
major sources of spam will ever find a way to defeat them. And without their  
involvement, to turn off all spam filtering for a day, nobody will notice.

The situation outside of US, I suspect, will mostly be the same.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Dec 2019 08:22:31 -0800

> most of the last 25 years of effort may not be the best answer.

If you want to formulate proposals that have any practical basis, it 
requires knowing quite a bit about the pragmatics of spam today.  There 
is nothing new about the idea of seeking 'strategic' change; there have 
been many over the last 25 years.  However such proposals typically are 
not burdened by requisite knowledge and typically embody exactly the 
naivete one would expect. Similarly, proposals need to attend to the 
realities of human factors, which is similarly lacking in typical proposals.

d/

---

From: Keith Moore <[email-masked]>
Date: Wed, 25 Dec 2019 11:43:21 -0500

On 12/25/19 11:22 AM, Dave Crocker wrote:

> requires knowing quite a bit about the pragmatics of spam today.  
> There is nothing new about the idea of seeking 'strategic' change; 
> there have been many over the last 25 years.  However such proposals 
> typically are not burdened by requisite knowledge and typically embody 
> exactly the naivete one would expect. Similarly, proposals need to 
> attend to the realities of human factors, which is similarly lacking 
> in typical proposals.
>
If existing spam filtering practice were working well today, we wouldn't 
be having this discussion.   More broadly, there's always a tension 
between doing things the old way, and seeking better ways to approach a 
problem.   There are always advantages to doing things the old way, 
especially for companies that have built products around doing that.   
There will always be some naiveté
with new proposals.   And yet, without new ideas, there's a huge 
tendency for old approaches to eventually do more harm than good.    
This isn't at all specific to email or networking or engineering, it 
also exists in medicine and in law.

In a sense, of course, you're right.   But I think it's a bit too early 
for damage control.  If we ever get around to forming a working group we 
can expect that it will attract interest from everybody who has an 
existing anti-spam product, all looking for ways to defeat anything that 
IETF might do that would hurt their market share, no matter how good or 
harmful their products are.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Dec 2019 09:01:36 -0800

> be having this discussion. 

That spam filtering practice today is working well is the only reason we 
are able to have this discussion (over email)...

Note that ignoring the expertise of the anti-abuse industry is not a new 
starting point.  Also note that the track-record of efforts with that 
foundation of ignorance is entirely predictable.

> More broadly, there's always a tension 
> problem.  

Nothing in what I've posted advises doing things in an old way.

Innovation is fine.  Innovation based on ignorance of a long history and 
current complexities is not.

> with new proposals. 

Oh?

d/

---

From: Keith Moore <[email-masked]>
Date: Wed, 25 Dec 2019 12:04:03 -0500

On 12/25/19 12:01 PM, Dave Crocker wrote:

>> If existing spam filtering practice were working well today, we 
>> wouldn't be having this discussion. 
>
> That spam filtering practice today is working well is the only reason 
> we are able to have this discussion (over email)...

LOL.   You know better than that.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Dec 2019 09:13:53 -0800

> LOL.   You know better than that.

Your doubting my assessment seems to demonstrate the problem I'm 
describing.  It nicely ensures the predictable outcome I cited.

d/

---

From: Keith Moore <[email-masked]>
Date: Wed, 25 Dec 2019 12:20:46 -0500

On 12/25/19 12:13 PM, Dave Crocker wrote:

>>> That spam filtering practice today is working well is the only 
>>> reason we are able to have this discussion (over email)...
>>

Actually, your handwaving argument is the perfect demonstration of why 
this work is needed.

Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 25 Dec 2019 19:29:11 -0500

--On Wednesday, December 25, 2019 08:22 -0800 Dave Crocker
<[email-masked]> wrote:

>> If we want to see mechanisms that focus on the spam
>> originators and spammers, rather than the receiving end,
>> focusing on people, and especially mechanisms, that have
>> characterized efforts for most of the last 25 years of effort
>> may not be the best answer.
 
> If you want to formulate proposals that have any practical
> basis, it requires knowing quite a bit about the pragmatics of
> spam today.  There is nothing new about the idea of seeking
> 'strategic' change; there have been many over the last 25
> years.  However such proposals typically are not burdened by
> requisite knowledge and typically embody exactly the naivete
> one would expect. Similarly, proposals need to attend to the

I don't see anything about that with which I disagree.  If my
previous comments seemed inconsistent with that I didn't explain
clearly enough.  While I would (and did) argue for a change in
direction and/or strategy rather than simply continuing with the
anti-spam arms race, I think any proposal for such a change and
direction or strategy must be rooted in a deep understanding of
what the environment is like, what has been done, and what the
strengths and weaknesses of those approaches have been.  Without
that understanding, while failure -- perhaps even failure with
nasty side-effects -- is not certain, it is definitely very,
very, likely.

   john

---

From: John C Klensin <[email-masked]>
Date: Wed, 25 Dec 2019 20:31:50 -0500

Responding to multiple, relatively small, issues...

--On Monday, December 23, 2019 23:37 -0500 Keith Moore
<[email-masked]> wrote:

>...
> But perhaps the fundamental problem is that too many people
> actually don't consider reliable email delivery as an
> essential service, and that's why they justify throwing away
> messages for dubious reasons.

Or they see it as a "what you get is what you get" service,
which is not quite the same as not considering reliable email
delivery as essential, but close.

--On Wednesday, December 25, 2019 10:44 -0500 Sam Varshavchik
<[email-masked]> wrote:

> That depends on what happened in the first place – whether
> the mail administrator did that on their own volition, or
> because the same boss complained "I'm getting too much spam,
> can you do anything about it", the mail administrator
> explained the options, but the boss's eyes started glazing
> over hearing all the technical mumbo-jumbo, and the boss just
> waved their hand "just do it". So now the mail administrator
> explained that the mail was rejected because the boss approved
> the change, and the boss will simply tell the mail
> administrator to undo it, then, and everyone lives happily
> ever after.

You have clearly been hanging around with a better class of boss
than some of the ones I had in mind.  For the latter, if a
problem occurs, even if was clearly their own fault, the normal
response would be to find someone to blame and throw under a
passing bus.  I certainly prefer your more positive view of the
world, bosses, and likely outcomes, but...

--On Sunday, December 22, 2019 14:59 -0500 Viktor Dukhovni
<[email-masked]> wrote:

> Of course if one wants to send mail to "luser@[192.0.2.1]"
> that'll work too (but that address form isn't by default
> accepted via SMTP from clients that are neither on a local
> network nor by some means authenticated, an open-relay
> configuration als possible, though it is not likely to be
> enabled just by accident.

It also prevents any sort of MX resolution, so whatever is at
192.0.2.1 had better be (at least as far as SMTP is concerned)
the final delivery host (and, for that particular address, very
local).  Of course, in some situations, that might be considered
an advantage.

--On Saturday, December 21, 2019 17:50 -0500 Keith Moore
<[email-masked]> wrote:

> The interesting thing to me is that if you don't have a valid
> DNS name, an IP address literal seems like the right thing to
> use. Even if behind a NAT at least it's some identifier that
> is associated with the sending host that can go in a Received
> header, still potentially useful for tracing if the server has
> logs showing the source IP address.   An EHLO tag based on a
> boilerplate hostname and boilerplate suffix is essentially
> useless for that purpose.

And, while the wording is different, that is essentially what
5321 (and 2821 before it) say.  So I think you are suggesting
that the core requirement is fine and that, if anything is
needed, it is better language about when it may, or may not, be
a good practice.

> 'shortname.localdomain' wouldn't be valid.

With several qualifications, yes.  The first is that 5321 (IMO,
intentionally) does make careful distinctions between the public
DNS and various more local arrangements.   Section 2.3.5
requires FQDNs, but "shortname.localdomain" is, at least by the
rules described in that section, an FQDN.  The second
qualification is that Section 4.1.4 prohibits rejecting an EHLO
command based on the domain name not resolving to the same
address that originated the connection, so it is not clear what
the implications of "not valid" are in terms of what a
conforming SMTP server should or should not do.

--On Saturday, December 21, 2019 17:20 -0500 Viktor Dukhovni
<[email-masked]> wrote:

>> So if you don't have an explicitly configured hostname, do
>> you simply refuse to send messages?   Or send EHLO
>> example.com?  or what?
> 
> gethostname(). And if that contains no dots, and "mydomain" is
> not set, then the domain defaults to "localdomain".  So worst
> case you get "shortname.localdomain".

Victor, while what you are describing would be a perfectly
reasonable implementation, 5321 doesn't know a thing about
gethostname() (and, given issues with stable references and
independence of particular programming languages and operating
systems, probably shouldn't).  Whether we now think 5321 is
correct or not going forward, it is very specific about what
happens if the sending host decides, for whatever reason, that
it has no appropriate FQDN to use and that it is to use the IP
address literal.  

Carrying Keith's analysis a bit further, if your systems want to
reject EHLO commands with IP address literals on the grounds
that you assume whomever controls the connection-originating
host is a spammer, and idiot, or both and you don't want mail
from anyone sending from a host run by such people, no one is
going to say you can't do that and, if they did, you would
presumable ignore them.  But changing SMTP to prohibit such
literals would essentially ban any host without an FQDN, maybe
even an FQDN on the public Internet, from participating in the
mail system (except, maybe, via a Submission Server that would
sort things out) would be a rather big step.  The same comment,
btw and incorporating another thread by reference, applies to
any system or host that, for whatever reasons, can't acquire a
valid and usable reverse-mapping record, especially one that
will return the same FQDN as the one it intends to use in EHLO.
And I still think that special considerations apply to SMTP
servers that appear to represent the IETF and its practices.

   john

---

From: Sam Varshavchik <[email-masked]>
Date: Wed, 25 Dec 2019 20:58:46 -0500

John C Klensin writes:

> strengths and weaknesses of those approaches have been.

I don't think there's anything here that requires deep understanding. The  
facts are pretty basic, and fundamental. The "P" in "SMTP" is as "S" as they  
come. It does its job with a minimum of fuss. Its job, of serving as means  
of delivering mail, gets done in the most straightforward manner imaginable.

I don't think any changes here are in the cards. I doubt that any change to  
SMTP will accomplish anything. And I already mentioned that I do not believe  
that any societal or legal changes will accomplish anything more than the  
CAN SPAM act of 2003 has actually accomplished. What difference has it made? 
What tangible result can anyone point out, that it accomplished towards the  
goal of controlling spam? I wasn't the only one who laughably predicted its  
utter uselessness in that regard, when it went up the flagpole. I don't see  
any non-technical measure being able to produce any better results than the  
CAN SPAM act.

I think that any solution to spam can only be a technical solution. But I do  
not think it involves SMTP per se, but be something at the same level of the  
protocol stack as DMARC is. DMARC itself proves that the mail originated  
from its purported sender. I recall that the original advocacy for DMARC  
included some claims of its purported anti-spam benefits. I never saw how  
proving that mail really came from a domain proved whether it's spam or not.  
A healthy portion of my spam folder is DMARC-signed, so that does a lot of  
good, in that regard… But once something along the same lines defines  
whether the mail was solicited by its recipient, only at that point you can  
have SMTP enter the picture and have SMTP servers check it and have an  
option of rejecting unsolicited email.

But for defining the anti-spam technical solution itself, that SMTP really  
doesn't need to be involved in this particular discussion. SMTP does the job  
it's assigned to do. And, that's it, nothing else needs to be said. First,  
it's necessary to have a technical approach to defining whether mail was  
solicited by its recipient. Implement this first, with SMTP out of the  
picture. Now, once you have done, an SMTP server can stick its nose, take a  
sniff, and accept or reject mail on that basis, if such option is available.

---

From: Keith Moore <[email-masked]>
Date: Wed, 25 Dec 2019 23:57:28 -0500

> needed, it is better language about when it may, or may not, be

Yes.  But "good practice" is in keeping with the purpose of EHLO, which 
is _not_ intended to be used for spam filtering.   Any spam filtering 
practice based on EHLO tags is inherently dubious. Even if it happens to 
"work" for some site or another at some particular time, it's not 
defensible as good practice in general.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Dec 2019 21:03:43 -0800

On 12/25/2019 8:57 PM, Keith Moore wrote:
> Any spam filtering practice based on EHLO tags is inherently dubious. 
> Even if it happens to "work" for some site or another at some particular 
> time, it's not defensible as good practice in general.

There is an appealing purity to such a view.  However...

Given a context that really is an arms race, criticizing an effective 
anti-abuse effort as being not defensible winds up having only two 
effects.  One is to make anti-abuse efforts less effective and the other 
is to alienate the folk whose job it is to do anti-abuse work.

It seems unlikely that either of those outcomes is defensible.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 00:06:43 -0500

On 12/26/19 12:03 AM, Dave Crocker wrote:

>> Even if it happens to "work" for some site or another at some 
>> particular time, it's not defensible as good practice in general.
>
>
> effects.  One is to make anti-abuse efforts less effective and the 
> other is to alienate the folk whose job it is to do anti-abuse work.
>

It seems unlikely that practices that further encourage the arms race, 
and further degrade the reliability of email, are defensible.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Dec 2019 21:12:12 -0800

> and further degrade the reliability of email, are defensible.

"Encourage" the arms race???  Wow.

So, ummm, you think that the bad actors will act less badly if the 
receive-side filtering engines are less effective?  That's the simple 
implication of saying that applying effective methods encourages their 
bad behaviors.

Or to seek a more affirmative tone:

    What exactly is the alternative that will be at least as effective 
as what is currently being done?

    And by asking this, I'm asking for tested practice, not untested 
theory.  The latter is almost always more appealing, since it benefits 
from not having to deal with the messiness of the former.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 00:20:00 -0500

On 12/26/19 12:12 AM, Dave Crocker wrote:

>> It seems unlikely that practices that further encourage the arms 
>> race, and further degrade the reliability of email, are defensible.
>
>

No, and I didn't say anything of the sort.

What I will say is that relying on filtering criteria that are 
completely unrelated to the content of the message not only degrades the 
reliability of email over time, and also makes it easier for the bad 
actors to game the system.... particularly when, as is likely for EHLO 
tags, those filtering criteria are used to the exclusion of everything else.

> as what is currently being done?

I've already explained this, and don't believe it's effective to repeat 
my previous arguments at this time.     And your arguments are without 
substance.

> And by asking this, I'm asking for tested practice, not untested theory.

Try putting some rigor in your own arguments then.    Even "tested 
practice" is useless without actual measurements and a description of 
how those measurements were conducted.

And if you refuse to consider anything that doesn't already exist, 
you're insisting that we continue to be stuck with the unacceptable 
status quo.   Internet users deserve better than that.

Keith

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 26 Dec 2019 12:13:28 +0100

On Wed 25/Dec/2019 06:11:00 +0100 John C Klensin wrote:
> That recognition will probably never come as long as we are successful in
> preventing enough of it from reaching legislators and their active
> constituents so that they don't feel significant pain.  I'm not quite
> serious about this, but, if one wanted to really stop spam by making life 
> painful for the spammers, the most useful possible step might be to declare
> a "let all of it go through to the end users" week every once in a while.

+1 for the international Spam Impact Day!
(week is probably too much)

Best
Ale
--

---

From: Paul Smith <[email-masked]>
Date: Thu, 26 Dec 2019 11:20:22 +0000

> too.

My personal view is that the ONLY way we'll get a significant and 
objective removal of spam is if we have some form of central 
authorisation (not authentication) system for sending email. This could 
be based on DNS/client certificates/something else.

(What I mean by 'authorisation' requires a central body (or network of 
central bodies) with whom you can register your email senders with 
information about which IP addresses and/or domains are used. There has 
to be some (small) cost in either effort, time or money to register (to 
discourage abusive registrations while not preventing legitimate ones) 
and so on.)

I see and (at least partially) understand the resistance to that idea, 
so I wouldn't dare propose it as a solution. However...

In essence, I actually think we're moving towards that anyway, with the 
big email providers (gmail, Microsoft) essentially becoming their own 
ecosystem which includes a form of "authorisation" and with spam DNS 
RBLs/whitelisting orgs doing stuff. But, at the moment it's happening 
with no form of open discussion and with no standardisation. Maybe 
accepting that it's happening and working together to get a good 
standardised system would be better than pretending it's not happening 
and having a disjointed mess?

--

---

From: Sam Varshavchik <[email-masked]>
Date: Thu, 26 Dec 2019 06:46:18 -0500

Paul Smith writes:

> In essence, I actually think we're moving towards that anyway, with the big  
> email providers (gmail, Microsoft) essentially becoming their own ecosystem  
> which includes a form of "authorisation" and with spam DNS RBLs/whitelisting  
> orgs doing stuff. But, at the moment it's happening with no form of open  
> discussion and with no standardisation. Maybe accepting that it's happening  
> and working together to get a good standardised system would be better than  
> pretending it's not happening and having a disjointed mess?

I do not see this happening. Gmail and Microsoft still have their customers  
to answer to, whether they're paying or not. If they start spam-binning  
their customers' E-mails that do not originate from sources that they bless  
(with the blessing required in advance, via some form of registration), it's  
going to turn into a major dumpster fire very quickly.

Not to say that I don't see one of them trying something like that. But, the  
resulting crapstorm is inevitable, and the whole thing will get rolled back  
within 24 hours after hitting the news.

And, in any case, the high volume spam senders will easily register and  
authorize themselves, I see nothing that'll prevent that, and we'll be back  
to the current status quo.

I do not see any solution based on some central authority, sitting on a  
throne and giving mail senders an official blessing, as a workable solution.  
The problem with spam isn't the sender's identity. Never was. Never will be.

---

From: Richard Clayton <[email-masked]>
Date: Thu, 26 Dec 2019 13:17:39 +0000

In message <[email-masked]>,
Keith Moore <[email-masked]> writes

>    Yes.  But "good practice" is in keeping with the purpose of EHLO, 
>    which is _not_ intended to be used for spam filtering.   Any spam 
>    filtering practice based on EHLO tags is inherently dubious.   Even 
>    if it happens to "work" for some site or another at some particular 

The highly effective spam filtering performed by the giants (who handle
billions of emails a day) is done by "machine learning" classifiers.

These classifiers collect a wide range of indicators of which EHLO (and
indeed HELO) will be one contributor of a numeric value indicating some
sort of view "correctness" or perhaps just a score to indicate the type
of string presented. These values will contribute to the classification
of email into spam/non-spam accept/reject etc.

It would not be expected that any particular feature would by itself
provide reliable classification -- hence the ubiquity of ML. Also
(depending on the type of ML used) it may be hard to say what exact
contribution towards the final decision is being made by any particular
feature (viz: we're not going to settle the present discussion by asking
AOL, Apple, Google, Microsoft and Yahoo whether HELO/EHLO matters
because they may well not know -- and the answer may vary week by week).

It should also be noted that the most competent senders of spam are
often the most squeaky-clean when it comes to conformance to the RFCs
(they send enough to quickly learn when their set-up is affecting their
delivery rates) and it is only the sloppier spammers and the rather more
slap-dash (or indeed the most set-up challenged) of legitimate senders
who fail to conform.

There are no silver bullets in this space -- and impacting legitimate
senders is seldom a way to improve customer satisfaction with your mail
reception service.

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Dec 2019 07:46:28 -0800

> status quo.   Internet users deserve better than that.

What I've been trying to suggest is that further engineering in this 
space be based on deep knowledge of the realities present in the conduct 
of online abuse, not idealized theory or, worse, denial of experience.

It's fine to explore enhancements to theory, but not in a standards 
group.  When a proposal has no operational foundation, it's called research.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 10:59:43 -0500

On 12/26/19 10:46 AM, Dave Crocker wrote:

> space be based on deep knowledge of the realities present in the 
> conduct of online abuse, not idealized theory or, worse, denial of 
> experience.

In some sense, I agree.  But one should not legitimize all existing 
practice as proven by experience, without conducting actual measurements 
to support such assertions.

> It's fine to explore enhancements to theory, but not in a standards group.

By that argument, there would never be any new standards produced, and 
nothing to counteract the entropy that plagues the Internet.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Dec 2019 08:11:35 -0800

On 12/26/2019 7:59 AM, Keith Moore wrote:
>   But one should not legitimize all existing

Hence my pressing for folk to learn more about current realities from 
folks with years or decades of hands-on experience, which happen to 
include quite a bit of measuring.

>> group.

Merely as an example:

    What Makes for a Successful Protocol?
    https://tools.ietf.org/html/rfc5218

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 11:19:07 -0500

> include quite a bit of measuring. 

By all means, cite documents that present such results.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Dec 2019 08:30:47 -0800

> By all means, cite documents that present such results.

There are two problem with that request.  The first is that it isn't 
specific, so we would wind up playing a guessing game of what will 
satisfy you.

The second is that this isn't an issue that can be satisfied with a 
point response.  The entire process of developing proposals needs to be 
based on deep and pragmatic experience, not just reference to one or 
another study.

Hence it requires meaningful participation of folk with that experience.

But if you want to do the work of formulating a specific proposal, which 
lacks that participation, and putting it forward, please do.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 12:00:50 -0500

Sent from my iPhone

> On Dec 26, 2019, at 11:30 AM, Dave Crocker <[email-masked]> wrote:
> 
> There are two problem with that request.  The first is that it isn't specific, so we would wind up playing a guessing game of what will satisfy you.

Shall we rather constrain ourselves to do only what satisfies you?

> The second is that this isn't an issue that can be satisfied with a point response.  The entire process of developing proposals needs to be based on deep and pragmatic experience, not just reference to one or another study.

Problem is, there’s no way to evaluate experience without measurement.  

> Hence it requires meaningful participation of folk with that experience.

I don’t actually disagree with this.  But my experience is that when you’re first  trying to put together a proposal, the last thing you need is people looking for reasons to shoot it down.  What you need instead is people trying to figure out how it can be made to work.  And it can help to have experienced people in that discussion, if (and only if) they can still be flexible and creative.  But it can also help to have less experienced people in that discussion, because such people can often see potential approaches that the experienced people can’t.  Especially if circumstances have changed since the experienced people wrote off that idea, or a similar idea, as unworkable. 

Later on, of course, new ideas need to be vetted by a broader community before any significant scale of implementation should be encouraged.  

> 
> But if you want to do the work of formulating a specific proposal,

Well, that’s always been my idea.  I certainly don’t expect, and wouldn’t ask, for a working group to be formed without fairly concrete proposals in place.  (And perhaps some running code for demonstration and experiment.)

What I’m trying to do at the moment is see how many people are actually interested in improving the reliability of internet email, and who are willing to brainstorm about how this might be accomplished.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Dec 2019 09:07:50 -0800

On 12/26/2019 9:00 AM, Keith Moore wrote:
> ut my experience is that when you’re first  trying to put together a proposal, the last thing you need is people looking for reasons to shoot it down.

Welcome to the world of security design.  Carefully and thoroughly 
considering creative threat vectors is a requisite part of the design 
process.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Dec 2019 12:14:56 -0500

> On Dec 26, 2019, at 12:08 PM, Dave Crocker <[email-masked]> wrote:
> 
> Welcome to the world of security design.  Carefully and thoroughly considering creative threat vectors is a requisite part of the design process.

Indeed, and that’s what I spend a fair amount of my time doing.

Keith

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 26 Dec 2019 19:25:07 +0100

On Thu 26/Dec/2019 02:58:46 +0100 Sam Varshavchik wrote:
> John C Klensin writes:
> 
>> While I would (and did) argue for a change in direction and/or strategy
>> rather than simply continuing with the anti-spam arms race, I think any
>> proposal for such a change and direction or strategy must be rooted in a
>> deep understanding of what the environment is like, what has been done,
>> and what the strengths and weaknesses of those approaches have been.> 
> protocol stack as DMARC is. DMARC itself proves that the mail originated from
> its purported sender. I recall that the original advocacy for DMARC included
> some claims of its purported anti-spam benefits. I never saw how proving that
> mail really came from a domain proved whether it's spam or not.

The strategy of email authentication —referenced at least since 2004[*] (15yrs ago)— is to do coarse-grain domain-based identification.  Of course, authentication has to be complemented by authorization and reputation.  Authentication hit the mark.  IME, the vast majority of messages have at least one "=pass" (which in my case is any of spf, dkim, dmarc, or dnswl).

> A healthy portion of my spam folder is DMARC-signed, so that does a
> lot of good, in that regard…

Those domains are bad.  Knowing their reputation, their messages could have gone to the spam folder based on authentication rather than content.  Reputation, as standardized by rfc7073, is normally not used.  Even if reputation is a key ingredient, it's not up to SMTP to mention that mail sites should maintain (and share) a database of email identities.

> doesn't need to be involved in this particular discussion.

I agree SMTP shouldn't specify authentication protocols.  (BATV was one of the early ones, it is not mentioned in any rfc*21, and that's good.)  However, authentication in general ought to be mentioned.  In addition, there are traits of rfc5321, such as Section 3.9 "Mailing Lists and Aliases", which deserve being restructured not just to comply with DMARC, but to establish more clearly who is responsible of what.  Mailing lists and email address portability are currently considered SMTP core features.  They are such minor parts of the spec that rfc5321bis could leave them out, delegating them to some ancillary spec, without altering the protocol so much as to have to get back to proposed standard.  Such a reshaped, simplified SMTP really wouldn't need to be involved in these discussions, but we aren't there yet.

Best
Ale
-- 

[*] https://web.archive.org/web/20120603201012/https://www.ftc.gov/bcp/workshops/e-authentication/

---

From: Sam Varshavchik <[email-masked]>
Date: Thu, 26 Dec 2019 17:36:05 -0500

Alessandro Vesely writes:

> Those domains are bad.  Knowing their reputation, their messages could have  
> gone to the spam folder based on authentication rather than content.   
> Reputation, as standardized by rfc7073, is normally not used.  Even if  
> reputation is a key ingredient, it's not up to SMTP to mention that mail  
> sites should maintain (and share) a database of email identities.

Who gets to be in charge of judging domain reputation, anyway? From my  
experience I have no faith in any scheme that involves a third party  
reputation provider.

DNSBLs are the best known third party reputation source providers, and they  
have a long track record. And my take is that their track record is somewhat  
spotty. I use two of them. My current stats are that my custom HELO/EHLO,  
SPF, and CBV checks block more than four times as much garbage as both of  
them combined; and I do believe I check the DNSBLs first, before expending  
additional resources on all other checks.

Some of these DNSBLs have existed for decades. If they were truly effective,  
you'd think there'll be just a trickle of spam left by now.

That folder I mentioned earler – full of DMARC-signed spam – it's not like  
it's just from a limited set of senders. I already blacklisted the  
persistent spam sources manually, since it appears that they tend to be  
concentrated on several hosting providers that ignore spam complaints, and  
the aforementioned DNSBLs appear to have a blind spot for those sewers. That  
DMARC-signed spam comes from all over the place, and attempting to chase  
down each one is pointless, since the sources keep changing, each time.  
Since none of the two popular DNSBL reputation providers have done anything  
about those spam-spewing hosting providers, that I can see, I have very  
little confidence in them tracking the churning domain sources of spam that  
remain after the low-hanging fruit gets filtered out by the existing low- 
cost checks.

Third party reputation service providers have some marginal value, in my  
eyes. They provide some value today. But I'm skeptical that they can be  
a complete solution to spam.

The only technical solution that I think has a chance of eventually getting  
rid of spam is the one that conclusively proves or disproves whether the  
mail sender is known to the /individual addressee/. Spam, by definition,  
comes from an unknown source, and it will not be able to prove that it's a  
known source. That means it can be filtered out, at that point. That's the  
root of the problem: spam, by definition, comes from a completely unknown  
source to the sender. Focus the technical solution on /that/. Having a  
reputation provider vouch for the reputation of the sender does nothing to  
address the fundamental nature of what spam is.

Any reputation-based scheme is doomed to eventually get corrupted. So, you  
have a reputation provider somehow used to assign a reputation to a sender,  
in some form or fashion. Details don't matter. It's only a matter of time  
before: 1) A deep-pocketed organization with a good reputation starts  
spamming, they think this gives them license to spam, 2) at this point  
either the reputation provider does nothing, crashing the whole scheme, or  
revokes the organization's reputation, 3) the deep-pocked organization sues  
the reputation provider for defamation, libel, or some other tort, they'll  
just make something up, 4) hillarity ensues.

Forget about a deep-pocketed organization. If sufficiently motivated, I  
could probably dig up a historical incident, or two, from circa 10-15 years  
ago, where a fly-by-night spambag sued a small, but popular and effective,  
volunteer-run blacklist, causing them to incur non-trivial legal expenses.  
Who's going to fund the reputation provider for something like this, and how?

No, I do not believe that a third party reputation provider is the way to go  
here.

---

From: Richard Clayton <[email-masked]>
Date: Thu, 26 Dec 2019 23:46:18 +0000

In message <[email-masked]>,
Sam Varshavchik <[email-masked]> writes

>known source.

that's your definition ... the people who write the various statutes
have another one ...

... BUT the big mailbox providers (the ones who receive billions of
emails a day and use machine learning for their filtering) have a
completely different definition:

the ML systems need some notion of what is or is not spam and for that
they use recipient behaviour. If they deliver email to the inbox and the
recipient "marks it spam" then the ML systems "learn" that similar email
is spam, If they deliver email to the spam folder and the user
"recovers" it to the inbox then again the ML system learns of its error

the result is that for these large systems spam is "email that users do
not currently want in their inbox today"

since real people mark their utility bills as spam (remarkably often!)
and many dislike the offers that they once signed up to receive from
their favourite airline (whilst others are super keen to receive that
email) there are complex additional systems, adjunct to the ML, to
handle transactional email and to deal with mail about which there is no
consensus as to how spammy it might be ...

> That means it can be filtered out, at that point. That's the  
>source to the sender. Focus the technical solution on /that/. 

if you decide to focus on that then you will deal with a chunk of the
problem but not I think the chunk that anyone really has much problem
with -- and in doing so you will disrupt a great many edge cases where
email from apparent strangers is in fact of considerable value to the
recipient.

>Having a  

I'd agree with the generality expressed here -- but the reputation
providers are of significant use today, particularly to sites that only
see small amounts of email (less than 10m items or so per day) ...

... and since it is clearly unwise to expect a technical fix to address
a social problem (that's my generality for the mix!) the social nature
of their approach is at least heading in the right sort of direction

- -- 
richard                                                   Richard Clayton

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 Dec 2019 10:13:03 +0100

On Thu 26/Dec/2019 23:36:05 +0100 Sam Varshavchik wrote:
> experience I have no faith in any scheme that involves a third party reputation
> provider.
> 
> spotty. I use two of them. My current stats are that my custom HELO/EHLO, SPF,
> and CBV checks block more than four times as much garbage as both of them
> combined; and I do believe I check the DNSBLs first, before expending

DNSWL also exist, and some of them do a fairly good job.  Of course, they
happen to whitelist spammers, or, as well, whitelisted MTAs happen to spew spam.

> That folder I mentioned earler – full of DMARC-signed spam – it's not like it's
> just from a limited set of senders. I already blacklisted the persistent spam
> sources manually, since it appears that they tend to be concentrated on several
> hosting providers that ignore spam complaints, and the aforementioned DNSBLs
> appear to have a blind spot for those sewers. That DMARC-signed spam comes from
> all over the place, and attempting to chase down each one is pointless, since
> the sources keep changing, each time. Since none of the two popular DNSBL
> reputation providers have done anything about those spam-spewing hosting
> providers, that I can see, I have very little confidence in them tracking the
> churning domain sources of spam that remain after the low-hanging fruit gets
> filtered out by the existing low-cost checks.
> 
> Third party reputation service providers have some marginal value, in my eyes.
> They provide some value today. But I'm skeptical that they can be a complete
> solution to spam.
> 
> The only technical solution that I think has a chance of eventually getting rid
> of spam is the one that conclusively proves or disproves whether the mail
> sender is known to the /individual addressee/.

If users send out mail using the domain's MTA —as they should— the MTA can keep
a count of messages sent to and received from each domain.  I don't count
messages to/from abuse or postmaster accounts (perhaps I should also exclude
info).  It is unusual to get spam from a domain that has both counters
positive.  I'd guess that that is nearly as good as individual addressee.  On
the other hand, while different domains have different policies about allowing
users to use varying From: values, receivers only get an SPF and/or DKIM
authentication, which covers the domain-part of the address only.

However, what usually happens is to get mail from new domains.  Even after
years, the number of new domains per day is not decreasing.  It is a mixture of
good and bad domains.  Many of the bad ones are trash domains, especially with
the new TLDs.

> Having a reputation provider vouch for the reputation of the sender does
> nothing to address the fundamental nature of what spam is.

Correct.  However, it can tell if a domain is a stable mail site, with a
responding abuse team, or, say, sulphurlog.xyz (one of yesterday's pile).

> Any reputation-based scheme is doomed to eventually get corrupted. So, you have
> a reputation provider somehow used to assign a reputation to a sender, in some
> form or fashion. Details don't matter. It's only a matter of time before: 1) A
> deep-pocketed organization with a good reputation starts spamming, they think
> this gives them license to spam, 2) at this point either the reputation
> provider does nothing, crashing the whole scheme, or revokes the organization's
> reputation, 3) the deep-pocked organization sues the reputation provider for
> defamation, libel, or some other tort, they'll just make something up, 4)
> hilarity ensues.

Another possibility is to devise some sort of neighboring communities whereby a
mail domain can get the worthiness of the sending domain from a multitude of
third party domains, possibly weighting their advice by the worthiness of each
purveying domain in turn, and averaging the result.  Not so straightforward,
but maybe possible.

Then, it will always happen to have a user's identity stolen or some such.
Like car accidents, one can never stop them for good.  Authentication, if done
well, provides for reporting abuse.  Complaining about someone's behavior is
the proper way to cure the social aspect of spam, to improve sensible behavior
in order to reduce the number of accidents.

Best
Ale
--

---

From: Sam Varshavchik <[email-masked]>
Date: Fri, 27 Dec 2019 07:13:32 -0500

Richard Clayton writes:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>

Yes: it's "that which our donors do not send".

> with

So, today people don't have much of a problem with spam from unknown sources  
to them?

>      -- and in doing so you will disrupt a great many edge cases where

But that's a known source to the recipients: those apparent strangers didn't  
simply have their E-mail address pop into their heads. They got it from  
somewhere; the recipients' web site, or some other public forum. So, the  
source of their E-mail isn't really unknown, it's known where they got their  
E-mail from, in this case.

This is why I said early on: the technical solution here must involve not  
just SMTP.

> .... and since it is clearly unwise to expect a technical fix to address

That's where I will disagree. There may be some social aspects to spam, but  
it is mostly a technical problem. The technical problem is the open nature  
of SMTP. That will never change, and this technical problem is only solvable  
mostly by other technical means, that are rooted in something other than  
SMTP.

In any case, there's nothing that can anyone show that any social or legal  
approach to fighting spam will make any difference whatsoever. Again: look  
how effective the CAN SPAM Act of 2003 turned out to be. Any proposal, along  
the same lines, will need to explain why it will succeed, and the CAN SPAM  
Act didn't.

---

From: Keith Moore <[email-masked]>
Date: Fri, 27 Dec 2019 11:18:11 -0500

On 12/27/19 4:13 AM, Alessandro Vesely wrote:

> If users send out mail using the domain's MTA —as they should

While this would be convenient, I'm not sure it's justified. Sure, it's 
easier to do that with modern UAs than it used to be, but I've seen 
several UAs that preferred one submission server over others.   I've 
also seen environments that tried to force outgoing mail to a specific 
submission server.

This might be an item for a "best email practices" RFC.   But email 
usage is incredibly diverse.   I suspect we need an alternative way for 
users to authenticate their mail than to expect them to submit using 
their domain's submission server.

Keith

---

From: John C Klensin <[email-masked]>
Date: Fri, 27 Dec 2019 12:35:22 -0500

--On Friday, December 27, 2019 11:18 -0500 Keith Moore
<[email-masked]> wrote:

> While this would be convenient, I'm not sure it's justified.
> Sure, it's easier to do that with modern UAs than it used to
> be, but I've seen several UAs that preferred one submission
> server over others.   I've also seen environments that tried
> to force outgoing mail to a specific submission server.
> 
> This might be an item for a "best email practices" RFC.  
> But email usage is incredibly diverse.   I suspect we need
> an alternative way for users to authenticate their mail than
> to expect them to submit using their domain's submission
> server.

FWIW, and speaking just as an individual rather than as editor,
I completely agree with Keith.  In addition, as long as many
domains do not support email services themselves but instead
either outsource the services and encourage their users to find
a large public (and free) mail provider, and many of those
providers make no attempt to authenticate those who sign up for
their services, it is not clear what "using the domain's MTA"
actually means.

   john

---

From: S Moonesamy <[email-masked]>
Date: Fri, 27 Dec 2019 09:41:50 -0800

Hi Sam,
>approach to fighting spam will make any difference whatsoever. Again: look

There is a (IRTF) RFC which has the following text: "rise of spam and 
other anti-social behavior ..."  I did a search of the IETF mailing 
list archives for other discussions of the topic.  I could not find 
anything significant using the IETF search engine.  I couldn't use a 
well-known search engine for the search as, some time back, the IETF 
blocked the search engine from indexing its mailing list.  Anyway, 
from what I remember of IETF-related discussions, spam used to be 
describe as a social problem.

A technical mechanism used by the IETF to curtail spam spurred this 
thread.  I gather that there is agreement that technical approaches 
are being used to curtail spam.  However, the merits of the different 
approaches are debatable.

I noticed the following in an email: "Authentication-Results: 
[removed]; dkim=pass (1024-bit key) header.d=tpcoms.vn 
header.i=@tpcoms.vn".  There is a PTR RR for the IP address from 
which the message was sent.  The author of the message has the same 
name as a government official in .cz.  My guess is that it was a 
phishing attempt.  It would be very difficult to prove that without 
additional information, and that may require taking a legal approach.

There are countries which have enacted laws [1] related to email.  A 
few weeks ago, I had a conversation with a group of legislators who 
were interested in that topic.  It is useful to have such laws 
instead of leaving email as an undefined space [2].

Regards,
S. Moonesamy

1. http://r.elandsys.com/r/67728
2. That does not mean that those laws will stop unwanted emails from 
being sent.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 Dec 2019 19:05:10 +0100

On Fri 27/Dec/2019 18:35:22 +0100 John C Klensin wrote:
> --On Friday, December 27, 2019 11:18 -0500 Keith Moore wrote:

I'd be tempted to define a domain's MTA after authentication methods, e.g.
having the mailout IP address listed in the domain's SPF record, or using the
domain's private DKIM key to sign outgoing messages.  However, the line quoted
above implied just letting the domain admins know at least the metadata of what
users send using the domain's name.

Users of free-mail providers may have privacy objections.

Best
Ale
--

---

From: Keith Moore <[email-masked]>
Date: Fri, 27 Dec 2019 13:14:25 -0500

> describe as a social problem.

I don't even know what people mean when they claim that spam is a 
"technical problem".   It sounds like a ridiculous statement to me.

Spam is absolutely anti-social behavior, because it's a small number of 
people stealing the attention of a huge number of people for completely 
selfish reasons.

But people used to also say things like "you can't address a social 
problem by technical means" which I also thought was a ridiculous 
statement.   Social problems are addressed by technical means, if 
imperfectly, all the time.

Keith

---

From: Sam Varshavchik <[email-masked]>
Date: Fri, 27 Dec 2019 18:49:54 -0500

S Moonesamy writes:

> Hi Sam,
>
> There is a (IRTF) RFC which has the following text: "rise of spam and other  
> anti-social behavior ..."  I did a search of the IETF mailing list archives  
> for other discussions of the topic.  I could not find anything significant  
> using the IETF search engine.  I couldn't use a well-known search engine for  
> the search as, some time back, the IETF blocked the search engine from  
> indexing its mailing list.  Anyway, from what I remember of IETF-related  
> discussions, spam used to be describe as a social problem.

It is true that spam is anti-social behavior. But…

> A technical mechanism used by the IETF to curtail spam spurred this thread.   
> I gather that there is agreement that technical approaches are being used to  
> curtail spam.  However, the merits of the different approaches are debatable.

… notice that all these "different approaches" are technical in nature.  
Socially wagging one's finger at a spammer – "don't you dare do it  
again" – has not worked very well. So, we've all been trying various  
technical solutions. Blacklists. CBVs. Various kinds of HELO checks. DMARC,  
SPF and various other kinds of authentication schemes.

These were all various technical attempts to solve them problem. They  
offered some relief, but noone can claim that they effectively addressed the  
problem. But they were all technical attempts, so I see spam as a technical  
problem to solve.

But whether spam is a technical or a social problem I don't really see that,  
in itself, as being very important in the grand scheme of things. My only  
point is that any headway on spam will involve a technical fix. Not a social  
fix. Or a legal fix.

> I noticed the following in an email: "Authentication-Results: [removed];  
> dkim=pass (1024-bit key) header.d=tpcoms.vn header.i=@tpcoms.vn".  There is  
> a PTR RR for the IP address from which the message was sent.  The author of  
> the message has the same name as a government official in .cz.  My guess is  
> that it was a phishing attempt.  It would be very difficult to prove that  
> without additional information, and that may require taking a legal approach.

Well, lots of luck to anyone planning to pursue a legal approach in … 
Vietnam, on the account of your email.

> There are countries which have enacted laws [1] related to email.  A few

And what tangible impact have these laws had, anywhere?

> weeks ago, I had a conversation with a group of legislators who were  
> interested in that topic.  It is useful to have such laws instead of leaving  
> email as an undefined space [2].

Useful in which way, exactly? All of these laws, that have either been  
enacted or proposed, they're all typically the same in spirit, and only  
differ in secondary characteristics.

Again, someone please show me some objective data that the CAN SPAM act of  
2003 has produced, in terms of reduction of spam, and then I'll consider  
giving any weight to any legal or legislative course of action.

---

From: Keith Moore <[email-masked]>
Date: Fri, 27 Dec 2019 21:57:22 -0500

On 12/27/19 6:49 PM, Sam Varshavchik wrote:

> … notice that all these "different approaches" are technical in 
> nature. Socially wagging one's finger at a spammer – "don't you dare 
> do it again" – has not worked very well. So, we've all been trying 
> various technical solutions. Blacklists. CBVs. Various kinds of HELO 
> checks. DMARC, SPF and various other kinds of authentication schemes.
>
> offered some relief, but noone can claim that they effectively 
> addressed the problem. But they were all technical attempts, so I see 
> spam as a technical problem to solve.
>
> But whether spam is a technical or a social problem I don't really see 
> that, in itself, as being very important in the grand scheme of 
> things. My only point is that any headway on spam will involve a 
> technical fix. Not a social fix. Or a legal fix. 

I don't see any of these fixes being sufficient by itself.  But I think 
it's possible that a combination of legal and technical fixes might 
improve the current situation.

In some countries there are laws against spam, and laws requiring 
advertisers to allow opt-out and make their physical addresses known.   
As far as I can tell, at least the laws on advertising in the US have 
had a useful effect.   I rarely get spam from these "legitimate" 
businesses more than once (because I always unsubscribe, and few ignore 
the unsubscribe requests).   Once is still too many IMO but that still 
reduces the volume of advertising that I'd receive without the laws 
being in place.

The problems with legal solutions to "non-legitimate" spam (i.e. spam 
from parties who don't reveal their identity and might not even be 
operating any kind of legitimate business) seem to be reliably 
identifying those who send such spam, proving that they've sent spam, 
and holding offenders from distant countries accountable.   But it seems 
possible that additional technical mechanisms (or slight tweaks to 
existing ones) could improve the ability of the legal system to do its job.

I wonder whether there's anything that we can learn from the SHAKEN/STIR 
example, which now has some legal support at least in the US.

Keith

---

From: Sam Varshavchik <[email-masked]>
Date: Fri, 27 Dec 2019 22:57:50 -0500

> example, which now has some legal support at least in the US.

This piqued my curiosity, so I briefly Googled up on its history.

The FCC regulates telephone companies, and has some authority to impose  
rules on them, and the SHAKEN/STIR framework was, slowly, pushed onto the  
industry within this existing regulatory framework. The following page  
provides a nice timeline: https://www.fcc.gov/call-authentication

And this is where the analogy breaks down. Anything that's listening on port  
25 will accept mail from anywhere else in the world. There's no existing  
regulatory authority of any kind here. The pitifully little authority that  
exists is over turnkey Internet providers. Government-imposed controls over  
communications are found only in certain authoritarian political regions,  
and only that kind of a control could be in a theoretical position of  
imposing something similar for SMTP. And, after some thinking, I still think  
they'd have a problem handling SMTP entering the aforementioned political  
region.

P.S. I started getting robo-calls in Mandarin, for the very first time,  
after my cell phone carrier activated its SHAKEN/STIR implementation. I  
presume that the shown caller ID was valid. A lot of value that is to me. I  
definitely agree: SHAKEN/STIR is an excellent analogy to various SMTP  
validation schemes, like SPF and DMARC, and from the looks of it, is just as  
effective – has some marginal benefit, but somewhat misses the mark. The  
problem was always not the sender's identity, but its unsolicited/unknown  
nature. I will maintain that the focus on verifying the sender's John  
Hancock is misplaced; until the unsolicited nature, itself, of either spam  
or robo-calls, gets /directly/ targeted, the respective problem will remain  
unsolved.

So, how do I think it should be targeted. Well, I don't think it needs to  
involve SMTP. DMARC is implemented in mail headers, and SMTP's involvement,  
if any, is limited to accepting or rejecting mail that fails DMARC  
validation. I think this gets handled similarly – in terms of header-based  
metadata proving/disproving that the message's source is known[1] to the  
recipient or it is completely unknown/unsolicited: spam. For that reason, I  
don't think this is SMTP's problem to solve.

[1] Note that "message's source is known" includes things like sending mail  
to a publicly listed contact, so the objections about the usefulness of  
being able to receive mail from new contacts is covered. The "message's  
source is known" would simply include things like "public contact info given  
at such and such place".

---

From: S Moonesamy <[email-masked]>
Date: Sat, 28 Dec 2019 00:30:28 -0800

Hi Sam,
>And what tangible impact have these laws had, anywhere?

It depends on whether the business takes a breach of a law, e.g. data 
protection as a serious matter.  I was surprised to hear that a 
regulatory agency based some of its investigations on tax evasion on 
email advertising.  As for whether those laws have a tangible impact, 
it would better to ask people with expertise in that field.

>differ in secondary characteristics.

It is useful to adhere to applicable laws.  The alternative is to 
operate in an unpredictable regulatory environment with the risks 
which that entails.

Regards,
S. Moonesamy

---

From: Sam Varshavchik <[email-masked]>
Date: Sat, 28 Dec 2019 08:39:06 -0500

S Moonesamy writes:

> protection as a serious matter.  I was surprised to hear that a regulatory  
> agency based some of its investigations on tax evasion on email  
> advertising.  As for whether those laws have a tangible impact, it would  
> better to ask people with expertise in that field.

Well, I don't really know who those people are, except for a couple of names  
off the top of my head, that are widely known; but I have no recollection of  
any one of them opining on this specific topic, beyond lukewarm endorsements  
on the order of "well, it doesn't really do any harm". And I don't even know  
how one goes about being accredited for having experise in this field.

All I know is, that I've participated in some well-known anti-spam  
communities since the mid 1990s, and still do in what's left of them; for a  
brief period of time I ran my own DNSBL, before abandoning that project; I  
wrote and still passively maintain a free SMTP server, with flexible mail  
filtering capabilities. And what I said here would be pretty much what I  
would say if someone were to ask me.

> It is useful to adhere to applicable laws.  The alternative is to operate in  
> an unpredictable regulatory environment with the risks which that entails.

There's a difference between being useful to adhere to applicable laws, and  
the applicable laws actually being useful.