Subject: Re: [ietf-smtp] [Emailcore] Proposed ESMTP keyword RCPTLIMIT

From: George Schlossnagle <[email-masked]>
Date: Sun, 18 Apr 2021 19:47:15 -0400

As a commercial MTA and ESP provider, this proposal is very exciting to
me.  Today we guess at these things and largely guess at things based off
of either observed behavior or out-of-band discussion with those operating
the receiver side of the email exchange.  Having this defined in the
session would be great.

George

On Mon, 15 Mar 2021 at 19:47, John Levine <[email-masked]> wrote:

> It appears that Sam Varshavchik  <[email-masked]> said:
> >MXes which do not announce their limits are going to dominate for some
> time.
> >So, I still need to do this for the foreseeable time. So, what does this
> >SMTP extension buy me? The only thing I can think of: not duplicating
> >messages when I don't need to.
> >
> >But the thing is: I don't recall anyone raising this to me, as a
> problem.
> >This is not on my radar screen as a problem that wants to be solved.
>
> I wouldn't think that your users would be very interested, but bulk
> mailers are desperate for
> any hints about how much mail they can stuff at people without getting
> blocked or rate limited.
>
> R's,
> John
>
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

-- 
*GEORGE SCHLOSSNAGLE*
chief evangelist
www.sparkpost.com
(el) [email-masked]

---

From: Mike Hillyer <[email-masked]>
Date: Mon, 19 Apr 2021 12:36:37 +0000

As someone who works with a lot of commercial mailers (and commercials MTAs), I have to agree. The whole commercial sending space is full of admins looking to do the right thing, they just want to know what the right thing is. Tell them what to do at the start of the session and they will follow it.

Mike

From: ietf-smtp <[email-masked]> On Behalf Of George Schlossnagle
Sent: Sunday, April 18, 2021 7:47 PM
To: John Levine <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [ietf-smtp] [Emailcore] Proposed ESMTP keyword RCPTLIMIT

As a commercial MTA and ESP provider, this proposal is very exciting to me.  Today we guess at these things and largely guess at things based off of either observed behavior or out-of-band discussion with those operating the receiver side of the email exchange.  Having this defined in the session would be great.

George

On Mon, 15 Mar 2021 at 19:47, John Levine <[email-masked]<mailto:[email-masked]>> wrote:
It appears that Sam Varshavchik  <[email-masked]<mailto:[email-masked]>> said:
>MXes which do not announce their limits are going to dominate for some time.
>But the thing is: I don't recall anyone raising this to me, as a problem.

I wouldn't think that your users would be very interested, but bulk mailers are desperate for
any hints about how much mail they can stuff at people without getting blocked or rate limited.

R's,
John

_______________________________________________
ietf-smtp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ietf-smtp

--
GEORGE SCHLOSSNAGLE
chief evangelist
www.sparkpost.com<http://www.sparkpost.com>
(el) [email-masked]<mailto:[email-masked]>

---

From: Ned Freed <[email-masked]>
Date: Mon, 19 Apr 2021 14:18:17 -0700 (PDT)

> the receiver side of the email exchange.  Having this defined in the

Thanks for the support. 

A question for you: Are there any additional limits you think are sufficiently
important, and perhaps more to the point, in some sense foundational, that they
warrant being in the base specification?

I'm especially interested in people's thoughts on rate limits. The problem I
have with rate limits is, well, how to express them. For example, a rate limit
of 10 transactions a minute is not the same as 600 transactions an hour: In the
former case it's unlikely that a sender will be allowed to bang out 600 messages
in a minute followed by 59 minutes of silence, whereas the latter allows that.

This suggests using a vulgar fraction whatever/second as a means of expressing
rate limits.

Then there's the question of what the limit applies to: Is it by IP, by
(possibly DKIM authenticated) sending domain, by a combination of both, or 
something else?

And all this glosses over the fact that there are any number of ways for
servers to determine a rate.

My personal preference would be to defer this to a subsequent specification -
one which I'm happy to coauthor and even edit, but one where I'd be a lot more
comfortable not being the sole author. But there's an argument to be made that
having a limit that's not a simple integer in the base specification would be a
good thing, in order to prevent people from making assumptions in their
implementations that turn out to be false.

A middle ground would be to put one simple rate limit, say MAILMAXRATEPERIP,
that suffices to establish minimal conventions for such things.

Thoughts?

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 19 Apr 2021 17:58:35 -0400

On Mon, 19 Apr 2021, Ned Freed wrote:
> have with rate limits is, well, how to express them. ...

A little surveying might be in order.

I see that Postfix lets you set both the numerator and denominator of the 
rate limit, but the denominator defaults to a minute.  It has separate 
rate limits for connections and messages.

Exim also lets you set both, the examples use an hour.  The documentation 
says that if you use a large interval that lets people send bursts.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 19 Apr 2021 15:01:41 -0700

> in a minute followed by 59 minutes of silence, whereas the latter allows that.

The question is, in many cases the rate limits will change during the 
course of the SMTP connection..

And we may not have enough information to declare a rate limit, and 
declaring a rate limit that isn't accurate means it isn't smart to 
advertise a limit that is not guaranteed.

That rate limit may change on many variables that occur during the SMTP 
connection..

And because of this, many MTA's will not be able to advertise one 
correctly, I know that in our MTA's that will be the case, and of course 
we expect that any sender properly handles any form of instruction to 
stop sending, whether any pre-advertised limit is in place or not.

Given that, (MTA's ability to respond to a 'stop sending' (4xx or 5xx)) 
we will continue using dynamic rate limits, rather than preset limits, 
instead of implementing two mechanisms.

I am sure others out there with advanced SMTP architectures are facing 
the same concerns.

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 19 Apr 2021 23:29:29 +0100

On 19/04/2021 22:18, Ned Freed wrote:
> Are there any additional limits you think are sufficiently

Just came to mind: a limit on concurrent connections. By server IP seems
most obvious, but perhaps by recipient domain might be wanted in some
cases?

> I'm especially interested in people's thoughts on rate limits.

Much harder to define and handle than the limits so far defined.
I'd prefer to leave any such for a later spec.
-- 
Cheers,
   Jeremy

---

From: George Schlossnagle <[email-masked]>
Date: Mon, 19 Apr 2021 19:32:55 -0400

The common limits we see in the real world (in order of most common
occurrence/impact) are:

Messages per connection
Recipients per message
Simultaneous connections per sending IP (this would be my number one
suggested add)
Message throughput rate (messages/time period where time period is
typically per second, per minute or per hour)
Connection establishment rate (connections/time period where time period is
typically per second, per minute or per hour)
Simultaneous connections per sending IP block (seems particularly hard to
handle here)
Recipients per connection (aggregated across all messages in that
connection)
Bytes sent per time period.

While I agree with Michael that these things can change during the session,
in my experience that's uncommon and that a method of signaling that works
in a majority of cases but still recognizes that providers can 'change the
rules' on the fly in the ways they do now (sending often provider-specific
SMTP response codes) is still a much better place to be.

Thanks again for taking this up.

George

---

From: Ned Freed <[email-masked]>
Date: Mon, 19 Apr 2021 16:33:40 -0700 (PDT)

> says that if you use a large interval that lets people send bursts.

Sounds familiar. Oracle Messaging Server lets you set both; the denominator
default is one hour. 

We store the limits in memcache or Redis so they can apply across mutiple MTAs.
The usual limit is per-IP, but we support per-host and per-domain as well.

The demoninator in PowerMTA is a string rather than a number, possible
values are sec, s, min, m, hr, or h. It supports per-IP and per-host and
probably other stuff; my familiarity here is limited.

I assume the Exim and Postfix limits are per-host, which in most cases
translates to per-IP.

I have to say there's more consistency here than I expected.

				Ned

---

From: Sam Varshavchik <[email-masked]>
Date: Mon, 19 Apr 2021 20:09:23 -0400

> Simultaneous connections per sending IP (this would be my number one  

This is surprising: that messages per connection is more commonly checked  
than everything else.

I would expect that a receiving server would prefer reusing the same  
connection, to send consecutive messages, than have the sender establish a  
connection, send one message, then tear it down.

A long time ago that was Qmail's well-known bad rep: its simplistic  
implementation, how it created a connection for every individual message.  
So, a dozen messages to the same domain resulted in dozens of concurrent  
connections, all to send one message and disconnect.

I see nothing to gain from forcing a sending server to artificially limit  
itself; how after every N sent messages it has to close its socket, and  
reconnect again. What is that supposed to accomplish? I don't get it.

---

From: Laura Atkins <[email-masked]>
Date: Tue, 20 Apr 2021 10:17:09 +0100

> On 20 Apr 2021, at 01:09, Sam Varshavchik <[email-masked]> wrote:
> 
>> The common limits we see in the real world (in order of most common occurrence/impact) are:
>> 
>> 
>> Messages per connection
>> Recipients per message
>> Simultaneous connections per sending IP (this would be my number one suggested add)
> 
> This is surprising: that messages per connection is more commonly checked than everything else.
> 
> I would expect that a receiving server would prefer reusing the same connection, to send consecutive messages, than have the sender establish a connection, send one message, then tear it down.
> 
> A long time ago that was Qmail's well-known bad rep: its simplistic implementation, how it created a connection for every individual message. So, a dozen messages to the same domain resulted in dozens of concurrent connections, all to send one message and disconnect.
> 
> I see nothing to gain from forcing a sending server to artificially limit itself; how after every N sent messages it has to close its socket, and reconnect again. What is that supposed to accomplish? I don't get it.

It’s a limit that was implemented more than a decade ago by some of the highest volume receivers around. I don’t think we have to understand why they did it - if even the folks who made the decision are still around. [1] We can just say: this is commonly occurring behavior and it makes sense for SMTP to document it and have a way to explain it. 

I don’t think a discussion about why or if it’s sensible will make any difference. The companies who are closing the connections after X number of messages (5, 10 and 20 are all used) are not likely to either defend their technical decisions to the IETF or change their behavior after a decade and at least one whole infrastructure rearchitecture. 

What we should be doing is figuring out how to make these limits more clear to senders. 

laura

[1] In at least one case the decision maker passed away a few years ago. 

-- 
Having an Email Crisis?  We can help! 800 823-9674 

Laura Atkins
Word to the Wise
[email-masked]
(650) 437-0741		

Email Delivery Blog: https://wordtothewise.com/blog

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 20 Apr 2021 11:39:25 +0200

On Tuesday 20 April 2021 11:17:09 CEST, Laura Atkins wrote:
> I don’t think a discussion about why or if it’s sensible will 
> make any difference.

Agree. But I'll note that sensible technical reasons do exist. If a system 
is written with short-lived connections in mind (which can be a good idea 
or a bad one), supporting long-lived connections may be more trouble than 
it's worth, in several ways. Both in terms of time and in terms of actions 
done via that connection.

Arnt

---

From: Mike Hillyer <[email-masked]>
Date: Tue, 20 Apr 2021 12:24:00 +0000

Generally speaking when I advise senders on their rate limits, I have them use the smallest denominator possible, because most commercial MTAs will take a per-hour rate limit and implement it as "send the hour's allotment in a single burst and then wait 59 minutes", which I don't think most receivers want, so I advise setting a per second limit when possible. One message per second is better than 3600 messages per hour if it means a 3600 message burst and then a long wait.

Mike

-----Original Message-----
From: ietf-smtp <[email-masked]> On Behalf Of Ned Freed
Sent: Monday, April 19, 2021 7:34 PM
To: John R Levine <[email-masked]>
Cc: Ned Freed <[email-masked]>; [email-masked]
Subject: Re: [ietf-smtp] [Emailcore] Proposed ESMTP keyword RCPTLIMIT

> > I'm especially interested in people's thoughts on rate limits. The 
> > problem I have with rate limits is, well, how to express them. ...

> I see that Postfix lets you set both the numerator and denominator of 
> the rate limit, but the denominator defaults to a minute.  It has 
> separate rate limits for connections and messages.

> Exim also lets you set both, the examples use an hour.  The 
> documentation says that if you use a large interval that lets people send bursts.

Sounds familiar. Oracle Messaging Server lets you set both; the denominator default is one hour. 

The demoninator in PowerMTA is a string rather than a number, possible values are sec, s, min, m, hr, or h. It supports per-IP and per-host and probably other stuff; my familiarity here is limited.

I assume the Exim and Postfix limits are per-host, which in most cases translates to per-IP.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 20 Apr 2021 06:42:21 -0700

On 4/20/2021 2:17 AM, Laura Atkins wrote:
> I don’t think a discussion about why or if it’s sensible will make any 
> difference. The companies who are closing the connections after X number 
> of messages (5, 10 and 20 are all used) are not likely to either defend 
> their technical decisions to the IETF or change their behavior after a 
> decade and at least one whole infrastructure rearchitecture.
> 

+1

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Richard Clayton <[email-masked]>
Date: Tue, 20 Apr 2021 15:19:15 +0100

In message <[email-masked]>,
Laura Atkins <[email-masked]> writes

>It’s a limit that was implemented more than a decade ago by some of the highest 
>volume receivers around. I don’t think we have to understand why they did it - 
>if even the folks who made the decision are still around. [1] We can just say: 
>this is commonly occurring behavior and it makes sense for SMTP to document it 
>and have a way to explain it. 

As you will know, the highest volume receivers have complex machine
learning systems which can -- after a number of messages -- come to the
conclusion that sending a SYN-ACK at the start was somewhat of a
mistake... use of resources is more efficient if you make such senders
(which dominate) start at the beginning again and you just refuse to let
them connect "ever again".

>From an engineering standpoint the overhead of 5xx-ing a small number of
emails per session once the sender is understood to be bad may be a
reasonable tradeoff compared with the complexity of chasing round all of
your servers (of which you have rather more than one or two) identifying
which open connections should be aborted.

Other (related) considerations apply to email submitted over
authenticated connections (you may realise as the email piles up that it
is likely that the credentials are being misused)

>What we should be doing is figuring out how to make these limits more clear to 

as I observed before -- don't expect much more than a generic statement
(which comes down to "sending N good emails is the most you can expect
to be able to manage") from such systems. Anything which revealed the
current view ("actually, you've pretty much exhausted our patience")
just is not going to happen (IMO)

-- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

From: Laura Atkins <[email-masked]>
Date: Tue, 20 Apr 2021 15:33:22 +0100

> On 20 Apr 2021, at 15:19, Richard Clayton <[email-masked]> wrote:
> 

Absolutely this is true. 

But many of these systems also have rules that apply universally and have nothing to do with any reputation or performance issues. I was keeping track of them and making them publicly available many years ago. The commercial MTAs mostly handle this in an automated fashion and these limits aren’t published the same way they were in the past (on postmaster pages). 

To give an example of limits I was sharing: 

Provider 1: 		25 connections per IP address	100 recipients per message		
Provider 2: 		5 connections per IP address		100 messages per connection
Provider 3: 		1 connection per IP				1 email per connection
Provider 4:									500 emails per connection; 100 recipients per message
Provider 5: 									20 emails per connection; reputation based rate limiting

Providers 3 and 5 still have those limits in place. The others I’m not as sure about. 

This data is publicly available, I’m just hiding the providers because I really don’t think the details of who is doing what is relevant to the discussion. It’s enough to know that major providers are doing this and have been doing this for at least a decade.

Providing a programatic way to advertise the maximum limits seems to me to be a good addition to the information transmitted. Less reliance on individual people manually maintaining lists of information is never a bad thing.  

[… snip ...]

> just is not going to happen (IMO)

That’s the statement I am looking for here. We accept no more than this, even for the servers with the absolute best reputation. With the understanding that those limits may be lower (sometimes significantly lower) for senders with poorer reputation. 

laura 

Laura Atkins
Word to the Wise
[email-masked]
(650) 437-0741

---

From: Ned Freed <[email-masked]>
Date: Tue, 20 Apr 2021 07:28:03 -0700 (PDT)

> them connect "ever again".

True, but it's also the case that such conclusions can be reached
by an all-too-human staff; no fancy ML required.

> > which open connections should be aborted.

Especially since you can do it at the MAIL FROM, and if the sender proceeds to
send the message anyway or otherwise misbehave rely on the server's built-in
logic to deal with that.

(We actually support both approaches.)

> just is not going to happen (IMO)

I don't really understand the concern here, and more importantly, how to address
it. Any time you connect to someone else's SMTP server there's a implicit
understanding that it will only accept what you send as long as you send what it
considers to be good stuff. The LIMITS extension operates within this context;
all it does is let the server add to the client's understand of what's
acceptable; in no way does it change that basic understanding.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 20 Apr 2021 07:44:32 -0700 (PDT)

Interesting and good to know. Thanks.

				Ned

> Generally speaking when I advise senders on their rate limits, I have them use
> the smallest denominator possible, because most commercial MTAs will take a
> per-hour rate limit and implement it as "send the hour's allotment in a single
> burst and then wait 59 minutes", which I don't think most receivers want, so I
> advise setting a per second limit when possible. One message per second is
> better than 3600 messages per hour if it means a 3600 message burst and then a
> long wait.

---

From: "John R Levine" <[email-masked]>
Date: 20 Apr 2021 11:01:03 -0400

It looks like something like this would handle a lot, quite possibly most, 
of the rate limits that receivers are willing to disclose:

   CONNRATE=m/s MSGRATE=m/s

Where m is a number of connections or messages or whatever, and s is a 
number of seconds.

I suppose it could also have something like SCOPE=HOST or SCOPE=GROUP to 
say whether it's for a single host or a group but I don't see any way of 
describing groups that wouldn't be too complicated and I doubt that many 
mail providers would want to tell hostile strangers that much about their 
internal architecture.

R's,
John

Regards,

---

From: Richard Clayton <[email-masked]>
Date: Tue, 20 Apr 2021 16:53:45 +0100

In message <[email-masked]>, Ned Freed
<[email-masked]> writes

>I don't really understand the concern here, 

Some bad senders start by sending good email and then, once their
reputation is established, send bad email. It is generally thought to be
unwise to give any clues whatsoever as to how well this process might be
going... (to what extent the machine learning system has been fooled)

>and more importantly, how to address
>it. 

It is inherent that you cannot -- so mailbox providers who operate in
the space where this is a significant issue are not going to provide any
fine-grained data whatsoever -- just generic values.

I take Laura's point that good senders can deduce the upper limits --
and hence providing data about those upper limits at the start of the
session has a small amount of value (if only in ensuring that sessions
close smoothly and fewer exception logging event are generated)

-- 
richard                                                   Richard Clayton

---

From: Ned Freed <[email-masked]>
Date: Tue, 20 Apr 2021 10:31:02 -0700 (PDT)

> fine-grained data whatsoever -- just generic values.

That's not what I meant by "address". It's important that standards
discuss all known security issues; and this definitely qualifies.
As such, I'm going to add this text to the Security Considerations:

  Use of the Limits extension to provide client-specific information - as 
  opposed to general server limits - unavoidably provides senders with
  feedback about their reputation. Malicious senders can exploit this
  in various ways, e.g., start by sending good email and then, once their
  reputation is established, sending bad email.

> close smoothly and fewer exception logging event are generated)

Speaking as someone who works on software used for bulk sending, the the value
isn't small, especially when you take the ability to shape subsequent traffic
into account.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Tue, 20 Apr 2021 21:09:00 +0100

> I assume the Exim and Postfix limits are per-host, which in most cases

Exim can rate-limit per almost anything, because the lookup key for the
measured rate is an arbitrary string expansion. (typically it will be
the client's IP address or authenticated username)

It also has options for counting the rate of connections, or messages, or
recipients.

Some of this flexibility was because I didn't know what policies would
work in practise, and it did take a fair amount of experimentation to get
something that was not too disruptive. (I cared about catching outbound
spam, typically from compromised accounts.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  https://dotat.at/
Tyne, Dogger: Variable 3 or less, becoming north, 4 or 5, occasionally
6 later in Dogger. Smooth or slight, becoming slight or moderate
later. Occasional rain at first. Good, occasionally moderate at first.

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 20 Apr 2021 16:59:19 -0400

> On Apr 19, 2021, at 7:33 PM, Ned Freed <[email-masked]> wrote:
> 

The Postfix limits are per-service (essentially host:port), and if we're
just concerned about port 25, then yes, basically per host, but it is
possible on a multi-homed host to treat all the IPs as a single service,
or as separate services.  The distinction is not particularly visible
from outside, so the safe bet is that the limits host-wide.  Things are
perhaps complicated if the same host has multiple names...

The potential load limits that can be enabled are:

  * CONNECTION COUNT
  * CONNECTION RATE
  * MESSAGE RATE
  * RECIPIENT RATE
  * TLS SESSION NEGOTIATION RATE  (TLS resumptions are free)
  * SASL AUTH RATE

Relevant settings:

       anvil_rate_time_unit (60s)
              The time unit over which client connection rates and other rates
              are calculated.

       smtpd_client_connection_count_limit (50)
              How many simultaneous connections any client is allowed to make
              to this service.

       smtpd_client_connection_rate_limit (0)
              The maximal number of connection attempts any client is allowed
              to make to this service per time unit.

       smtpd_client_message_rate_limit (0)
              The maximal number of message delivery requests that any client
              is allowed to make to this service per time unit, regardless of
              whether or not Postfix actually accepts those messages.

       smtpd_client_recipient_rate_limit (0)
              The maximal number of recipient addresses that any client is
              allowed to send to this service per time unit, regardless of
              whether or not Postfix actually accepts those recipients.

       smtpd_client_new_tls_session_rate_limit (0)
              The maximal number of new (i.e., uncached) TLS sessions that a
              remote SMTP client is allowed to negotiate with this service per
              time unit.

       smtpd_client_auth_rate_limit (0)
              The maximal number of AUTH commands that any client is allowed
              to send to this service per time unit, regardless of whether or
              not Postfix actually accepts those commands.

-- 
	Viktor.