Subject: Re: Abort data transfer?
From: Alessandro Vesely <[email-masked]>
To: John R Levine <[email-masked]>
Cc: IETF SMTP list <[email-masked]>
Date: Fri, 23 Oct 2009 12:53:44 +0200
List-Id: <ietf-smtp.imc.org>

John R Levine wrote:
> [...]  The infamous Ron Guilmette had an amazing tarpit 
> MTA about a decade ago that used that technique to keep 4000 
> simultaneous connections open, running on a 486.  But since the rise of 
> botnets, tarpitting lost whatever effectiveness it might have had, so I 
> don't know of anyone still doing it.

Courier-MTA does it, for one. I cannot think of another method for 
handling dictionary attacks, targeting either user/passwd or rcpt. 
It's obviously not enough against a botnet, but I think it's a good 
practice. In such scenario, it is normal to have some SMTP daemons 
loitering about with their tarpits; IME they usually arrive to a 
regular disconnection. However, in case the maximum number of daemons 
is reached, one should force some of those connections to abort. I 
think that can happen by chance, not necessarily implying a deliberate 
DoS attack.

Forcefully aborting connections, as well as disallowing further access 
from those IPs, is better dealt with at the system's firewall level: 
If knowledge about bad behavior is encapsulated within the SMTP 
software, different daemons have to duplicate both the functionality 
and the data, resulting in an overall weaker system. BTW, I reckon 
4000 IPs can be handled with a ~400K .db file keyed on the IP number, 
and would need ~400G to map the whole IPv4 address space that way.

Date: Thu, 22 Oct 2009 14:44:38 -0700 (PDT)
From: [email-masked]
Subject: Re: Abort data transfer?
To: Paul Smith <[email-masked]>
Cc: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Sender: [email-masked]

> David MacQuigg wrote:
> >
> > Seems to me RFC-5321 could have said:  The client MUST pause every
> > 1000KB (or 100 seconds, whichever is less) and look for a REJECT.
> > Clients who fail to do this risk being blocked by a firewall.

> What would be the point of this? A 'bad' client won't do this, and even
> if you do send a reject and it does honour it, it will just start
> sending data again, so - what's the point. Just block them at the
> firewall anyway - not that that would help against a botnet.

I have to agree.

> (A much better 'solution' in RFC 5321 would have been to enforce the
> SIZE extension, and allow the server to drop the connection if the
> client sends more than 10% over the advertised size of the email - it
> still wouldn't help in the bigger picture, but it might close this
> unexploited loophole)

Mandate an extension in a revision like this? Absent a "do this or the email
service is gone" sort of reason, There was simply no way that was ever going to
happen.

> As I see it, you could somehow work out a protocol for telling something
> that is sending you 5000 TB of data to 'stop it please', but (a) it
> could ignore you or (b) it could stop, but then start again straight
> away, so it sends you 10MB, then another 10MB, then another 10MB ad
> disk-blow-upeum.

Exactly.

> This can happen with pretty much any protocol anywhere.

That's the point I've been trying to make all along.

> It has to be handled by heuristics of some sort, or not handled at all -
> in my experience serious DoS attacks are relatively rare, and if they
> happen there's not a lot you can do at any one level to stop them.
> Baddies seem to be more interested in using their botnets to send spam
> or phish attacks or whatever - something which has financial gain.

Well, if there's an apetite for standards work in this area, might I suggest a
standardized protocol that can be used tell the routing infrastructure to
impose or remove a block as close to the ingress point as possible? I'd
personally prefer a RESTful web service, but almost anything would work. (But
please, let it not be SNMP with some security model no client library
implements. In fact let's steer clear of SNMP entirely here.)

I don't know about you, but using expect or whatever to script Cisco IOS
commands fails to impress.

Our MTA, and I suspect most others as well, is perfectly capable of
providing a block/unblock notification stream based on either simple
limits or fairly elaborate heuristics. So once you've terminated one of
these endless data streams, or one byte per minute streams, or any of the
many other forms of abuse, you can prevent it from consuming more resources
on anything inside of your administrative domain.

But the problem is typicaly that there's no viable consumer for those events.
And while defining a standard for such things is no guarantee that it will see
implementation, at least there would be a consistent way to talk to such
consumers.

				Ned

Date: Thu, 22 Oct 2009 15:12:01 -0400
From: John C Klensin <[email-masked]>
To: Paul Smith <[email-masked]>, David MacQuigg <[email-masked]>
cc: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

--On Thursday, October 22, 2009 19:21 +0100 Paul Smith
<[email-masked]> wrote:

>...
> As I see it, you could somehow work out a protocol for telling
> something that is sending you 5000 TB of data to 'stop it
> please', but (a) it could ignore you or (b) it could stop, but
> then start again straight away, so it sends you 10MB, then
> another 10MB, then another 10MB ad disk-blow-upeum.
> 
> This can happen with pretty much any protocol anywhere.
>...

And that analysis is almost certainly one of the reasons why the
multipart/partial idea in MIME has never gone much of anywhere
as a useful tool.

   john

Date: Thu, 22 Oct 2009 11:21:09 -0700 (PDT)
From: [email-masked]
Subject: RE: Abort data transfer?
To: Murray S Kucherawy <[email-masked]>
Cc: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Sender: [email-masked]

> > -----Original Message-----
> > From: [email-masked] [mailto:owner-ietf-
> > [email-masked]] On Behalf Of David MacQuigg
> > Sent: Thursday, October 22, 2009 9:43 AM
> > To: IETF SMTP list
> > Subject: Re: Abort data transfer?
> >
> > If the receiver is accepting data so fast, that it runs a few seconds
> > ahead of the milter process, then a few seconds of data might
> > accumulate
> > *beyond* where the milter says REJECT.  That's certainly less of a
> > problem than receiving data forever.

> This isn't how Sendmail works.  The entire message is cached to the queue
> before milter is told anything about the headers or body.  There's no "a few
> seconds ahead", it's all the way ahead.  Milter has no opportunity to say
> REJECT in the middle of the SMTP DATA phase because the filter doesn't even
> know that's where the MTA is.

Interesting... Our milter implementation actually sends the message to the
client as it is received. Of course any REJECTs or ACCEPTs or whatever that
occur along the way are noted and further output to the milter is blocked.
The status is then considered after the message is complete.

> Read the code in srvrsmtp.c and note that collect() is called before
> milter_data(); the former collects the entire message from the incoming stream,
> and the latter is what sends header and body information down to the filters.

Implementing it the way we did was something of a PITA; I certainly  can
understand why sendmail opted for the collect then send approach.

> > I just did an experiment with Sendmail using telnet as the client.  If
> > I
> > send one line of data, then pause, Sendmail waits 10 minutes then
> > disconnects with no message to telnet.  There is no timeout in telnet,
> > so it just sits there until I force a disconnect on that end.

> That's an MTA timeout and has nothing to do with milter.  Upon reaching this
> timeout, the filter has received the envelope data but none of the header or
> body, and will be given an "abort" instruction.

FWIW, our implementation has a similar timeout, but when it is reached it
spits back a 421 4.4.2 blah blah blah sort of line and closes the
connection. And of course we abort any milter that's in progress at this
point.

I have to say I don't understand the distinction between disconnection and
closing the connection that's being made here. AFAIK the only available
semantic here is to close the socket, and AFAIK the only socket option that
might affect how the close is done is SO_LINGER. We don't mess with SO_LINGER
when this sort of timeout happens, and when I try it with a telnet client I get
the 421 response and then the telnet client exits.

				Ned

Date: Thu, 22 Oct 2009 19:21:42 +0100
From: Paul Smith <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:
>
> Seems to me RFC-5321 could have said:  The client MUST pause every 
> 1000KB (or 100 seconds, whichever is less) and look for a REJECT.  
> Clients who fail to do this risk being blocked by a firewall.
What would be the point of this? A 'bad' client won't do this, and even 
if you do send a reject and it does honour it, it will just start 
sending data again, so - what's the point. Just block them at the 
firewall anyway - not that that would help against a botnet.

(A much better 'solution' in RFC 5321 would have been to enforce the 
SIZE extension, and allow the server to drop the connection if the 
client sends more than 10% over the advertised size of the email - it 
still wouldn't help in the bigger picture, but it might close this 
unexploited loophole)

As I see it, you could somehow work out a protocol for telling something 
that is sending you 5000 TB of data to 'stop it please', but (a) it 
could ignore you or (b) it could stop, but then start again straight 
away, so it sends you 10MB, then another 10MB, then another 10MB ad 
disk-blow-upeum.

This can happen with pretty much any protocol anywhere.

It has to be handled by heuristics of some sort, or not handled at all - 
in my experience serious DoS attacks are relatively rare, and if they 
happen there's not a lot you can do at any one level to stop them. 
Baddies seem to be more interested in using their botnets to send spam 
or phish attacks or whatever - something which has financial gain.

Date: Thu, 22 Oct 2009 11:12:09 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Murray S. Kucherawy wrote:
> This isn't how Sendmail works.  The entire message is cached to the queue
> before milter is told anything about the headers or body.  
...
> Read the code in srvrsmtp.c and note that collect() is called before
> milter_data();

Pretty silly design.

You'd almost be led to think that someone considered email to be a 
store-and-forward service.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Thu, 22 Oct 2009 19:10:38 +0100
From: Paul Smith <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:
> I just did an experiment with Sendmail using telnet as the client.  If 
> I send one line of data, then pause, Sendmail waits 10 minutes then 
> disconnects with no message to telnet.  There is no timeout in telnet, 
> so it just sits there until I force a disconnect on that end.
I'd expect most MTAs to do that. If they get data, then nothing, they 
will eventually timeout - the time may vary, but they'll all timeout. 
The client is then expected to retry.

That has nothing to do with a 'never ending' connection of zillions of 
MB of data - which was where we started. If you drop a connection 
because you don't like it for whatever reason, you are just asking for a 
retry, which achieves nothing if you are trying to stop the connection.

From: "Murray S. Kucherawy" <[email-masked]>
To: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Thu, 22 Oct 2009 11:02:45 -0700
Subject: RE: Abort data transfer?
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:owner-ietf-
> [email-masked]] On Behalf Of David MacQuigg
> Sent: Thursday, October 22, 2009 9:43 AM
> To: IETF SMTP list
> Subject: Re: Abort data transfer?
>=20
> If the receiver is accepting data so fast, that it runs a few seconds
> ahead of the milter process, then a few seconds of data might
> accumulate
> *beyond* where the milter says REJECT.  That's certainly less of a
> problem than receiving data forever.

This isn't how Sendmail works.  The entire message is cached to the queue b=
efore milter is told anything about the headers or body.  There's no "a few=
 seconds ahead", it's all the way ahead.  Milter has no opportunity to say =
REJECT in the middle of the SMTP DATA phase because the filter doesn't even=
 know that's where the MTA is.

Read the code in srvrsmtp.c and note that collect() is called before milter=
_data(); the former collects the entire message from the incoming stream, a=
nd the latter is what sends header and body information down to the filters=
.

> I just did an experiment with Sendmail using telnet as the client.  If
> I
> send one line of data, then pause, Sendmail waits 10 minutes then
> disconnects with no message to telnet.  There is no timeout in telnet,
> so it just sits there until I force a disconnect on that end.

That's an MTA timeout and has nothing to do with milter.  Upon reaching thi=
s timeout, the filter has received the envelope data but none of the header=
 or body, and will be given an "abort" instruction.

Date: Thu, 22 Oct 2009 13:58:39 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:

> 
> I just did an experiment with Sendmail using telnet as the client.  If I 
> send one line of data, then pause, Sendmail waits 10 minutes then 
> disconnects with no message to telnet.  There is no timeout in telnet, 
> so it just sits there until I force a disconnect on that end.
> 
> So it seems that Sendmail is doing what I suggested earlier - the least 
> possible work on the receiver side, no TCP reset, no attempt to close 
> the connection, just disconnect and let the transmitter hang.

It probably broke out of the receiver DATA loop, and is probably 
waiting for the client to send QUIT.  That is what our receiver will 
do and if that doesn't happen in 5 minutes, it will close the socket 
and kill the session.

220-winserver.com Wildcat! ESMTP Server v6.3.452.9 ready
220-************** WARNING:  FOR AUTHORIZED USE ONLY! 
**********************
220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY 
COMPUTERS    *
220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE 
UNSOLICITED *
220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT 
ACCESS *
220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY. 
      *
220 
************************************************************************
EHLO HDEV1
250-winserver.com, Pleased to meet you.
250-SIZE 10240000
250-SUBMITTER
250-ETRN
250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
250-AUTH=LOGIN
250 HELP
MAIL FROM:<xxxxxxxxx>
250 <xxxxxx>... Sender ok.
RCPT TO:<xxxxxxxxxxxxx>
250 <xxxxxxxxxx>... Recipient ok
DATA
354 Start mail input; end with <CRLF>.<CRLF>
421 Idle Timeout - Server closing transmission channel.

Connection to host lost.
               ----------------------------

Watching my clock, 354 was issued at 1:37pm.  The telnet terminal 
"connection to host lost" message came at 1:45pm.

Looking at the server code, we have a default 3 minute IDLE timeout 
for client data to arrive.  It dropped out of the loop (1:40pm) and 5 
minutes later (1:45pm) it closed the session.

The server side log is:

13:38:46 C: DATA
13:38:46 S: 354 Start mail input; end with <CRLF>.<CRLF>
13:46:47 S: 421 Idle Timeout - Server closing transmission channel.
13:46:47 ** Completed

So as you can see (server clock is off by 1 or my watch is <g>), the 
server does try to send out a 421 reply code and in TELNET you will 
most likely see this.

But I don't think that a dedicated SMTP client will be ready to see 
that 421 - or rather we can't presume it is designed to be ready to 
read and see it.  We don't know.

> Does anyone have experience with other MTAs?  It's a simple experiment.

Overall, I found when server drops logic was used, unwanted 
retransmissions occur due the SMTP's implicit client 451 reply code 
assumption.  I did not find any other way but to allow the client to 
terminate gracefully within the provided time limits.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 22 Oct 2009 09:42:36 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Murray S. Kucherawy wrote:
>> -----Original Message-----
>> From: [email-masked] [mailto:owner-ietf-
>> [email-masked]] On Behalf Of David MacQuigg
>> Sent: Wednesday, October 21, 2009 10:52 AM
>> To: IETF SMTP list
>> Subject: Re: Abort data transfer?
>>
>> My guess is that well-established programs already know how to deal
>> with
>> this.  In my own receiver, which is standard issue Sendmail plus my own
>> milter routines, I get a call after every header, and after every 60KB
>> body chunk.  If the number of chunks exceeds 20, I return a REJECT.
>> What Sendmail does with that REJECT, I have not verified, but I get no
>> further milter calls on that message.
>>     
>
> About sendmail in particular:
>
> By the time your filter gets the body call, the entire message has already been collected (and probably queued).  If the MTA runs out of disk or other resources while the message is in transit, your filter will be told the transaction was aborted.  It won't even have seen the header block yet, although it will have the envelope and know that DATA was at least started.
>   

If the receiver is accepting data so fast, that it runs a few seconds 
ahead of the milter process, then a few seconds of data might accumulate 
*beyond* where the milter says REJECT.  That's certainly less of a 
problem than receiving data forever.

I just did an experiment with Sendmail using telnet as the client.  If I 
send one line of data, then pause, Sendmail waits 10 minutes then 
disconnects with no message to telnet.  There is no timeout in telnet, 
so it just sits there until I force a disconnect on that end.

So it seems that Sendmail is doing what I suggested earlier - the least 
possible work on the receiver side, no TCP reset, no attempt to close 
the connection, just disconnect and let the transmitter hang.

Does anyone have experience with other MTAs?  It's a simple experiment.

-- Dave

Date: Wed, 21 Oct 2009 16:44:13 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: SMTP and DKIM/POLICY Rejection Handling [OFF LIST]
Sender: [email-masked]

Hector Santos wrote:

> I think ideally I could see a rejection code that maybe tells the 
> client "No Need to Forward" a bounce.  I believe that is the motive of 
> ADSP discard - to reduce backscattering.
>
> Maybe a SMTP extension can do that, call it a NoBounce
>
>   EHLO example.com
>   250-....
>   250 NOBOUNCE xyz
>
> where xyz is the optional code used to instruct the client it should 
> not create a DSN for an xyz rejection.
>
> This will assist new augmented 5322 DATA processing technology to 
> create a NoBounce rejection without the need to do a message acceptance.
>
> I like this better than to perpetuate ideas like discarding of mail 
> after message acceptance.

I don't think we can ever get back to the same level of delivery 
assurance we had before all the abuse, but perhaps we can establish two 
categories of mail.  First class is for senders who offer robust 
authentication, and maintain an A rating on reputation.  Email from 
these senders can be whitelisted, can expect near 100% reliable 
delivery, and can be assured of getting a DSN if there is a problem.  
All the rest can go through the usual spam filters, quarantine, etc.

Since ratings are done by individual receivers (or a group sharing 
reputation data), the sender cannot assume his A rating will be accepted 
by all receivers.  There must be a way to inform a sender of a 
"conditional" acceptance.  Currently, we have only 250 accept or 550 
reject.  If you want to send a message, it has to be a reject:

    Sorry! We cannot guarantee delivery of this message. 
    - <your domain> does not offer sufficient authentication to prevent
    forgery.
    - <your domain> does not have sufficient reputation for <recipient>.
    We will run it through our spam filter, and keep it in our
    quarantine, but the recipient may not read it.
    See  http://open-mail.org/rejects for help.

Perhaps we need a 350 reply code.  The command is accepted, but from 
this point forward, there is no guarantee of delivery.  That will 
certainly motivate any legitimate senders to correct whatever problems 
they have in authentication.

-- Dave

Date: 21 Oct 2009 18:28:23 -0400
From: "John R Levine" <[email-masked]>
To: "Ned Freed" <[email-masked]>
Cc: "Paul Smith" <[email-masked]>, "IETF SMTP list" <[email-masked]>
Subject: Re: Abort data transfer?
Cleverness: None detected
Sender: [email-masked]

> P.S. Another attack we're seen in the field a couple of times is the 
> "send one byte every few seconds and keep lots of connections open" one. 
> In fact if our experience is representative (I have no idea if it is or 
> not) this is actually more of a concern.

That I can believe.  The infamous Ron Guilmette had an amazing tarpit MTA 
about a decade ago that used that technique to keep 4000 simultaneous 
connections open, running on a 486.  But since the rise of botnets, 
tarpitting lost whatever effectiveness it might have had, so I don't know 
of anyone still doing it.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

Date: Wed, 21 Oct 2009 17:39:42 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:

> 
> 100KB seems like a sensible size chunk.  No need for a separate process 
> monitoring the reply channel.  No need to wait to make sure there is no 
> REJECT on each chunk.  A full handshake is not required.  Just don't 
> ignore any REJECTS that are sent.

Unfortunately David, as always, it is the bad guys using legacy SMTP 
client software for these DATA attacks could not be restricted by any 
new graceful control logic.

The force disconnect (drop) is all we have but that has an implicit 
451 reply code for the client.

I can only see some form of learning or behavior tracking to 
selectively use a drop.

For example, if an client shows extended client mode by using a return 
path SIZE= attribute and it exceeds this size in the data  transfer, 
then that might be a trigger for bad behavior tracing.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 21 Oct 2009 13:29:44 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Ned Freed wrote:

>> It just seems like bad design to assume some input will never happen.
>
> Of course it is, which is why it's a good idea to have facilities in 
> servers to
> address this. OTOH, implementations of such capabilities is a long way 
> from
> rockey science, and obsessing over and overdesigning facilities to 
> deal with
> this is also bad design.
>
>> In this case, a DoS attack would not even require a botnet.  Even if the
>> receiver starts discarding data, an attacker with one machine could jam
>> the receiver with hundreds of SMTP processes, all waiting for a very
>> slow stream of data, just enough that they can't be swapped out to
>> consume less memory.
>
> The slow data attack is really a different beast, and is somewhat harder
> to deal with because timers and heuristics are involved.
>
> It might, and I emphasize might, be worth adding some discussion of 
> this case,
> assuming we can come up with something that's both general and useful.
>
>> My guess is that well-established programs already know how to deal with
>> this.  In my own receiver, which is standard issue Sendmail plus my own
>> milter routines, I get a call after every header, and after every 60KB
>> body chunk.  If the number of chunks exceeds 20, I return a REJECT.
>> What Sendmail does with that REJECT, I have not verified, but I get no
>> further milter calls on that message.
>
> I can't speak for anyone else's server, but we have both time and 
> data-based
> disconnect limits. I've considered adding a minimum data/time, but 
> concern over
> customer lack of understanding and abuse of such a limit has stopped 
> me so far.
>
>> This looks to me like a security flaw in SMTP.
>
> I dsiagree, or rather, if it is, it's a flaw in pretty much every 
> protocol in
> existance. And since even if you lock down every possible case at the 
> server
> level there are still very effective attacks at the TCP level, this is 
> arguably
> a flaw in the design of the entire stack.

Sure there are DoS problems at other layers of the stack - SYN floods 
being a good example - but these are outside our current scope.  What 
I'm interested in is the application layer, and specifically SMTP.  It 
seems to me that there should be a simple way to avoid any 
vulnerabilities added by SMTP.  I can set limits on time and total data 
in my receiver, but if the transmitter ignores my REJECT, it becomes 
transport layer problem, and I hand it off to the firewall - block this 
IP address for 24 hours!

Seems to me RFC-5321 could have said:  The client MUST pause every 
1000KB (or 100 seconds, whichever is less) and look for a REJECT.  
Clients who fail to do this risk being blocked by a firewall.

100KB seems like a sensible size chunk.  No need for a separate process 
monitoring the reply channel.  No need to wait to make sure there is no 
REJECT on each chunk.  A full handshake is not required.  Just don't 
ignore any REJECTS that are sent.

-- Dave

Date: Wed, 21 Oct 2009 12:15:14 -0700 (PDT)
From: [email-masked]
Subject: Re: Abort data transfer?
To: David MacQuigg <[email-masked]>
Cc: IETF SMTP list <[email-masked]>
Sender: [email-masked]

> Paul Smith wrote:
> > John R Levine wrote:
> >
> >> No, I meant where the data in a single DATA connection never stops,
> >> since that is the alleged problem that this argument is about.
> >>
> >> If, as I suspect, it has never happened in the entire history of the
> >> Internet, why in the world is anyone wasting time worrying about it?
> >> The scenario you describe is rare enough and has little enough impact
> >> that you could deal with it by hand.
> >>
> > I guess the thing is that it COULD be used as a DoS attack - quite
> > easily. However, it doesn't seem to be, because there are lots of other
> > ways of doing a DoS attack - and those are possibly better. A 'lasting
> > forever' connection is more likely to be spotted by someone than lots of
> > smaller connections, so I would suspect that botnet operators would
> > prefer the latter to the former.
> >
> > A rotating set of 10,000 bots at a time flooding your server with small
> > messages is less likely to be spotted or stoppable than a fixed set of
> > 10,000 bots trying to send infinitely long messages. If nothing else,
> > the latter will almost definitely NOT fill up the server's disk (most
> > servers will start discarding data after it gets over a certain amount),
> > whereas the former could well do so.
> >

> It just seems like bad design to assume some input will never happen.

Of course it is, which is why it's a good idea to have facilities in servers to
address this. OTOH, implementations of such capabilities is a long way from
rockey science, and obsessing over and overdesigning facilities to deal with
this is also bad design.

> In this case, a DoS attack would not even require a botnet.  Even if the
> receiver starts discarding data, an attacker with one machine could jam
> the receiver with hundreds of SMTP processes, all waiting for a very
> slow stream of data, just enough that they can't be swapped out to
> consume less memory.

The slow data attack is really a different beast, and is somewhat harder
to deal with because timers and heuristics are involved.

It might, and I emphasize might, be worth adding some discussion of this case,
assuming we can come up with something that's both general and useful.

> My guess is that well-established programs already know how to deal with
> this.  In my own receiver, which is standard issue Sendmail plus my own
> milter routines, I get a call after every header, and after every 60KB
> body chunk.  If the number of chunks exceeds 20, I return a REJECT.
> What Sendmail does with that REJECT, I have not verified, but I get no
> further milter calls on that message.

I can't speak for anyone else's server, but we have both time and data-based
disconnect limits. I've considered adding a minimum data/time, but concern over
customer lack of understanding and abuse of such a limit has stopped me so far.

> This looks to me like a security flaw in SMTP.

I dsiagree, or rather, if it is, it's a flaw in pretty much every protocol in
existance. And since even if you lock down every possible case at the server
level there are still very effective attacks at the TCP level, this is arguably
a flaw in the design of the entire stack.

				Ned

Date: Wed, 21 Oct 2009 11:40:32 -0700 (PDT)
From: [email-masked]
Subject: Re: Abort data transfer?
To: John R Levine <[email-masked]>
Cc: Paul Smith <[email-masked]>, IETF SMTP list <[email-masked]>
Sender: [email-masked]

> > I guess the thing is that it COULD be used as a DoS attack -

> There's lots of things COULD be used as an attack.  The fact that it's
> never happened in 30 years suggest this is not one worth worrying about.

FWIW, we had a customer who ran into exactly this attack about five years back:
Endless data coming in on an SMTP connection.

We have facilities both to ignore data past a (settable) limit and to drop
connections past a (settable) limit, but they weren't engaged. As I recall,
engaging them solved the immediate problem, but the attacker switched  almost
immediately to a stream of SYNs, which the mail server could handle but which
nevertheless ate up all available bandwidth. At that point this ceased to be a
mail server issue and became a router issue so we were out of the picture and I
don't recall hearing the resolution.

In any case, I think this particular branch on the attack tree is pretty thin
compared to some of its neighbors, but it does exsit.

				Ned

P.S. Another attack we're seen in the field a couple of times is the "send one
byte every few seconds and keep lots of connections open" one. In fact if our
experience is representative (I have no idea if it is or not) this is actually
more of a concern.

Date: Wed, 21 Oct 2009 14:42:17 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: IETF SMTP list <[email-masked]>
CC: Paul Smith <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

John R Levine wrote:

> 
>> I guess the thing is that it COULD be used as a DoS attack -
> 
> There's lots of things COULD be used as an attack.  The fact that it's 
> never happened in 30 years suggest this is not one worth worrying about.

Unfortunately, that same attitude was applied to accept/bounce 
attacks.  Small or rare enough that operators can handle it by hand.

That is until 18-20 or so years later when the 2003 SORBIG dual blitz 
offense accept/bounce attacks woke up the industry in 2003.

Just consider that if a hacker is reading this list, he is finding out 
pretty quickly that many systems are vulnerable to this form of very 
large or never ending DATA payload attacks.  Systems that not prepare 
for it and don't have people available to handle it "by hand" will be 
shutdown.

So even if is rare, it does not mean it should be ignored.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 21 Oct 2009 20:15:22 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Abort data transfer?
Cc: John R Levine <[email-masked]>, [email-masked]
Sender: [email-masked]

John R Levine writes:
>>>> This doesn't deal with the case where the data never stops.
>>>
>>> In all the years I've been running mail servers, I don't ever recall 
>>> seeing a sender that never stopped. How often have you seen one?
>>
>> Once. A windows box in the UAE, if I remember correctly. It 
>> connected, sent MAIL FROM and RCP TO, got a 50x, disconnected, and 
>> tried again five minutes later. When I noticed it, it had been doing 
>> so for as long as I had logs, probably 99 weeks.
>
> No, I meant where the data in a single DATA connection never stops, 
> since that is the alleged problem that this argument is about.

Oh, sorry. I think of all these attack variations as interchangeable 
since the attacker can change the attack so easily.

> If, as I suspect, it has never happened in the entire history of the 
> Internet, why in the world is anyone wasting time worrying about it?

It has been discussed several times, I even remember a flamewar. The 
outcomes, AFAICR, always tended towards "if resource (ram/disk/cpu) 
usage is the problem, resource usage is the thing to watch" or "use 
setrlimit". That is, who cares whether the problem is long RCPT lines, 
many of them, long BODY, many concurrent connections tying up resources 
just below your administrative limit, or some x*y*z combination: just 
watch for unacceptable resource usage.

>  The scenario you describe is rare enough and has little enough impact 
> that you could deal with it by hand.

Yes. I wish I had more problems that go harmlessly unnoticed for a year 
or more.

Arnt

From: "Murray S. Kucherawy" <[email-masked]>
To: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Wed, 21 Oct 2009 11:10:12 -0700
Subject: RE: Abort data transfer?
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:owner-ietf-
> [email-masked]] On Behalf Of David MacQuigg
> Sent: Wednesday, October 21, 2009 10:52 AM
> To: IETF SMTP list
> Subject: Re: Abort data transfer?
>=20
> My guess is that well-established programs already know how to deal
> with
> this.  In my own receiver, which is standard issue Sendmail plus my own
> milter routines, I get a call after every header, and after every 60KB
> body chunk.  If the number of chunks exceeds 20, I return a REJECT.
> What Sendmail does with that REJECT, I have not verified, but I get no
> further milter calls on that message.

About sendmail in particular:

By the time your filter gets the body call, the entire message has already =
been collected (and probably queued).  If the MTA runs out of disk or other=
 resources while the message is in transit, your filter will be told the tr=
ansaction was aborted.  It won't even have seen the header block yet, altho=
ugh it will have the envelope and know that DATA was at least started.

Date: Wed, 21 Oct 2009 10:51:58 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Paul Smith wrote:
> John R Levine wrote:
>   
>> No, I meant where the data in a single DATA connection never stops,
>> since that is the alleged problem that this argument is about.
>>
>> If, as I suspect, it has never happened in the entire history of the
>> Internet, why in the world is anyone wasting time worrying about it?
>> The scenario you describe is rare enough and has little enough impact
>> that you could deal with it by hand.
>>     
> I guess the thing is that it COULD be used as a DoS attack - quite
> easily. However, it doesn't seem to be, because there are lots of other
> ways of doing a DoS attack - and those are possibly better. A 'lasting
> forever' connection is more likely to be spotted by someone than lots of
> smaller connections, so I would suspect that botnet operators would
> prefer the latter to the former.
>
> A rotating set of 10,000 bots at a time flooding your server with small
> messages is less likely to be spotted or stoppable than a fixed set of
> 10,000 bots trying to send infinitely long messages. If nothing else,
> the latter will almost definitely NOT fill up the server's disk (most
> servers will start discarding data after it gets over a certain amount),
> whereas the former could well do so.
>   

It just seems like bad design to assume some input will never happen.  
In this case, a DoS attack would not even require a botnet.  Even if the 
receiver starts discarding data, an attacker with one machine could jam 
the receiver with hundreds of SMTP processes, all waiting for a very 
slow stream of data, just enough that they can't be swapped out to 
consume less memory.

My guess is that well-established programs already know how to deal with 
this.  In my own receiver, which is standard issue Sendmail plus my own 
milter routines, I get a call after every header, and after every 60KB 
body chunk.  If the number of chunks exceeds 20, I return a REJECT.  
What Sendmail does with that REJECT, I have not verified, but I get no 
further milter calls on that message.

This looks to me like a security flaw in SMTP.

-- Dave

Date: 21 Oct 2009 13:19:49 -0400
From: "John R Levine" <[email-masked]>
To: "Paul Smith" <[email-masked]>
Cc: "IETF SMTP list" <[email-masked]>
Subject: Re: Abort data transfer?
Cleverness: None detected
Sender: [email-masked]

> I guess the thing is that it COULD be used as a DoS attack -

There's lots of things COULD be used as an attack.  The fact that it's 
never happened in 30 years suggest this is not one worth worrying about.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

Date: Wed, 21 Oct 2009 17:38:36 +0100
From: Paul Smith <[email-masked]>
To: John R Levine <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, IETF SMTP list <[email-masked]>, [email-masked]
Subject: Re: Abort data transfer?
Sender: [email-masked]

John R Levine wrote:
>
> No, I meant where the data in a single DATA connection never stops,
> since that is the alleged problem that this argument is about.
>
> If, as I suspect, it has never happened in the entire history of the
> Internet, why in the world is anyone wasting time worrying about it?
> The scenario you describe is rare enough and has little enough impact
> that you could deal with it by hand.
I guess the thing is that it COULD be used as a DoS attack - quite
easily. However, it doesn't seem to be, because there are lots of other
ways of doing a DoS attack - and those are possibly better. A 'lasting
forever' connection is more likely to be spotted by someone than lots of
smaller connections, so I would suspect that botnet operators would
prefer the latter to the former.

A rotating set of 10,000 bots at a time flooding your server with small
messages is less likely to be spotted or stoppable than a fixed set of
10,000 bots trying to send infinitely long messages. If nothing else,
the latter will almost definitely NOT fill up the server's disk (most
servers will start discarding data after it gets over a certain amount),
whereas the former could well do so.

-- 
Paul Smith

VPOP3 - POP3/SMTP/IMAP4/Webmail Email server for Windows

Date: 21 Oct 2009 11:18:25 -0400
From: "John R Levine" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>
Cc: "IETF SMTP list" <[email-masked]>, [email-masked]
Subject: Re: Abort data transfer?
Cleverness: None detected
Sender: [email-masked]

>>> This doesn't deal with the case where the data never stops.
>> 
>> In all the years I've been running mail servers, I don't ever recall seeing 
>> a sender that never stopped. How often have you seen one?
>
> Once. A windows box in the UAE, if I remember correctly. It connected, sent 
> MAIL FROM and RCP TO, got a 50x, disconnected, and tried again five minutes 
> later. When I noticed it, it had been doing so for as long as I had logs, 
> probably 99 weeks.

No, I meant where the data in a single DATA connection never stops, since 
that is the alleged problem that this argument is about.

If, as I suspect, it has never happened in the entire history of the 
Internet, why in the world is anyone wasting time worrying about it? The 
scenario you describe is rare enough and has little enough impact that you 
could deal with it by hand.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

Date: Wed, 21 Oct 2009 10:19:55 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Sabahattin Gucukoglu <[email-masked]>
CC: David MacQuigg <[email-masked]>, [email-masked]
Subject: Re: Abort data transfer?
Sender: [email-masked]

Another way to look at this and probably how it can manifest itself is 
with prolonged DATA processing on the server side, enough so that the 
client drops the line and retries.

This is indirectly noted in RFC 5321 section 6.1 with:

    To avoid receiving duplicate messages as the result of timeouts, a
    receiver-SMTP MUST seek to minimize the time required to respond to
    the final <CRLF>.<CRLF> end of data indicator.  See RFC 1047 [40]
    for a discussion of this problem.

The reference RFC 1047 (Duplicate Message and SMTP) describes:

    Many mailers delay responding to the final dot because they are
    doing sophisticated processing of the message, in an attempt to
    confirm that they can deliver the message.  For example,
    the mailers may expand an entire mailing list to confirm that
    it can reach all addressees or may attempt to physically
    deposit the message into the mailboxes of local users,
    before confirming receipt of the final dot.  These practices
    are not unreasonable, but they often cause the synchronization
    gap to continue for several minutes, and increase the
    likelihood that the sending mailer will timeout or the network
    will fail before the accepting 250 reply is sent.

    AVOIDING SYNCHRONIZATION PROBLEMS

    The best way to avoid the synchronization problem is to minimize
    the length of the synchronization gap.  In other words,
    receiving mailers should acknowledge the final dot as soon
    as possible and do more complex processing of the message later.

However, the issues we found.

The 1988 RFC 1047 note was molded in the days of single threaded, 
process, slower machines and lower bandwidth. The relative rare (less 
volume) malicious accept/bounce mail attacks were not taken as 
serious. The mindset was that a "user" attempting to fake mail should 
not degrade or disabling the benefits of a "relaxed" open email 
system.  This mindset is shown in RFC 5321 section 7.1.

The problem of course is today's reality:  Its no longer a "user" but 
an multi-billion dollar problem and industry of users trying to fake 
mail.  With multi-threaded designs, faster machines, better 
scalability and efficiency makes it all possible for more SMTP 
receivers to do processing dynamically at the SMTP DATA level.

Nonetheless, if a server data processing is prolonged, we found there 
are clients who drop at 3 minutes after the <CRLF>.<CRLF> and before 
the server issues a reply code.

Because of the SMTP NO QUIT CANCEL SMTP semantics, the server detects 
the drop and may cancel the transaction. Systems following the RFC 
1047 guidelines, always accept and post the message are subject to 
duplicates.

So the lesson is probably that the DATA processing, if any, needs to 
be fast and less than 2-3 minutes.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Sabahattin Gucukoglu wrote:

> 
> On 20 Oct 2009, at 02:20, David MacQuigg wrote:
>> Arnt Gulbrandsen wrote (about letting the transmitter hang):
>>> Some people do this in order to be as occupy the sender's resources. 
>>> There's a fancy name for it. You send one byte every few minutes, so 
>>> it takes three hours to send the whole SMTP response. I don't know 
>>> whether doing so really inconveniences the $#@$#@! spammers and other 
>>> evil creatures who initiate most SMTP connections.
>>
>> I wasn't thinking of a tarpit.  That would involve keeping a 
>> connection on my end.  Just dropping the connection will leave the 
>> transmitter with an SMTP process filling memory for maybe ten minutes.
> 
> Really?  I rather fancy it will inform the sender's TCP that you're done 
> with it, and the next few failed sends will go completely unnoticed by 
> ratware and actioned upon by connection closure by a real MTA on a 
> failed send or on an error detected on read by some more fancy 
> event-driven process.  In any case the length of the long wait is one 
> TCP buffer, or not very long at all except perhaps for sneakernet. :-)
> 
> Perhaps you mean something more complex, maybe losing TCP state 
> completely at your end.  That might make a few long-suffering 
> implementations wait a good while to give up.  But if you were going to 
> go that far, you might as well turn a real BOFH, advertise an MSS of 1 
> to evil clients, drop every packet except the fifteenth a turn, and 
> delay each acknowledgement by five minutes. >:->
> 
>> SM wrote:
>>> Section 4.5.3.1.7 of RFC 5321 discusses about the maximum total 
>>> length of a message content.  There isn't a permissible way within 
>>> SMTP to abort during DATA.  It's a "don't do it unless you know what 
>>> you are doing".
>>
>> I see no technical problem on the server side with sending "552 Too 
>> much mail data," then a TCP reset.  If the client keeps retrying, 
>> without fixing the problem, I would block that IP address for a few 
>> hours.
> 
> It should work, but you can't depend on it.  It would work with my 
> client since all errors are detected during reads and not writes.  But 
> you simply can't depend on a client maintaining state the way you hope 
> it should.  If sends fail because you've reset the connection, you may 
> simply cause the sender to abort the attempt and retry.  It's that simple.
> 
>> Sabahattin Gucukoglu wrote:
>>> My implementations note the reason for rejection, then switch the 
>>> event handler for the connection to one that just reads each buffer 
>>> and then throws it away.  There is nothing more that can be done.
>>
>> This could leave your receiver tied up for as long as the client keeps 
>> spewing garbage.
> 
> Correct.  SMTP is not "Broken", as some people would have it, but it's a 
> cruel world out there.  Be prepared for the worst.
> 
> Though I don't say you should not try to defend against DoS attacks, 
> realistically any malicious person already has numerous ways to attack 
> available services no matter how you protect them.  For SMTP, rate 
> limit, connection limit per IP, etc are about the best you can hope 
> for.  A session timeout may also be plausible, though I didn't implement 
> one in the server, only RFC-specified inactivity timeouts and am only 
> half-convinced of a need to put it into the client.
> 
>>> Even trying to inform the sender by writing the error to it while it 
>>> is sending won't necessarily work, because it might not choose to 
>>> look at its inbuffers until it's quite finished spewing.  And in any 
>>> case that's standards violation territory you're wandering into.
> 
>>
>> Which standard?  The paragraph above from Section 3.3. of 5321 seems 
>> to be talking about after the 354 reply is issued (past tense).  I 
>> assume that means any time after that, not necessarily waiting for the 
>> end of data.
>>
> Nothing in RFC 5321, as far as I know and understand it, allows you to 
> circumvent the lock-step nature of SMTP.  That section seems to 
> corroborate that.  The best you can do is suddenly become unavailable, 
> with apologies to the client.  I suppose you can interpret the text to 
> mean that closing the connection for other reasons is acceptable but as 
> has already been explained by others this isn't called for for any case 
> other than resource unavailability and will, quite reasonably, give you 
> client retries.
> 
>>> So you'll just have to do what you can to keep the connection alive 
>>> for the data to come in, then reject it and hopefully at that point 
>>> you can resume the dialogue to finish up and ensure the sender will 
>>> leave you alone.  Anything else is basically suboptimal.
>>
>> If we have a broken or malicious sender, it seems that the optimal 
>> solution would be one with the least resources on the server side.  
>> That would be not even keeping the connection alive.
>>
>> That works only if you don't want retries.  So it's suboptimal.  
>> Maybe, as has been said before, we've simply never been bothered 
>> enough about it to put it right, and make it more optimal.  But right 
>> now, accepting data is basically mandatory, except when it isn't, 
>> which is to say, at the beginning of the response to the DATA command 
>> or after the final dot, and not inbetween.
> 
> Cheers,
> Sabahattin
> 
> 
> 

Date: Wed, 21 Oct 2009 09:55:03 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:

>> I would just continue accepting the message, and throw away the data,
>> then give a 'message too big' error at the end.  

 >

> This doesn't deal with the case where the data never stops.

What would be the criteria?

That is why our server, once upon a time, had a block transfer count 
check against local policy size limits with the drop idea as well.

But it was pulled due to retransmissions and mostly seen from 
legitimate users.  We want the client MUA to see a normal server 
response to this and have it popup the dialog to the user hopefully 
showing the server response text.   The odds are good nor should we 
expect the MUA to read some forced DATA abort server text from the 
server.  At best we can expect the client to be modeled as followed in 
a client/server state machine fashion:

DATA state:

     Data
       read data
       err = send data
       if err is socket closed then
          // RFC 2821/5321 says assume 451
          close resources
          requeue
          return false
       end if
     Loop
     wait 5 mins (default) for server response

We can't assume that for an socket error that it will exit the loop 
and capture some legacy transmission data remaining in the socket 
buffers.  It MAY be there, but we can't assume this.

     Data
       read data
       err = send data
       if err is not successful then
          // see if there is any buffered data in input stream
          exit loop
       end if
     Loop
     wait 5 mins (default) for server response

We have one rule left related to this on the server's data transfer 
loop and outside the loop.  If the detected extended SMTP sender 
provided MAIL SIZE info and it exceeded the size in the transfer, then 
this is trapped and the sender info is recorded.  But we don't drop 
the line anymore.  I can't tell you off hand if we have any logs on this.

So I would lean towards Levine's point, that it is rare to see this, 
and probably simply because that it would be trapped under normal 
circumstances after the data transfer is complete.  Sure, a pain in 
the butt - but they are not defeating your server.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 21 Oct 2009 11:31:28 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Abort data transfer?
Cc: John Levine <[email-masked]>, [email-masked]
Sender: [email-masked]

John Levine writes:
>> This doesn't deal with the case where the data never stops.
>
> In all the years I've been running mail servers, I don't ever recall 
> seeing a sender that never stopped. How often have you seen one?

Once. A windows box in the UAE, if I remember correctly. It connected, 
sent MAIL FROM and RCP TO, got a 50x, disconnected, and tried again 
five minutes later. When I noticed it, it had been doing so for as long 
as I had logs, probably 99 weeks.

On one hand, it was extremely irritating once I was aware of it. On the 
other, I didn't exactly notice it at once ;)

Arnt

Date: 21 Oct 2009 04:01:11 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: Abort data transfer?
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>This doesn't deal with the case where the data never stops.

In all the years I've been running mail servers, I don't ever
recall seeing a sender that never stopped.  How often have you
seen one?

R's,
John

Date: Tue, 20 Oct 2009 19:46:46 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Paul Smith wrote:
> David MacQuigg wrote:
>   
>> Forget about DKIM.  What if there is some other reason to abort, like
>> the payload is too large?  We cannot just continue receiving data
>> forever.  Is there a "permissible" way within SMTP to abort during
>> data, or should I just ignore these ambiguous requirements, and
>> either: 1) Send a TCP reset, or 2) Let the transmitter hang.
>>     
> I would just continue accepting the message, and throw away the data,
> then give a 'message too big' error at the end.  

This doesn't deal with the case where the data never stops.

> I suppose you could try to be clever and do a grey-listing-style thing
> of recording the IP address, sender & recipient, and dropping the
> connection, then when the retry is done, reject the message before the
> data is sent - but I wouldn't be too confident about the reliability of
> that.
>   

Good suggestion.  The IP address is all we really need to identify the 
transmitter, however.  We need to allow at least 3 retries, in case the 
sender is legit, and trying to fix a problem.  After that, I don't think 
anyone should complain about having their transmitter blacklisted for a 
day or two.  If it takes more than 3 tries (or some other measure, like 
total minutes the receiver has been listening to crap) if it takes more 
to fix the problem, they shouldn't be practicing on our client's receiver.

We could set up a special test server just for such practice.  Any 
message to that server gets a nice reply with complete diagnostics on 
any problems we find with their authentication setup, maybe even a 
history of problems from that transmitter, in case they don't know why 
they were blacklisted.

Yes, we could separate different message streams based on sender & 
recipient addresses, and it wouldn't be that much extra work, but then 
our rejects and domain ratings are more likely to be ignored.  The 
Transmitter Agent (operator of the outgoing border MTA) is responsible 
for problems generated by his clients.  Most legitimate operators are 
honest and competent, and the mail from their transmitters is never a 
problem.  The few who have problems may be incompetent, or they may feel 
they have business reasons to allow the abuse.  It doesn't matter what 
the reason.  If lumping all their customers together is unfair to some, 
they should put pressure on their agent to fix the problem, or find 
another agent.  A Transmiter Agent can insist that his clients follow 
acceptable practices, and can provide per-account rate limits and other 
measures to help their clients avoid abuse by individual accounts.

-- Dave

Date: Tue, 20 Oct 2009 16:17:01 +0100
From: Paul Smith <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:
>
> Forget about DKIM.  What if there is some other reason to abort, like
> the payload is too large?  We cannot just continue receiving data
> forever.  Is there a "permissible" way within SMTP to abort during
> data, or should I just ignore these ambiguous requirements, and
> either: 1) Send a TCP reset, or 2) Let the transmitter hang.
I would just continue accepting the message, and throw away the data,
then give a 'message too big' error at the end.

I suppose you could try to be clever and do a grey-listing-style thing
of recording the IP address, sender & recipient, and dropping the
connection, then when the retry is done, reject the message before the
data is sent - but I wouldn't be too confident about the reliability of
that.

-- 
Paul Smith

VPOP3 - POP3/SMTP/IMAP4/Webmail Email server for Windows

Date: Tue, 20 Oct 2009 10:23:00 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: SMTP and DKIM/POLICY Rejection Handling
Sender: [email-masked]

Alessandro Vesely wrote:

> That still leaves the remailer's problem. Say Alice runs a mailing list 
> that requires specific subject-prefix and footer, and Bob posts a 
> contribute that contains neither of them. Now, if the contribute comes 
> from a bank that signed the subject and omitted the l= tag for the body 
> length, and the bank's ADSP RR is "dkim=discardable", what should Alice 
> do? 

Good Question. :)

> Her choices are to either pass it, possibly removing the broken 
> signature, or reject the contribute, requiring that Bob adds the 
> subject-prefix and the footer himself. In facts, after verifying the 
> signature, Alice can safely send a bounce to Bob, and explain the issue 
> in full detail.

+1

But should ALICE even bother to check Bob's ADSP record (ignore ADSP) 
and just forward it?

PRO:

   - keep mail following, bank mail original
     signature stripped, resigned and forwarded

CON:

   - Bank is a spoofed domain submission

   - membership with ADSP receivers can reject

   - memberships can get subscription removal due
     to exhausted retries.

If we assume all ADSP receivers will always follow a silent discard 
rather than reject, then the CON reduces to the unprotected spoof 
domain submission risk.

I think the bank should not be using a ADSP "discardable" if it knows 
it is going to submit mail to a DKIM ready remailer not supporting 
ADSP. So this is the bank's problem to fix.

But I think its probably better to just filter it and maybe notify the 
bank it has a "configuration" problem.   That might help reduce the 
problem of ADSP compliant SMTP receivers rejecting mail.

> As for possibly discarding broken signatures --which 
> would seem to just save the recipients some CPU cycles, since the effect 
> is the same-- RFC 4871 sec. 4.2 says
> 
>  Signers SHOULD NOT remove any DKIM-Signature header fields from
>  messages they are signing, even if they know that the signatures
>  cannot be verified.
> 
> Does that mean Alice may remove the broken signature only if she does 
> not in turn sign the message --i.e. she is not a signer? Hm...

I believe the above means that the 5322 headers the DKIM signature 
bound in the hash defined with the h= tag, i.e. h=From:SubjectTo:Date 
etc should not be removed from the 5322 headers when the 
5322.DKIM-Signature header itself is stripped.

>> So you suggest a specific "dkim=rejectable" would apply for SMTP 
>> rejects and "dkim=discardable" for post message acceptance silent 
>> discards?
> 
> Rejection is probably safer in most cases.

Yes, +1.

But maybe not when the reason is a remailer breaking/forwarding ADSP 
protected domain messages.

Maybe we do need a specific temp rejection code here that says

     453 4.5.3 ADSP rejection

to avoid issuing a 250 message acceptance.  453 might tell the 
remailer that what it did should probably be address short of not 
sending subscription removal notifications to members.

>> I don't think domains declaring a "actionable" ADSP policy such as a 
>> DKIM=DISCARDABLE|REJECTABLE really care how a SMTP verifier deals with 
>> ADSP policy violation other than to suggest "get rid it, don't accept 
>> it" - they don't want to claim any responsibility for the broken 
>> DKIM/ADSP message and is providing explicit receiver handling 
>> suggestions.
> 
> That sounds reasonable. (RFC 5016 just talks about "great suspicion".) 
> Do you mean they used "discardable" for "either droppable or rejectable"?

Well, the reality is that both type of SMTP systems exist and 
increasingly so with faster machined, better scaled and dynamic 
processing, do perform DATA analysis rather than wait to after the 
message is accepted. This helps mitigate the accept/bounce attack problem.

So it should be expected that lacking specific handling guidelines, 
you will find a growing amount of SMTP implementators like David 
Macquigg perform this task at the DATA state.

Your dkim=rejectable idea is good, but do we really need to have both?

Of course, RFC5617 should be covering this better :)

>> I am not sure we need an SMTP extension for this, IMO, codifying new 
>> 5321/5322 related standard track specifications could also resolve 
>> conflictive guidance for implementators.
> 
> The advantage of an extension is to mention a handful of existing 
> anti-spam checks directly, allowing future changes either via further 
> extensions or rewritings of the same spec. The new SMTP spec will be a 
> Full Standard, not expected to change any time soon.

Not in my life time.:)

I think ideally I could see a rejection code that maybe tells the 
client "No Need to Forward" a bounce.  I believe that is the motive of 
ADSP discard - to reduce backscattering.

Maybe a SMTP extension can do that, call it a NoBounce

   EHLO example.com
   250-....
   250 NOBOUNCE xyz

where xyz is the optional code used to instruct the client it should 
not create a DSN for an xyz rejection.

This will assist new augmented 5322 DATA processing technology to 
create a NoBounce rejection without the need to do a message acceptance.

I like this better than to perpetuate ideas like discarding of mail 
after message acceptance.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Subject: Re: Abort data transfer?
From: Sabahattin Gucukoglu <[email-masked]>
Date: Tue, 20 Oct 2009 13:06:52 +0100
Cc: [email-masked]
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

On 20 Oct 2009, at 02:20, David MacQuigg wrote:
> Arnt Gulbrandsen wrote (about letting the transmitter hang):
>> Some people do this in order to be as occupy the sender's  
>> resources. There's a fancy name for it. You send one byte every few  
>> minutes, so it takes three hours to send the whole SMTP response. I  
>> don't know whether doing so really inconveniences the $#@$#@!  
>> spammers and other evil creatures who initiate most SMTP connections.
>
> I wasn't thinking of a tarpit.  That would involve keeping a  
> connection on my end.  Just dropping the connection will leave the  
> transmitter with an SMTP process filling memory for maybe ten minutes.

Really?  I rather fancy it will inform the sender's TCP that you're  
done with it, and the next few failed sends will go completely  
unnoticed by ratware and actioned upon by connection closure by a real  
MTA on a failed send or on an error detected on read by some more  
fancy event-driven process.  In any case the length of the long wait  
is one TCP buffer, or not very long at all except perhaps for  
sneakernet. :-)

Perhaps you mean something more complex, maybe losing TCP state  
completely at your end.  That might make a few long-suffering  
implementations wait a good while to give up.  But if you were going  
to go that far, you might as well turn a real BOFH, advertise an MSS  
of 1 to evil clients, drop every packet except the fifteenth a turn,  
and delay each acknowledgement by five minutes. >:->

> SM wrote:
>> Section 4.5.3.1.7 of RFC 5321 discusses about the maximum total  
>> length of a message content.  There isn't a permissible way within  
>> SMTP to abort during DATA.  It's a "don't do it unless you know  
>> what you are doing".
>
> I see no technical problem on the server side with sending "552 Too  
> much mail data," then a TCP reset.  If the client keeps retrying,  
> without fixing the problem, I would block that IP address for a few  
> hours.

It should work, but you can't depend on it.  It would work with my  
client since all errors are detected during reads and not writes.  But  
you simply can't depend on a client maintaining state the way you hope  
it should.  If sends fail because you've reset the connection, you may  
simply cause the sender to abort the attempt and retry.  It's that  
simple.

> Sabahattin Gucukoglu wrote:
>> My implementations note the reason for rejection, then switch the  
>> event handler for the connection to one that just reads each buffer  
>> and then throws it away.  There is nothing more that can be done.
>
> This could leave your receiver tied up for as long as the client  
> keeps spewing garbage.

Correct.  SMTP is not "Broken", as some people would have it, but it's  
a cruel world out there.  Be prepared for the worst.

Though I don't say you should not try to defend against DoS attacks,  
realistically any malicious person already has numerous ways to attack  
available services no matter how you protect them.  For SMTP, rate  
limit, connection limit per IP, etc are about the best you can hope  
for.  A session timeout may also be plausible, though I didn't  
implement one in the server, only RFC-specified inactivity timeouts  
and am only half-convinced of a need to put it into the client.

>> Even trying to inform the sender by writing the error to it while  
>> it is sending won't necessarily work, because it might not choose  
>> to look at its inbuffers until it's quite finished spewing.  And in  
>> any case that's standards violation territory you're wandering into.

>
> Which standard?  The paragraph above from Section 3.3. of 5321 seems  
> to be talking about after the 354 reply is issued (past tense).  I  
> assume that means any time after that, not necessarily waiting for  
> the end of data.
>
Nothing in RFC 5321, as far as I know and understand it, allows you to  
circumvent the lock-step nature of SMTP.  That section seems to  
corroborate that.  The best you can do is suddenly become unavailable,  
with apologies to the client.  I suppose you can interpret the text to  
mean that closing the connection for other reasons is acceptable but  
as has already been explained by others this isn't called for for any  
case other than resource unavailability and will, quite reasonably,  
give you client retries.

>> So you'll just have to do what you can to keep the connection alive  
>> for the data to come in, then reject it and hopefully at that point  
>> you can resume the dialogue to finish up and ensure the sender will  
>> leave you alone.  Anything else is basically suboptimal.
>
> If we have a broken or malicious sender, it seems that the optimal  
> solution would be one with the least resources on the server side.   
> That would be not even keeping the connection alive.
>
> That works only if you don't want retries.  So it's suboptimal.   
> Maybe, as has been said before, we've simply never been bothered  
> enough about it to put it right, and make it more optimal.  But  
> right now, accepting data is basically mandatory, except when it  
> isn't, which is to say, at the beginning of the response to the DATA  
> command or after the final dot, and not inbetween.

Cheers,
Sabahattin

Date: Mon, 19 Oct 2009 23:28:08 -0700
To: IETF SMTP list <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

At 17:12 19-10-2009, Hector Santos wrote:
>Isn't it sufficient to note SMTP is a single channel client/server 
>state machine protocol?  Clients send commands, Servers respond.

I'll quote Section 1.1 as it says it better:

   "SMTP is independent of the particular transmission subsystem and
    requires only a reliable ordered data stream channel."

At 18:20 19-10-2009, David MacQuigg wrote:
>I see no technical problem on the server side with sending "552 Too 
>much mail data," then a TCP reset.  If the client keeps retrying, 
>without fixing the problem, I would block that IP address for a few hours.

The (SMTP client) retry can be from another IP address.

Regards,
-sm 

Date: Mon, 19 Oct 2009 18:20:27 -0700
From: David MacQuigg <[email-masked]>
To: [email-masked]
Subject: Re: Abort data transfer?
Sender: [email-masked]

> On 19 Oct 2009, at 16:03, David MacQuigg wrote:
>
>> Thanks to all who answered this question.
>>
>> Hector Santos wrote:
>>> SM wrote:
>>>> Hi David,
>>>> At 14:29 17-10-2009, David MacQuigg wrote:
>>>>> Is it permissible to abort data transfer before the end of data, 
>>>>> say after you have received all headers, and you know the DKIM sig 
>>>>> is bad?
>>>>>
>>>>> RFC-5321 section 3.3 says:
>>>>>
>>>>>  If the verb is initially accepted and the 354 reply issued, the DATA
>>>>>  command should fail only if the mail transaction was incomplete (for
>>>>>  example, no recipients), if resources were unavailable 
>>>>> (including, of
>>>>>  course, the server unexpectedly becoming unavailable), or if the
>>>>>  server determines that the message should be rejected for policy or
>>>>>  other reasons.
>>>>
>>>> That paragraph is about the DATA command.  You haven't received any 
>>>> headers yet.  You don't know whether the DKIM signature is bad.  
>>>> This is more of a SMTP question though.  You can do anything you 
>>>> wish for policy reasons.  As long as you adhere to the RFC 5321, 
>>>> you won't cause interoperability or hard to debug problems.  I 
>>>> suggest not aborting the data transfer to avoid such problems.
>>>
>>> +1. Aborted Data transfers can cause retransmissions. Unfortunately, 
>>> it is better to receive the entire payload and then issue a negative 
>>> response.
>>
>> Forget about DKIM.  What if there is some other reason to abort, like 
>> the payload is too large?  We cannot just continue receiving data 
>> forever.  Is there a "permissible" way within SMTP to abort during 
>> data, or should I just ignore these ambiguous requirements, and 
>> either: 1) Send a TCP reset, or 2) Let the transmitter hang.

Arnt Gulbrandsen wrote (about letting the transmitter hang):
> Some people do this in order to be as occupy the sender's resources. 
> There's a fancy name for it. You send one byte every few minutes, so 
> it takes three hours to send the whole SMTP response. I don't know 
> whether doing so really inconveniences the $#@$#@! spammers and other 
> evil creatures who initiate most SMTP connections.

I wasn't thinking of a tarpit.  That would involve keeping a connection 
on my end.  Just dropping the connection will leave the transmitter with 
an SMTP process filling memory for maybe ten minutes.  Assuming the 
transmitter is a zombie, a bunch of these hung processes might slow the 
machine enough to get the owner's attention.  He will then call his ISP 
or IT department, and that should lead to cleaning the machine and 
avoiding further infections.

If the ISP or IT department is on the ball, they will eventually figure 
out how to block port 25, and avoid tech support calls.

SM wrote:

> Arnt mentioned the RFC for for SMTP Service Extension for message size 
> declaration.

This works only if the client declares the message size, and then abides 
by its declaration.

> You can send a TCP reset to abort the connection.  The SMTP client 
> will retry delivery.  If you want to say "payload too large", you'll 
> need to send a 552 code before you get to the DATA phase.
>
> Section 4.5.3.1.7 of RFC 5321 discusses about the maximum total length 
> of a message content.  There isn't a permissible way within SMTP to 
> abort during DATA.  It's a "don't do it unless you know what you are 
> doing".

I see no technical problem on the server side with sending "552 Too much 
mail data," then a TCP reset.  If the client keeps retrying, without 
fixing the problem, I would block that IP address for a few hours.

Sabahattin Gucukoglu wrote:

> My implementations note the reason for rejection, then switch the 
> event handler for the connection to one that just reads each buffer 
> and then throws it away.  There is nothing more that can be done.  

This could leave your receiver tied up for as long as the client keeps 
spewing garbage.

> Even trying to inform the sender by writing the error to it while it 
> is sending won't necessarily work, because it might not choose to look 
> at its inbuffers until it's quite finished spewing.  And in any case 
> that's standards violation territory you're wandering into.

Which standard?  The paragraph above from Section 3.3. of 5321 seems to 
be talking about after the 354 reply is issued (past tense).  I assume 
that means any time after that, not necessarily waiting for the end of data.

> So you'll just have to do what you can to keep the connection alive 
> for the data to come in, then reject it and hopefully at that point 
> you can resume the dialogue to finish up and ensure the sender will 
> leave you alone.  Anything else is basically suboptimal.

If we have a broken or malicious sender, it seems that the optimal 
solution would be one with the least resources on the server side.  That 
would be not even keeping the connection alive.

Hector Santos wrote:
> Isn't it sufficient to note SMTP is a single channel client/server 
> state machine protocol?  Clients send commands, Servers respond.
> Server signals are not part of the SMTP client/server framework.
> Server Connection Drops are not expected by clients.
>
> In 3.8, it includes an implicit response for such unexpected drops.
>
>    SMTP clients that experience a connection close, reset, or other
>    communications failure due to circumstances not under their control
>    (in violation of the intent of this specification but sometimes
>    unavoidable) SHOULD, to maintain the robustness of the mail system,
>    treat the mail transaction as if a 451 response had been received
>    and act accordingly.
>
> Clients are expected to end the session with QUIT.
>
> But in principle, for client/server designs, the clients initiates the 
> commands, servers respond.
>

I guess section 3.8 depends on how we define "normal circumstances", 
especially with the new normal being mostly abusive.  Section 3.8 lists 
such things as "a timeout ... waiting for the client to send a command 
or data".  I guess if that section were being written today, we might 
include "an overload, waiting for the client to stop sending crap".

It looks to me like DKIM was designed with the possibility of supporting 
an abort after receiving the headers.  Why else waste space putting the 
body hash in the sig header?

-- Dave

Date: Mon, 19 Oct 2009 20:12:50 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Isn't it sufficient to note SMTP is a single channel client/server 
state machine protocol?  Clients send commands, Servers respond.
Server signals are not part of the SMTP client/server framework.
Server Connection Drops are not expected by clients.

In 3.8, it includes an implicit response for such unexpected drops.

    SMTP clients that experience a connection close, reset, or other
    communications failure due to circumstances not under their control
    (in violation of the intent of this specification but sometimes
    unavoidable) SHOULD, to maintain the robustness of the mail system,
    treat the mail transaction as if a 451 response had been received
    and act accordingly.

Clients are expected to end the session with QUIT.

But in principle, for client/server designs, the clients initiates the 
commands, servers respond.

--

SM wrote:

> 
> At 08:03 19-10-2009, David MacQuigg wrote:
>> Forget about DKIM.  What if there is some other reason to abort, like 
>> the payload is too large?  We cannot just continue receiving data 
>> forever.  Is there a "permissible" way within SMTP to abort during 
>> data, or should I just ignore these ambiguous requirements, and 
>> either: 1) Send a TCP reset, or 2) Let the transmitter hang.
> 
> Arnt mentioned the RFC for for SMTP Service Extension for message size 
> declaration.  You can send a TCP reset to abort the connection.  The 
> SMTP client will retry delivery.  If you want to say "payload too 
> large", you'll need to send a 552 code before you get to the DATA phase.
> 
> Section 4.5.3.1.7 of RFC 5321 discusses about the maximum total length 
> of a message content.  There isn't a permissible way within SMTP to 
> abort during DATA.  It's a "don't do it unless you know what you are 
> doing".
> 
> Regards,
> -sm
> 
> 

-- 
Sincerely

Hector Santos
http://www.santronics.com

Cc: [email-masked]
From: Sabahattin Gucukoglu <[email-masked]>
To: David MacQuigg <[email-masked]>
Subject: Re: Abort data transfer?
Date: Mon, 19 Oct 2009 20:06:29 +0100
Sender: [email-masked]

On 19 Oct 2009, at 16:03, David MacQuigg wrote:

> Thanks to all who answered this question.
>
> Hector Santos wrote:
>> SM wrote:
>>> Hi David,
>>> At 14:29 17-10-2009, David MacQuigg wrote:
>>>> Is it permissible to abort data transfer before the end of data,  
>>>> say after you have received all headers, and you know the DKIM  
>>>> sig is bad?
>>>>
>>>> RFC-5321 section 3.3 says:
>>>>
>>>>  If the verb is initially accepted and the 354 reply issued, the  
>>>> DATA
>>>>  command should fail only if the mail transaction was incomplete  
>>>> (for
>>>>  example, no recipients), if resources were unavailable  
>>>> (including, of
>>>>  course, the server unexpectedly becoming unavailable), or if the
>>>>  server determines that the message should be rejected for policy  
>>>> or
>>>>  other reasons.
>>>
>>> That paragraph is about the DATA command.  You haven't received  
>>> any headers yet.  You don't know whether the DKIM signature is  
>>> bad.  This is more of a SMTP question though.  You can do anything  
>>> you wish for policy reasons.  As long as you adhere to the RFC  
>>> 5321, you won't cause interoperability or hard to debug problems.   
>>> I suggest not aborting the data transfer to avoid such problems.
>>
>> +1. Aborted Data transfers can cause retransmissions.  
>> Unfortunately, it is better to receive the entire payload and then  
>> issue a negative response.
>
> Forget about DKIM.  What if there is some other reason to abort,  
> like the payload is too large?  We cannot just continue receiving  
> data forever.  Is there a "permissible" way within SMTP to abort  
> during data, or should I just ignore these ambiguous requirements,  
> and either: 1) Send a TCP reset, or 2) Let the transmitter hang.

My implementations note the reason for rejection, then switch the  
event handler for the connection to one that just reads each buffer  
and then throws it away.  There is nothing more that can be done.   
Even trying to inform the sender by writing the error to it while it  
is sending won't necessarily work, because it might not choose to look  
at its inbuffers until it's quite finished spewing.  And in any case  
that's standards violation territory you're wandering into.

So you'll just have to do what you can to keep the connection alive  
for the data to come in, then reject it and hopefully at that point  
you can resume the dialogue to finish up and ensure the sender will  
leave you alone.  Anything else is basically suboptimal.

Cheers,
Sabahattin

Date: Mon, 19 Oct 2009 11:57:55 -0700
To: David MacQuigg <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Abort data transfer?
Cc: IETF SMTP list <[email-masked]>
Sender: [email-masked]

At 08:03 19-10-2009, David MacQuigg wrote:
>Forget about DKIM.  What if there is some other reason to abort, 
>like the payload is too large?  We cannot just continue receiving 
>data forever.  Is there a "permissible" way within SMTP to abort 
>during data, or should I just ignore these ambiguous requirements, 
>and either: 1) Send a TCP reset, or 2) Let the transmitter hang.

Arnt mentioned the RFC for for SMTP Service Extension for message 
size declaration.  You can send a TCP reset to abort the 
connection.  The SMTP client will retry delivery.  If you want to say 
"payload too large", you'll need to send a 552 code before you get to 
the DATA phase.

Section 4.5.3.1.7 of RFC 5321 discusses about the maximum total 
length of a message content.  There isn't a permissible way within 
SMTP to abort during DATA.  It's a "don't do it unless you know what 
you are doing".

Regards,
-sm 

Date: Mon, 19 Oct 2009 17:35:31 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Abort data transfer?
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

David MacQuigg writes:
> Forget about DKIM.  What if there is some other reason to abort, like 
> the payload is too large?

RFC 1870 handles this special case, acceptably AFAICT.

The rest of general problem seems to be something most people choose to 
live with rather than solve.

> We cannot just continue receiving data forever.  Is there a 
> "permissible" way within SMTP to abort during data, or should I just 
> ignore these ambiguous requirements, and either: 1) Send a TCP reset,

IMO yes.

> or 2) Let the transmitter hang.

Some people do this in order to be as occupy the sender's resources. 
There's a fancy name for it. You send one byte every few minutes, so it 
takes three hours to send the whole SMTP response. I don't know whether 
doing so really inconveniences the $#@$#@! spammers and other evil 
creatures who initiate most SMTP connections.

Arnt

Date: Mon, 19 Oct 2009 08:03:25 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Thanks to all who answered this question.

Hector Santos wrote:
> SM wrote:
>> Hi David,
>> At 14:29 17-10-2009, David MacQuigg wrote:
>>> Is it permissible to abort data transfer before the end of data, say 
>>> after you have received all headers, and you know the DKIM sig is bad?
>>>
>>> RFC-5321 section 3.3 says:
>>>
>>>   If the verb is initially accepted and the 354 reply issued, the DATA
>>>   command should fail only if the mail transaction was incomplete (for
>>>   example, no recipients), if resources were unavailable (including, of
>>>   course, the server unexpectedly becoming unavailable), or if the
>>>   server determines that the message should be rejected for policy or
>>>   other reasons.
>>
>> That paragraph is about the DATA command.  You haven't received any 
>> headers yet.  You don't know whether the DKIM signature is bad.  This 
>> is more of a SMTP question though.  You can do anything you wish for 
>> policy reasons.  As long as you adhere to the RFC 5321, you won't 
>> cause interoperability or hard to debug problems.  I suggest not 
>> aborting the data transfer to avoid such problems.
>
> +1. Aborted Data transfers can cause retransmissions. Unfortunately, 
> it is better to receive the entire payload and then issue a negative 
> response.

Forget about DKIM.  What if there is some other reason to abort, like 
the payload is too large?  We cannot just continue receiving data 
forever.  Is there a "permissible" way within SMTP to abort during data, 
or should I just ignore these ambiguous requirements, and either: 1) 
Send a TCP reset, or 2) Let the transmitter hang.

-- Dave

Date: Sun, 18 Oct 2009 19:44:34 +0200
From: Alessandro Vesely <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: SMTP and DKIM/POLICY Rejection Handling
Sender: [email-masked]

Hector Santos wrote:
> [...] and section 4.2.1
> 
>   discardable
> 
>      All mail from the domain is signed with an
>      Author Domain Signature.  Furthermore, if a
>      message arrives without a valid Author Domain
>      Signature due to modification in transit,
>      submission via a path without access to a
>      signing key, or any other reason, the domain
>      encourages the recipient(s) to discard it.
> 
> [...]
>>>> Failed DKIM validation is to be treated as if no signature is present.
>>>
>>> which violates RFC 5617 DKIM=DISCARDABLE policy which would justify a 
>>> SMTP level rejection or POST SMTP message acception silent discard.
>>
>> Actually doesn't. Broken signatures tantamount missing ones, which 
>> avoids the problem of checking whether a message had actually been 
>> remailed.
>
> Can you elaborate?

The snippet quoted above from sec. 4.2.1 identifies three cases, one 
of which is remailing, but suggest to treat all of them the same way. 
Therefore no violation can result from that.

That still leaves the remailer's problem. Say Alice runs a mailing 
list that requires specific subject-prefix and footer, and Bob posts a 
contribute that contains neither of them. Now, if the contribute comes 
from a bank that signed the subject and omitted the l= tag for the 
body length, and the bank's ADSP RR is "dkim=discardable", what should 
Alice do? Her choices are to either pass it, possibly removing the 
broken signature, or reject the contribute, requiring that Bob adds 
the subject-prefix and the footer himself. In facts, after verifying 
the signature, Alice can safely send a bounce to Bob, and explain the 
issue in full detail. As for possibly discarding broken signatures 
--which would seem to just save the recipients some CPU cycles, since 
the effect is the same-- RFC 4871 sec. 4.2 says

  Signers SHOULD NOT remove any DKIM-Signature header fields from
  messages they are signing, even if they know that the signatures
  cannot be verified.

Does that mean Alice may remove the broken signature only if she does 
not in turn sign the message --i.e. she is not a signer? Hm...

>> Alas, "dkim=rejectable" is not provided for: this is 
>> consistent with the current trend of undermining SMTP's reliability.
>
> So you suggest a specific "dkim=rejectable" would apply for SMTP rejects 
> and "dkim=discardable" for post message acceptance silent discards?

Rejection is probably safer in most cases; rejecting after receiving 
the body should never be misinterpreted as a non-existing recipient, 
by a mailing list agent.

> I don't think domains declaring a "actionable" ADSP policy such as a 
> DKIM=DISCARDABLE|REJECTABLE really care how a SMTP verifier deals with 
> ADSP policy violation other than to suggest "get rid it, don't accept 
> it" - they don't want to claim any responsibility for the broken 
> DKIM/ADSP message and is providing explicit receiver handling suggestions.

That sounds reasonable. (RFC 5016 just talks about "great suspicion".) 
Do you mean they used "discardable" for "either droppable or rejectable"?

> But I agree that the RFC should correctly apply for both SMTP message 
> handling implementation methods.

The "drop" method is described in RFC 5321, sections 6.1, 6.2, 7.8, 
and 7.9, using a somewhat intricate set of inner references. That way 
RFC 5321 manages the tradeoff between reliability and safety. However, 
formal references to that handling method are not straightforward.

> [moved from above]
>> IMHO, we need an SMTP extension that explicitly binds anti-spam 
>> checks with the appropriate SMTP behavior.
> 
> I am not sure we need an SMTP extension for this, IMO, codifying new 
> 5321/5322 related standard track specifications could also resolve 
> conflictive guidance for implementators.

The advantage of an extension is to mention a handful of existing 
anti-spam checks directly, allowing future changes either via further 
extensions or rewritings of the same spec. The new SMTP spec will be a 
Full Standard, not expected to change any time soon.

Date: Sun, 18 Oct 2009 09:02:00 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: SMTP and DKIM/POLICY Rejection Handling
Sender: [email-masked]

Alessandro Vesely wrote:

> 

> Indeed, SMTP is referenced among the *Informative References* for ADSP. 
> It is left to the implementor's common sense to derive that temporary 
> errors deserve a 4xx response, and that "dkim=discardable" calls for 
> silently dropping --rather than rejecting-- a message. From a normative 
> POV, this attitude leads to a lack of specification that may 
> progressively thwart the design, implementation, or even installation of 
> new mail systems. 

+1.  RFC 5617 concerns is whats keeping us from implementing DKIM.

Nonetheless, it does include recommendations for handling messages 
such as in section 3.3:

    o  All messages from this domain are signed with an Author Domain
       Signature and are discardable, i.e., if a message arrives
       without a valid Author Domain Signature, the domain
       encourages the recipient(s) to discard it.

and section 4.2.1

   discardable

      All mail from the domain is signed with an
      Author Domain Signature.  Furthermore, if a
      message arrives without a valid Author Domain
      Signature due to modification in transit,
      submission via a path without access to a
      signing key, or any other reason, the domain
      encourages the recipient(s) to discard it.

and in section 3.3:

SMTP developers could apply this at the SMTP level for systems who do 
not want forward/pass hostile messages to recipients.

> IMHO, we need an SMTP extension that explicitly binds 
> anti-spam checks with the appropriate SMTP behavior.

I am not sure we need an SMTP extension for this, IMO, codifying new 
5321/5322 related standard track specifications could also resolve 
conflictive guidance for implementators.

>>> Failed DKIM validation is to be treated as if no signature is present.
>>
>> which violates RFC 5617 DKIM=DISCARDABLE policy which would justify a 
>> SMTP level rejection or POST SMTP message acception silent discard.
> 
> Actually doesn't. Broken signatures tantamount missing ones, which 
> avoids the problem of checking whether a message had actually been 
> remailed. 

Can you elaborate?

 > Alas, "dkim=rejectable" is not provided for: this is

> consistent with the current trend of undermining SMTP's reliability.

So you suggest a specific "dkim=rejectable" would apply for SMTP 
rejects and "dkim=discardable" for post message acceptance silent 
discards?

I don't think domains declaring a "actionable" ADSP policy such as a 
DKIM=DISCARDABLE|REJECTABLE really care how a SMTP verifier deals with 
  ADSP policy violation other than to suggest "get rid it, don't 
accept it" - they don't want to claim any responsibility for the 
broken DKIM/ADSP message and is providing explicit receiver handling 
suggestions.

But I agree that the RFC should correctly apply for both SMTP message 
handling implementation methods.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sun, 18 Oct 2009 13:13:42 +0200
From: Alessandro Vesely <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: SMTP and DKIM/POLICY Rejection Handling
Sender: [email-masked]

Hector Santos wrote:
> Dave CROCKER wrote:
>>
>> It would also violate the DKIM specification.
> 
> Not if its a DKIM POLICY based reason. I presumed he meant RFC 5617 
> DKIM=DISCARDABLE|ALL where From: and DKIM-Signature: or lack there of, 
> headers is all that is needed.

Much like similar anti-spam (in a broad sense) specs, RFC 5617 does 
not mandate a behavior. It just says

  ADSP does not provide any benefit -- nor, indeed, have any effect
  at all -- unless an external system acts upon the verdict, either
  by treating the message differently during the delivery process or
  by showing some indicator to the end recipient.  Such a system is
  out of scope for this specification.

Indeed, SMTP is referenced among the *Informative References* for 
ADSP. It is left to the implementor's common sense to derive that 
temporary errors deserve a 4xx response, and that "dkim=discardable" 
calls for silently dropping --rather than rejecting-- a message. 
 From a normative POV, this attitude leads to a lack of 
specification that may progressively thwart the design, 
implementation, or even installation of new mail systems. IMHO, we 
need an SMTP extension that explicitly binds anti-spam checks with 
the appropriate SMTP behavior.

>> Failed DKIM validation is to be treated as if no signature is present.
> 
> which violates RFC 5617 DKIM=DISCARDABLE policy which would justify a 
> SMTP level rejection or POST SMTP message acception silent discard.

Actually doesn't. Broken signatures tantamount missing ones, which 
avoids the problem of checking whether a message had actually been 
remailed. Alas, "dkim=rejectable" is not provided for: this is 
consistent with the current trend of undermining SMTP's reliability.

Date: Sat, 17 Oct 2009 22:22:33 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To:  [email-masked]
CC: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: SMTP and DKIM/POLICY Rejection Handling
Sender: [email-masked]

Dave CROCKER wrote:

> David MacQuigg wrote:
>> Failed DKIM would be a policy reason.
> 
> It would also violate the DKIM specification.

-1.

Not if its a DKIM POLICY based reason. I presumed he meant RFC 5617 
DKIM=DISCARDABLE|ALL where From: and DKIM-Signature: or lack there of, 
headers is all that is needed.

> Failed DKIM validation is to be treated as if no signature is present.

which violates RFC 5617 DKIM=DISCARDABLE policy which would justify a 
SMTP level rejection or POST SMTP message acception silent discard.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 17 Oct 2009 22:06:50 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

SM wrote:

> 
> Hi David,
> At 14:29 17-10-2009, David MacQuigg wrote:
>> Is it permissible to abort data transfer before the end of data, say 
>> after you have received all headers, and you know the DKIM sig is bad?
>>
>> RFC-5321 section 3.3 says:
>>
>>   If the verb is initially accepted and the 354 reply issued, the DATA
>>   command should fail only if the mail transaction was incomplete (for
>>   example, no recipients), if resources were unavailable (including, of
>>   course, the server unexpectedly becoming unavailable), or if the
>>   server determines that the message should be rejected for policy or
>>   other reasons.
> 
> That paragraph is about the DATA command.  You haven't received any 
> headers yet.  You don't know whether the DKIM signature is bad.  This is 
> more of a SMTP question though.  You can do anything you wish for policy 
> reasons.  As long as you adhere to the RFC 5321, you won't cause 
> interoperability or hard to debug problems.  I suggest not aborting the 
> data transfer to avoid such problems.

+1. Aborted Data transfers can cause retransmissions. Unfortunately, 
it is better to receive the entire payload and then issue a negative 
response.

However, as noted here and you offlist, if there is a DKIM policy 
rejection consideration (I presume he meant RC 5617), then he should 
be aware of the possible consequences of a SMTP level reject and 
consider a message acceptance and silent discard as permitted by RFC 
5617 and the new semantics in RFC 5321 as a reasonable reason for 
discarding SMTP accepted message deemed hostile.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 17 Oct 2009 21:34:17 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:

> 
> Is it permissible to abort data transfer before the end of data, say 
> after you have received all headers, and you know the DKIM sig is bad?
> 
> RFC-5321 section 3.3 says:
> 
>   If the verb is initially accepted and the 354 reply issued, the DATA
>   command should fail only if the mail transaction was incomplete (for
>   example, no recipients), if resources were unavailable (including, of
>   course, the server unexpectedly becoming unavailable), or if the
>   server determines that the message should be rejected for policy or
>   other reasons.
> 
> Failed DKIM would be a policy reason.

Independent of DKIM considerations,  in our experience where we had a 
condition to check the accumulated block transfer bytes received 
exceeding local policy limited and dropping the line created 
retransmissions.

It was better to wait until the DATA was completed and issue a 55x 
response in order to stop the retransmissions.

In regard to DKIM policy based rejections, I recommend:

    - Record this failed state detected at DATA
    - Accept the message,
    - Then under RFC 5617 silently discard the message.

The reason is because if the sender is a mailing list, a SMTP level 
rejects could initiate the the mailing list server to being sending 
"Last Warning" Subscription Removal notifications after a number of 
attempts are made.

In order words, any intermediary (re)signer who is not supporting RFC 
5617 intentionally or otherwise and continues to forwarded a broken 
ADSP protected domain message is subject to down link rejections at 
receivers who support DKIM/ADSP.

To mitigate this, the receiver should accept the message and discard 
it instead of creating a potential of harming the membership list.

For our Mailing list Server which will honor RFC 5617, because of this 
potential conflict, we will filter all submissions from ADSP domains.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 17 Oct 2009 15:57:24 -0700
To: David MacQuigg <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Abort data transfer?
Cc: IETF SMTP list <[email-masked]>
Sender: [email-masked]

Hi David,
At 14:29 17-10-2009, David MacQuigg wrote:
>Is it permissible to abort data transfer before the end of data, say 
>after you have received all headers, and you know the DKIM sig is bad?
>
>RFC-5321 section 3.3 says:
>
>   If the verb is initially accepted and the 354 reply issued, the DATA
>   command should fail only if the mail transaction was incomplete (for
>   example, no recipients), if resources were unavailable (including, of
>   course, the server unexpectedly becoming unavailable), or if the
>   server determines that the message should be rejected for policy or
>   other reasons.

That paragraph is about the DATA command.  You haven't received any 
headers yet.  You don't know whether the DKIM signature is bad.  This 
is more of a SMTP question though.  You can do anything you wish for 
policy reasons.  As long as you adhere to the RFC 5321, you won't 
cause interoperability or hard to debug problems.  I suggest not 
aborting the data transfer to avoid such problems.

Regards,
-sm 

Date: Sat, 17 Oct 2009 15:29:45 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: David MacQuigg <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

David MacQuigg wrote:
> Failed DKIM would be a policy reason.

It would also violate the DKIM specification.

Failed DKIM validation is to be treated as if no signature is present.

This was a carefully discussed topic for the specification and what made it into 
the final document is there because there are lots of reasons a signature can 
fail.  Hence you cannot reasonable make any assessment of the signer based the 
signature's failure.

And, yes, I'm merely trying to elaborate on what Steve A. posted.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Subject: Re: Abort data transfer?
From: Steve Atkins <[email-masked]>
Date: Sat, 17 Oct 2009 14:47:13 -0700
Cc: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Sender: [email-masked]

On Oct 17, 2009, at 2:29 PM, David MacQuigg wrote:

>
> Is it permissible to abort data transfer before the end of data, say  
> after you have received all headers, and you know the DKIM sig is bad?

>
> RFC-5321 section 3.3 says:
>
>  If the verb is initially accepted and the 354 reply issued, the DATA
>  command should fail only if the mail transaction was incomplete (for
>  example, no recipients), if resources were unavailable (including, of
>  course, the server unexpectedly becoming unavailable), or if the
>  server determines that the message should be rejected for policy or
>  other reasons.

That's talking about returning a failure response after you've received
the entire body, I believe. You can't do that before the end of data.

I think you're talking about simply dropping the connection. If you did
that most MTAs would retry the delivery.

>
> Failed DKIM would be a policy reason.
>

(As an aside, you're not likely to want to discard or reject mail due to
a failed DKIM signature, for a bunch of reasons that are more on topic
for the dkim list. Some specific types of failure might be cause for  
that,
maybe, but not ones I'd expect you to see much.)

Cheers,
   Steve

Date: Sat, 17 Oct 2009 14:29:46 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP list <[email-masked]>
Subject: Abort data transfer?
Sender: [email-masked]

Is it permissible to abort data transfer before the end of data, say 
after you have received all headers, and you know the DKIM sig is bad?

RFC-5321 section 3.3 says:

   If the verb is initially accepted and the 354 reply issued, the DATA
   command should fail only if the mail transaction was incomplete (for
   example, no recipients), if resources were unavailable (including, of
   course, the server unexpectedly becoming unavailable), or if the
   server determines that the message should be rejected for policy or
   other reasons.

Failed DKIM would be a policy reason.

-- Dave

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *