Subject: rfc2821 New Issue: Section 4.3.2, the "anywhere" codes, and related topics
From: John C Klensin <[email-masked]>
To: [email-masked]
Date: Fri, 30 Nov 2007 13:04:27 -0500
List-Id: <ietf-smtp.imc.org>

Hi.

In the process of trying to sort through the correspondence
about NOOP, I've gone back and carefully read 4.3.2.   I believe
that Hector correctly spotted a real issue that could use some
clarification even though, in looking at one particular piece of
misbehavior (the use of 421), we all missed what I now think is
the key point.  

Section 4.3.2 specifies three codes that can be used in response
to any command.  They are 421 (which we have been discussing),
500, and 501.  The 500 text reads:

# 500  For the "command line too long" case or if the command 
#     name was not recognized.  Note that producing a "command
#     not recognized" error in response to the required subset
#     of these commands is a violation of this specification.

Now, suppose that, instead of the 421 that Hector observed and
reported on, the server had responded to 
>  NOOP aa
with
>  500 5.5.2 Syntax error (command line too long)

Now, that would have avoided violating the several constraints
that we have been discussing with 421 as well as the disconnect
between the first digits of the primary and extended codes
(while I agree with Tony that a bogus string is valid, if the
server advertises ENHANCEDSTATUSCODES, I believe that a bogus
enhanced code is not, even though it is not a 2821bis problem).  

However, I contend it would still be a bug relative to the
2821[bis] spec and that the comments in the Microsoft Knowledge
Base article that Hector kindly identified reinforce that view.
That article talks about a "default filter value" for a "NOOP
SMTP command" being set at six characters (presumably
NOOP<crlf>), acknowledges that it is a problem and provides a
fix (the vendor already knows that they have a bug, although
they apparently haven't figured out the 421 problem).   

	(Incidentally, I can see nothing in that article or
	Hectors client-server exchange that leads me to believe
	that they are trying to conform to 821 and not 2821.
	They cite both specs, but respond to EHLO with a long
	list of extensions, quote extensively from 2821 and not,
	where they are different, from 821, etc.  The only
	justification for this behavior is the "defend
	themselves" provision, which they do not cite, and
	even that would be a stretch.)

Now, my reading of Section 4.5.3.1.4 (the "command line"
material in the "Size limits and minimums" section) is that one
doesn't get to use "command line too long" in an error reply
without violating the spec, i.e., that the notion that one can
figure out a maximum length, in the absence of extensions, on a
per-command basis and then apply it as a "filter" that results
in length-based rejection messages is incorrect and broken.
But, unlike the applicability of 421, I don't think that is
completely clear.

So, if others agree with my interpretation, I propose to modify
the description of 500 quoted above by adding to the end...

		Similarly, producing a "command line too long" message
		for a command line shorter than 1000 characters would
		violate the provisions of Section 4.5.3.1.4.

Hector's concerns about NOOP also raise another, more general,
issue, which is whether we are being clear enough about the
expected normal replies to commands.  I think that, if it isn't
obvious, one reads the command description, understands the
theory of reply codes (there is a cross-reference from the
introduction to Section 4.1, which lists the commands and syntax
to Section 4.2, which describes replies), and then looks in
Section 4.3.2 for a 2yz code associated with the particular
command.  That is the way it has been since 821 and, if one
believes it, then the answer to the question "what is the normal
reply to NOOP" is clearly already answered as "250".  The
commands that can return more than one 2yz codes should have the
differences between those codes described in the command
descriptions; I believe all of them do, but I'd welcome someone
else checking on that.  

But one of the implied goals for 2821bis has been making the
spec easier to read and understand without a comprehensive
understanding of how it is organized.  So, with the
understanding that this is a question and not a recommendation
(since I do not have an opinion), would it be useful to insert
text in the command descriptions for the commands that have only
a single positive reply (EHLO/HELO, DATA, RSET, NOOP, and QUIT)
that explicitly identifies the code that is expected for a
normal/successful reply?  Or would that be just clutter that
makes the document longer to no good end?

     john

Date: Fri, 30 Nov 2007 00:09:57 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Glenn Anderson <[email-masked]>
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

Glenn Anderson wrote:

> I have definitely encountered what appeared to be a proxy that returned 
> 421 and closed the connection when it encountered things it didn't like. 

I wouldn't be surprise.

> If I recall correctly, the proxy in question was returning 421 in 
> response to a MAIL FROM command with an address that didn't pass some 
> sort of validation it was doing, so this problem isn't limited to NOOP 
> commands.

Good point, let me do a quick google here....

Ok, I searched:  SMTP proxy "421 5.5.2"

Very interesting! This seems this is an issue with the Microsoft 
Internet Security and Acceleration (ISA) server. MS has this KB 
description and resolution for it:

     http://support.microsoft.com/kb/312213

It was definitely following RFC 821!!  From the article:

     Note There is no Requests for Comments (RFC) that specifies
     the default size of the NOOP command. The command is 6 bytes
     by default (N + O + O + P + CR + LF).

     For more information, see RFC 821, "Simple Mail Transfer
     Protocol" and RFC 2821, "Simple Mail Transfer Protocol." To
     see these RFCs, visit the following Web site:
     http://www.rfc-editor.org/

Go Figure!!

I hope there is some MS engineers lurking around here so they can at 
least the default setting for the length in the ISA NOOP filter. :-)

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 29 Nov 2007 22:12:58 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

Here's my take on the NOOP definition, and guidelines for last call
comments further on.

I do have a *slight* problem with the current wording in that

   If a parameter string is specified, servers SHOULD ignore it.

also means that servers "MAY pay attention to what's in the string". A
server issuing a 421 in an attempt to say "the service is no longer
available to you at this time" technically meets the spec.

IF were were going to make any change, I'd say it should be changed to:

   If a parameter string is specified, servers MUST ignore it.

Period. Full stop. Nothing more needs to be added.

But even so, a server is *still* free to issue a 421 at any time. I
could see a service doing this because it recognizes some particular
behavior of the client based on behavioral observations, which *include*
looking at the strings passed on NOOP commands, and decides that the
best thing to do when a certain gestalt is reached is to close down the
channel. It can even choose to issue a bogus random reason string if it
chooses, including one that says "the command is too long :-) ".

So would this change make any difference operationally between a server
and a client? No, not one bit.

So I withdraw even that suggestion.

Now, as the document *is* in Last Call, bringing up issues with the
document ARE allowed and encouraged. The purpose of the comments need to
be constructive and not open ended. And they should be oriented towards
the interoperability of the protocol. They should include specific
suggestions for change. These comments may even cause the document to be
reopened before being issued as an RFC; after all, the reason for having
a Last Call is to get any final issues out in the open and fixing Last
Call comments is part of the process.

Any and all issues *must* be passed through the filter of: will this
change enhance or harm the interoperability of the protocol?

	Tony Hansen
	[email-masked]

Date: Fri, 30 Nov 2007 14:55:37 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

At 8:09 pm -0500 29/11/2007, Hector Santos wrote:
>Glenn Anderson wrote:
>>
>>At 6:12 pm -0500 29/11/2007, Hector Santos wrote:
>>>So 421 is implied for real server critical shut down reasons in 
>>>2821 compliant servers across the board.
>>
>>However shutting down because it doesn't like a parameter on a NOOP 
>>command is to me a very clear violation of section 3.9 of RFC 2821 
>>(section 3.8 in 2821bis):
>>    In particular, a server that closes connections in response to
>>    commands that are not understood is in violation of this
>>    specification.
>
>
>+1, I agree.
>
>The question is, why did it happen? was the specs ambiguous?

I don't believe RFC 2821[bis] to be ambiguous on this matter.

>Was it a legacy 821 system which does not have an string for NOOP 
>who had to decide between 500 and 421, both wrong, but nonetheless 
>used in a major vendor product line, and possibly others?  Was it a 
>proxy and this major vendor had nothing to do with it?

I have definitely encountered what appeared to be a proxy that 
returned 421 and closed the connection when it encountered things it 
didn't like. If I recall correctly, the proxy in question was 
returning 421 in response to a MAIL FROM command with an address that 
didn't pass some sort of validation it was doing, so this problem 
isn't limited to NOOP commands.

>Whatever the reason, maybe 2821bis can help minimize NOOP issues in 
>future SMTP servers and also help legacy developers or proxy writers 
>fine tune their own products.

I think the culprit in this case either hadn't read section 3.9 of 
RFC 2821, or deliberately chose to ignore it, so rewording it 
wouldn't achieve anything.

Glenn.

Date: Thu, 29 Nov 2007 20:09:54 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Glenn Anderson <[email-masked]>
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

Glenn Anderson wrote:
> 
> At 6:12 pm -0500 29/11/2007, Hector Santos wrote:
>> So 421 is implied for real server critical shut down reasons in 2821 
>> compliant servers across the board.
> 
> However shutting down because it doesn't like a parameter on a NOOP 
> command is to me a very clear violation of section 3.9 of RFC 2821 
> (section 3.8 in 2821bis):
>    In particular, a server that closes connections in response to
>    commands that are not understood is in violation of this
>    specification.

+1, I agree.

The question is, why did it happen? was the specs ambiguous?  Was it a 
legacy 821 system which does not have an string for NOOP who had to 
decide between 500 and 421, both wrong, but nonetheless used in a major 
vendor product line, and possibly others?  Was it a proxy and this major 
vendor had nothing to do with it?

Whatever the reason, maybe 2821bis can help minimize NOOP issues in 
future SMTP servers and also help legacy developers or proxy writers 
fine tune their own products.

Thanks for your comments.

-- 
HLS

Date: Thu, 29 Nov 2007 20:01:30 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

John C Klensin wrote:
> Hector,
> 
> This is my last note on this issue unless you can present a
> succinct and focused argument as to why this is sufficiently
> important that we should reopen the document, risking having the
> IESG suspend the Last Call, etc.  However, since this note is
> (finally) the sort of focused discussion about the issue that I
> (and I think others) have been looking for (and for which I
> thank you), I want to respond directly to it.

John,

I would allow you to be the judge if my response is a succinct and 
focused, and I will respect this is you last note. I hope my response 
can also be the last note as well.

>> Maybe simply this change to the last sentence of the NOOP
>> command section is appropriate:
>>
>>   If a parameter string is specified, servers MUST ignore
>>   the string as a reason for a 421 response. It MAY be 
 >>   used for logging purposes only.
> 
> This is helpful in that it is a specific proposal, with a
> specific justification... and one that avoids distractions with
> the many sub-issues.
> 
> However, my reading and understanding of the specification in
> 2821bis is that there is one and only one valid "reason for a
> 421 response" and that is imminent shutdown or disconnect.
> Because one implementation has violated the specification by
> issuing a 421 when that reason does not apply does not, for me,
> justify reopening the document or inserting this text.

+1, but I suggest this is based on the premise they are not correctly 
2821, and not on a premise they are following 821 (incorrectly as well).

Look at this.  I went back to the same customer server using Microsoft 
ESMTP look at this:

220 XXXXXXXXXXXX Microsoft ESMTP MAIL Service, Version: 6.0.3790.395
9 ready at  Thu, 29 Nov 2007 16:25:47 -0800
EHLO LocalHost
250-XXXXXXXXXXXXXXXXXXX Hello [X.X.X.X]
250-TURN
250-SIZE
250-ETRN
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-8bitmime
250-BINARYMIME
250-CHUNKING
250-VRFY
250-X-EXPS GSSAPI NTLM
250-AUTH GSSAPI NTLM
250-X-LINK2STATE
250-XEXCH50
250 OK
NOOP a
250 2.0.0 OK
NOOP aa
421 5.5.2 Syntax error (command line too long)

Anything longer than 1 character produces an error. It would be 
interesting to know how current is this Microsoft ESMTP server.

Buggy?  Of course.  Did the MS programmer mis read the specs?  Who knows.

> Now, for the record (because I don't think this is a significant
> enough issue to justify reopening the document, but YMMD and you
> can try to convince others), I'd be fine with
> 
> 	If a parameter string is specified, servers MUST ignore
> 	that string. It MAY be used for logging purposes only.
> 
> I'd be fine with applying that same statement to RSET and QUIT
> if their syntax permits strings at all.  

+1. I didn't want to go that far, but if you think that helps makes it 
worth the effort. I'm all for it.

I just think that if it happened now, therefore it can potentially 
happen with other or new developers. So the question is why?  and what 
can 2821bis do to help minimize any potential ambiguity, help new 
developers and also help old developers see something interesting and 
say "hmmmmmmm, I need to fix that NOOP."

> Note that it doesn't say a think about 421, which I think 
 > is a distraction as well as a clear violation of the spec.
 > I don't think it is necessary because I think that principle
 > is clear from existing text.  But I don't think it is
 > harmful either and,

and honestly that has been my mindset all along.

Please understand that I am under the belief that a major purpose of 
2821bis was "codify" the specification to existing practice, to help 
clear ambiguious areas, with maximum effort to reduce any temptation to 
promote code changes.  And I very aware of the idea that the 2821bis is 
in last call, and the only reason I even bother was 100% because I saw 
other recent suggestions being opened up to add/make further 
"clarifications". e.g., Null Return Path Abuse issues.

So I thought it might be ok to make the rather simplistic suggestion.

In hindsight, it was over simplistic and I apologize for the thread's 
wasteful length caused by me not being more specific with the original 
post.  That was important to provide.

Thanks

--
HLS

Date: Fri, 30 Nov 2007 13:22:36 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

At 6:12 pm -0500 29/11/2007, Hector Santos wrote:
>So 421 is implied for real server critical shut down reasons in 2821 
>compliant servers across the board.

However shutting down because it doesn't like a parameter on a NOOP 
command is to me a very clear violation of section 3.9 of RFC 2821 
(section 3.8 in 2821bis):
    In particular, a server that closes connections in response to
    commands that are not understood is in violation of this
    specification.

Glenn.

Date: Thu, 29 Nov 2007 18:51:42 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>
cc: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

Hector,

This is my last note on this issue unless you can present a
succinct and focused argument as to why this is sufficiently
important that we should reopen the document, risking having the
IESG suspend the Last Call, etc.  However, since this note is
(finally) the sort of focused discussion about the issue that I
(and I think others) have been looking for (and for which I
thank you), I want to respond directly to it.

--On Thursday, 29 November, 2007 18:12 -0500 Hector Santos
<[email-masked]> wrote:

> 
> John C Klensin wrote:
> 
>> This is just my personal opinion, but if we add specific text
>> to deal with every clear violation of the existing standard
>> text, we will make 2821bis twice as long and add
>> approximately zero new information.  I don't particularly
>> object to this change but wonder where things end if we start
>> down that path.
>...

> John, I looked into this more, and I think I found how what is
> going on.
> 
> First the summary of all this:
> 
> Maybe simply this change to the last sentence of the NOOP
> command section is appropriate:
> 
>       If a parameter string is specified, servers MUST ignore
> the
>       string as a reason for a 421 response. It MAY be used for
>       logging purposes only.

This is helpful in that it is a specific proposal, with a
specific justification... and one that avoids distractions with
the many sub-issues.

However, my reading and understanding of the specification in
2821bis is that there is one and only one valid "reason for a
421 response" and that is imminent shutdown or disconnect.
Because one implementation has violated the specification by
issuing a 421 when that reason does not apply does not, for me,
justify reopening the document or inserting this text.  After
all, one could as easily say (at the risk of using a pair of
facetious examples), 

	If it is Thursday, servers MUST ignore the string as a
	reason for a 421 response.   
or
	If the mailbox name in the backward-pointing address
	contains an even number of characters, servers MUST
	ignore the string as a reason for a 421 response.

In other words, _any_ 421 response that isn't justified by the
only valid reason for a 421 response is an error.  We don't make
the specification better by listing all of the variations of
that error we can think of, or even find in the wild.

Now, for the record (because I don't think this is a significant
enough issue to justify reopening the document, but YMMD and you
can try to convince others), I'd be fine with

		If a parameter string is specified, servers MUST ignore
		that string. It MAY be used for logging purposes only.

I'd be fine with applying that same statement to RSET and QUIT
if their syntax permits strings at all.  Note that it doesn't
say a think about 421, which I think is a distraction as well as
a clear violation of the spec.  I don't think it is necessary
because I think that principle is clear from existing text.  But
I don't think it is harmful either and, if it didn't require
opening a document that had already gone into IETF Last Call,
I'd be inclined to add such text if any two or three of the
people on the list thought it was useful (Tony might, of course,
have other opinions).

> Background:
> 
> What's going on that these specific ESMTP (Extended Server)
> are still following RFC 821 where there is ABNF or expecting
> for a NOOP string:
>...

Very possible.  And I, at least, find the additional
understanding helpful.  But 2821 (and several of its
predecessors) were quite clear that, if one advertises EHLO, the
conformance expectations were higher than, and slightly
different from, the conformance expectations for 821.  So
conforming halfway to 821 and halfway to 2821 is non-conforming
behavior ... arguably not bad enough to earn some of the scorn
for the implementers that Valdis and I have expressed, but
broken enough to not require further comment nonetheless.

>...
> Please note that in RFC821, the 421 is specifically stated for
> ALL commands.  This is important when it comes to RFC 2821
> because it is now removed and implied across all commands, and
> it adds ABNF for NOOP.

Yes, but there was a specific reason for doing that, it was
discussed in DRUMS, and it wasn't just editorial.  Separating
the codes out makes it more clear that the "these can be used
anywhere" rule applies even to commands that are not explicitly
part of the spec.  

>...

> With RFC 2821 (same text in 2821bis):
> 
>     4.1.1.9 NOOP (NOOP)
> 
>       This command does not affect any parameters or
> previously entered
>       commands.  It specifies no action other than that the
> receiver send an OK reply.

And that is still, IMO, perfectly clear, regardless of what is
in the argument string.

If someone were returning "5yz syntax error in parameters", we
could be having a much more interesting discussion here.  But
returning 421 for a perceived syntax error is a clear violation
of both 2821 and 821, IMO... so clear that it doesn't deserve
any additional specific discussion.

>     4.3.2 Command-Reply Sequences
> 
>       NOOP
>          S: 250
> 
> 
> So 421 is implied for real server critical shut down reasons
> in 2821 compliant servers across the board.
> 
> For 821 compliant servers, the 421 is appropiate for an
> unexpected NOOP string.  No bug.

I believe that is a misreading and that this is a bug even under
821.  421 has a specific meaning in the "theory of reply codes".
The first digit indicates that it is a transient error, which an
unexpected string certainly is not (something that someone else
pointed out much earlier in this discussion).   Using 421, or
4yz for any value of yz, in some other way violates the intent
of 821.  Nothing in 821 says that, because a code can be
returned from a particular command, one can use that code to
report any condition one likes.  That would be madness.

>...
> This will make it backward compatible and helpful to legacy
> programmers of 821 base logic that might want to revisit their
> code, and it will also make it very clear to future
> programmers.

It is useful to "legacy programmers of 821 base logic" only if
those programmers had somehow reached the conclusion that the
inclusion of 421 as an explicit possible reply code to NOOP
justified using it to report a syntax error rather than a
pending shutdown.  I think that reading would be bizarre, so
bizarre that I would consider metaphors that question the
programmer's basic competence or state of mind to be entirely
appropriate.

Just my opinion, of course.

     john

Date: Thu, 29 Nov 2007 18:12:20 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

John C Klensin wrote:

> This is just my personal opinion, but if we add specific text to
> deal with every clear violation of the existing standard text,
> we will make 2821bis twice as long and add approximately zero
> new information.  I don't particularly object to this change but
> wonder where things end if we start down that path.
> 
> And, of course, if the "Microsoft ESMTP server" issued the 421
> because it intended to shut the connection down, then not only
> is the response valid but I think we would prefer it to
> "ignoring the command" by sending a 250 and suppressing that
> information.

John, I looked into this more, and I think I found how what is going on.

First the summary of all this:

Maybe simply this change to the last sentence of the NOOP command 
section is appropriate:

      If a parameter string is specified, servers MUST ignore the
      string as a reason for a 421 response. It MAY be used for
      logging purposes only.

Background:

What's going on that these specific ESMTP (Extended Server) are still 
following RFC 821 where there is ABNF or expecting for a NOOP string:

With RFC 821:

    4.1.  SMTP COMMANDS

          NOOP (NOOP)

             This command does not affect any parameters or previously
             entered commands.  It specifies no action other than that
             the receiver send an OK reply.

             This command has no effect on any of the reverse-path
             buffer, the forward-path buffer, or the mail data buffer.

    4.3.  SEQUENCING OF COMMANDS AND REPLIES

          NOOP
              S: 250
              E: 500, 421

Please note that in RFC821, the 421 is specifically stated for ALL 
commands.  This is important when it comes to RFC 2821 because it is now 
removed and implied across all commands, and it adds ABNF for NOOP.

With RFC 2821 (same text in 2821bis):

    4.1.1.9 NOOP (NOOP)

      This command does not affect any parameters or previously entered
      commands.  It specifies no action other than that the receiver send
      an OK reply.

      This command has no effect on the reverse-path buffer, the forward-
      path buffer, or the mail data buffer and may be issued at any time.
      If a parameter string is specified, servers SHOULD ignore it.

      Syntax:
         "NOOP" [ SP String ] CRLF

    4.3.2 Command-Reply Sequences

      NOOP
         S: 250

So 421 is implied for real server critical shut down reasons in 2821 
compliant servers across the board.

For 821 compliant servers, the 421 is appropiate for an unexpected NOOP 
string.  No bug.

So in summary, you might want to consider these history real 
possibilities where there some innocent "bugs" to help codify 2821 
expected behaviors, especially when the server indicated it is an 
EXTENDED SMTP (ESMTP) server. :-)

So maybe simply this change to the last sentence of the NOOP command 
section is appropriate:

      If a parameter string is specified, servers MUST ignore the
      string as a reason for a 421 response. It MAY be used for
      logging purposes only.

This will make it backward compatible and helpful to legacy programmers 
of 821 base logic that might want to revisit their code, and it will 
also make it very clear to future programmers.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 29 Nov 2007 17:12:13 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

[email-masked] wrote:

> On Thu, 29 Nov 2007 15:29:44 EST, Hector Santos said:
> 
>>     This command does not affect any parameters or previously entered
>>     commands.  It specifies no action other than that the receiver send
>>     an OK reply.  Server SHOULD ignore the NOOP command by issueing a
>>     250 response.
> 
> I wish I had a dollar for every time I've seen programmers and designers read
> a SHOULD in an RFC, gone back and looked at what RFC2119 actually says:
> 
> 3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
>    may exist valid reasons in particular circumstances to ignore a
>    particular item, but the full implications must be understood and
>    carefully weighed before choosing a different course.
> 
> and then proceeded to do otherwise because they were righteous in their
> faith that they had valid reasons and understood the full implications.
> 
> Heck, I'd take two bits and a nickel for each time I've seen somebody treat
> a MUST as a MAY.....
> 
> If you want to make sure you get a 250 back, you should change that SHOULD
> to a MUST - but I totally fail to see why you think that *any* amount of
> explanatory verbiage will help with programmers who think that syntax errors
> should get a 421..  I think *those* guys are pretty well summed like this:
> 
> "What we have heah, is a failure to communicate. There are some
> programmers you just.. cahn't.. reach."

Valdis,

I will try one more time.

If the specs say the string should be ignored, it can't be a condition 
to negatively reject the NOOP.

Therefore, in lieu of a real system critical issue where a 421 may be 
appropiate, there is only one other choice - 250.

But I am not saying it should be a MUST or anthing like that, and I 
think it reall inappropriate to presume or insinuate or begin to just 
'throw it out there' that I or "programmers" in general fail to 
understand RFC 2119 or specifically the differences of SHOULD/MUST/MAY.

I really unfortunately when things go off in tangents that are 
completely out of whack and has nothing to do what is being said.  In a 
way, it kinda sounds like you are the pipe yourself. :-)

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: New Issue: NOOP clarification
From: [email-masked]
Date: Thu, 29 Nov 2007 16:54:22 -0500
Sender: [email-masked]

--==_Exmh_1196373262_2822P
Content-Type: text/plain; charset=us-ascii

On Thu, 29 Nov 2007 15:29:44 EST, Hector Santos said:

>     This command does not affect any parameters or previously entered
>     commands.  It specifies no action other than that the receiver send
>     an OK reply.  Server SHOULD ignore the NOOP command by issueing a
>     250 response.

I wish I had a dollar for every time I've seen programmers and designers read
a SHOULD in an RFC, gone back and looked at what RFC2119 actually says:

3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

and then proceeded to do otherwise because they were righteous in their
faith that they had valid reasons and understood the full implications.

Heck, I'd take two bits and a nickel for each time I've seen somebody treat
a MUST as a MAY.....

If you want to make sure you get a 250 back, you should change that SHOULD
to a MUST - but I totally fail to see why you think that *any* amount of
explanatory verbiage will help with programmers who think that syntax errors
should get a 421..  I think *those* guys are pretty well summed like this:

"What we have heah, is a failure to communicate. There are some
programmers you just.. cahn't.. reach."

--==_Exmh_1196373262_2822P
Content-Type: application/pgp-signature

--==_Exmh_1196373262_2822P--

Date: Thu, 29 Nov 2007 15:29:44 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

[email-masked] wrote:

> One has to be careful here - adding "clarifications" just adds things for
> the crack programmers to totally misinterpret.
> 
> Seriously - write up your proposed text, and then *carefully* read it while
> thinking "How can somebody who thinks syntax errors are 4xx screw this one
> up, when they intentionally *try* to do so?"

I think the editor can do a "NOOP" document search and see there it is 
discussed, and pull out the semantics comparing NOOP to RESET and also 
in the reply code section, and put some of that into the NOOP section 
itself.  Its not hard.

I think the statement I added in the OP, was exactly that:

    This command does not affect any parameters or previously entered
    commands.  It specifies no action other than that the receiver send
    an OK reply.  Server SHOULD ignore the NOOP command by issueing a
    250 response.

and it doesn't go into details, or maybe explaining why or when other 
codes my be appropiate.  If more insights are needed, fine. Obviously 
the "top level" reason, for the human, help explain that it was a BUG, 
but I think it would be useful telling people that what is expected will 
definitely help fix this and other potential bugs and also help codify 
expected state machine practices.

I hope that makes sense. It does to me.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 29 Nov 2007 15:18:38 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

[email-masked] wrote:
> On Thu, 29 Nov 2007 13:38:50 EST, Hector Santos said:
> 
>>     For example, a command such as NOOP, whose successful execution does
>>     not offer the SMTP client any new information, will return a 250
>>     reply.  The reply is 502 when the command requests an unimplemented
>>     non-site-specific action.  A refinement of that is the 504 reply for
>>     a command that is implemented, but that requests an unimplemented
>>     parameter.
> 
> Note that a *valid* 421 due to a server shutdown or similar incident *does*
> offer the SMTP client new information - the fact that a shutdown or temporary
> issue has commenced.  So the "will return a 250" only matters if the
> server *doesn't* have anything new to say - but it's totally free to signal
> a new condition via a 421 or similar.

Sorry, it is clear my mindset is little different and probably explains 
why i have to waste my time and every else's time with details that 
really shouldn't apply here.  Atleast to me it doesn't.

I didn't ask for a SPECIFICATION CHANGE as John think I did.  I thought 
I was inline with the level of simply, non-protocol changing semantical 
and codificaton being done here for 2821bis  If not, maybe I am wrong.

I didn't think I have to go DEEP into explaining why because one can 
just look at the current statement and say "hhmmmm, maybe there might be 
some ambiguity here if the reader hasn't caught on yet, that NOOP pretty 
much requires a 250 or atleast it SHOULD be 250 as it futher indicated 
in section 4.2.1. Reply Code Severities and Theory.

    ...
    For example, a command such as NOOP, whose successful execution does
    not offer the SMTP client any new information, will return a 250
    reply.  The reply is 502 when the command requests an unimplemented
    non-site-specific action.  A refinement of that is the 504 reply for
    a command that is implemented, but that requests an unimplemented
    parameter.
    ...

I didn't think it mattered if it was proxy or bug or that the line was 
too long or too short.

Even if it was appropriate to issue 421, regarding of the string, the 
issue was not a CLIENT compatibility problem.  The client did what it 
suppose to do.  It issued QUIT to end the session.

It had nothing to do with any of that.

But I am sorry if I did need to justify it by providing the details or 
it will help you.  I do feel that you will agree that some clarification 
should be made in the NOOP section, but buried in other sections or 
maybe not.

But I leave it to the editor to decide what words he want if he wants it 
at all.

I didn't think this would be turned into a lengthy debate.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: New Issue: NOOP clarification
From: [email-masked]
Date: Thu, 29 Nov 2007 14:51:39 -0500
Sender: [email-masked]

--==_Exmh_1196365899_2822P
Content-Type: text/plain; charset=us-ascii

On Thu, 29 Nov 2007 13:38:50 EST, Hector Santos said:

>     For example, a command such as NOOP, whose successful execution does
>     not offer the SMTP client any new information, will return a 250
>     reply.  The reply is 502 when the command requests an unimplemented
>     non-site-specific action.  A refinement of that is the 504 reply for
>     a command that is implemented, but that requests an unimplemented
>     parameter.

Note that a *valid* 421 due to a server shutdown or similar incident *does*
offer the SMTP client new information - the fact that a shutdown or temporary
issue has commenced.  So the "will return a 250" only matters if the
server *doesn't* have anything new to say - but it's totally free to signal
a new condition via a 421 or similar.

--==_Exmh_1196365899_2822P
Content-Type: application/pgp-signature

--==_Exmh_1196365899_2822P--

Date: Thu, 29 Nov 2007 14:26:28 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

I understand Valdis.  That is why I leave it to the "Documenation" team 
to do the write up, if they choose it to be worthy. :-)  I personally 
think this suggestion is as worthly as most other suggested it in 
cleaning up nits or codifying practice for 2821bis.  If not, then once 
again, I'm wrong in the eyes of those here. :-)

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

[email-masked] wrote:
> On Thu, 29 Nov 2007 13:38:50 EST, Hector Santos said:
>>   S: 421 5.5.2 Syntax error (command line too long)
> 
> This is a *clear* example of "put the crack pipe down and move away
> slowly, with your hands in sight" programming.  Under what *conceivable*
> circumstances would a "syntax error" qualify for a 4xx return code, which
> would imply that re-trying the exact same command later would work?
> 
>> So it doesn't matter any of the above or if a proxy or not.  Obviously 
>> some SMTP servers are not following the recommmendations.  Obviously 
>> some are buggy. It doen't matter of who was right or wrong.
>>
>> A clarification should be adjusted into NOOP statement to help future 
>> and current implementations update their codes.
> 
> One has to be careful here - adding "clarifications" just adds things for
> the crack programmers to totally misinterpret.
> 
> Seriously - write up your proposed text, and then *carefully* read it while
> thinking "How can somebody who thinks syntax errors are 4xx screw this one
> up, when they intentionally *try* to do so?"

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: New Issue: NOOP clarification
From: [email-masked]
Date: Thu, 29 Nov 2007 14:14:10 -0500
Sender: [email-masked]

--==_Exmh_1196363650_2822P
Content-Type: text/plain; charset=us-ascii

On Thu, 29 Nov 2007 13:38:50 EST, Hector Santos said:
>   S: 421 5.5.2 Syntax error (command line too long)

This is a *clear* example of "put the crack pipe down and move away
slowly, with your hands in sight" programming.  Under what *conceivable*
circumstances would a "syntax error" qualify for a 4xx return code, which
would imply that re-trying the exact same command later would work?

> So it doesn't matter any of the above or if a proxy or not.  Obviously 
> some SMTP servers are not following the recommmendations.  Obviously 
> some are buggy. It doen't matter of who was right or wrong.
> 
> A clarification should be adjusted into NOOP statement to help future 
> and current implementations update their codes.

One has to be careful here - adding "clarifications" just adds things for
the crack programmers to totally misinterpret.

Seriously - write up your proposed text, and then *carefully* read it while
thinking "How can somebody who thinks syntax errors are 4xx screw this one
up, when they intentionally *try* to do so?"

--==_Exmh_1196363650_2822P
Content-Type: application/pgp-signature

--==_Exmh_1196363650_2822P--

Date: Thu, 29 Nov 2007 13:38:50 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

[email-masked] wrote:
> On Wed, 28 Nov 2007 17:39:13 EST, Hector Santos said:
> 
>> 4.1.1.9 NOOP (NOOP)
> 
>> I just came across a transaction where a Microsoft ESMTP server rejected 
>>   it with a 421 response.
> 
> There's 2 cases here:
> 
> 1) It's trying to tell you "421 It's nap time here".  In this case, it's a
> totally legitimate reply - and I've even seen systems that keep a hot
> connection open to recent/frequent destinations that intentionally send
> keep-alive NOOPs specifically to reap 421's and unexpected closes from the
> other end.
> 
> 2) It's a bug, and somebody needs to learn to put down the crack pipe before
> coding.
> 
> No change to the RFC is needed for the first, as it's working as designed.
> No change is needed for the second, because we can't legislate entry into
> substance abuse programs in an RFC.

It is possible to have semantics that are independent of whether's a 
bug, proxy or not?   Just like the rest of the document.

I caught it during a teleconference where we exchanging emails, so I 
didn't have time to look at any details other than note that it SHOULD 
had not happen, so bug or no bug, I felt, at a minimum, a small
clarification needs to be added to hhelp clarify and codify existence 
practice.

The exchange was this:

   C: NOOP WCSAP v2.09 Wildcat! Sender Authentication Protocol 
http://www.santronics.com
   S: 421 5.5.2 Syntax error (command line too long)

Now here are the facts:

a) it wasn't too long,
b) it didn't ignore the STRING per RFC.
c) NOOP is a requirement
d) The RFC recommended response is 250:

    For example, a command such as NOOP, whose successful execution does
    not offer the SMTP client any new information, will return a 250
    reply.  The reply is 502 when the command requests an unimplemented
    non-site-specific action.  A refinement of that is the 504 reply for
    a command that is implemented, but that requests an unimplemented
    parameter.

e) 421 was not appropriate for the command nor its "string reason"

      421 <domain> Service not available, closing transmission channel
         (This may be a reply to any command if the service knows it
          must shut down)

and so and so on,

So it doesn't matter any of the above or if a proxy or not.  Obviously 
some SMTP servers are not following the recommmendations.  Obviously 
some are buggy. It doen't matter of who was right or wrong.

A clarification should be adjusted into NOOP statement to help future 
and current implementations update their codes.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: New Issue: NOOP clarification
From: [email-masked]
Date: Thu, 29 Nov 2007 12:31:40 -0500
Sender: [email-masked]

--==_Exmh_1196357500_2822P
Content-Type: text/plain; charset=us-ascii

On Wed, 28 Nov 2007 17:39:13 EST, Hector Santos said:

> 4.1.1.9 NOOP (NOOP)

> I just came across a transaction where a Microsoft ESMTP server rejected 
>   it with a 421 response.

There's 2 cases here:

1) It's trying to tell you "421 It's nap time here".  In this case, it's a
totally legitimate reply - and I've even seen systems that keep a hot
connection open to recent/frequent destinations that intentionally send
keep-alive NOOPs specifically to reap 421's and unexpected closes from the
other end.

2) It's a bug, and somebody needs to learn to put down the crack pipe before
coding.

No change to the RFC is needed for the first, as it's working as designed.
No change is needed for the second, because we can't legislate entry into
substance abuse programs in an RFC.

--==_Exmh_1196357500_2822P
Content-Type: application/pgp-signature

--==_Exmh_1196357500_2822P--

Date: Thu, 29 Nov 2007 00:42:34 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

SM,

Maybe it was, maybe it was not. I was surprise too.

But it is what it showed.  The welcome signature said Microsoft ESMTP 
Server and the vendor I was dealing with at the time, when we saw it, 
confirmed it was their MS SMTP server. I did't bother to ask about PROXY 
setups, etc, or go into why it happen because it wasn't the purpose of 
conversation.

Sure, I was surprise by it too.   But even if was a AVS related proxy, 
it was still a violation, proxy or not :-)

So I merely focused on the semantics currently in the specs - it didn't 
ignore the string nor the NOOP command itself.

If the clarification "nit" is not deem worthy of anyone's time, then 
never mind.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

SM wrote:
> Hi Hector,
> At 14:39 28-11-2007, Hector Santos wrote:
>> I just came across a transaction where a Microsoft ESMTP server 
>> rejected  it with a 421 response.
> 
> Are you sure it's a Microsoft ESTMP server and not some application 
> proxy filtering SMTP commands?
> 
> Regards,
> -sm
> 
> 

Date: Wed, 28 Nov 2007 20:41:56 -0800
To: Hector Santos <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

Hi Hector,
At 14:39 28-11-2007, Hector Santos wrote:
>I just came across a transaction where a Microsoft ESMTP server 
>rejected  it with a 421 response.

Are you sure it's a Microsoft ESTMP server and not some application 
proxy filtering SMTP commands?

Regards,
-sm 

Date: Wed, 28 Nov 2007 19:37:04 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: New Issue: NOOP clarification
Sender: [email-masked]

--On Wednesday, 28 November, 2007 17:39 -0500 Hector Santos
<[email-masked]> wrote:

> 
> RFC 2821 (RFC 2821bis) says for the NOOP:
> 
> 4.1.1.9 NOOP (NOOP)
> 
>     This command does not affect any parameters or previously
> entered
>     commands.  It specifies no action other than that the
> receiver send
>     an OK reply.
> 
>     This command has no effect on the reverse-path buffer, the
> forward-
>     path buffer, or the mail data buffer and may be issued at
> any time.
>     If a parameter string is specified, servers SHOULD ignore
> it.
> 
>     Syntax:
>        "NOOP" [ SP String ] CRLF
> 
> I just came across a transaction where a Microsoft ESMTP
> server rejected it with a 421 response.

Unless 421 was used in the sense of "service shutting down,
closing connection" (in which case the response and code are
perfectly valid -- see the introductory material to Section
4.3.2), that is what is technically known as a "bug".  Less
technically, it might be described as "blatantly
non-conforming". 

The two cases can presumably be distinguished by issuing another
command.  If the server responds by closing the connection (or
has closed it already), the 421 was valid.  If it sends any
positive or negative response other than "503 bad sequence of
commands" or another 421, I think it would indicate a bug.

Note that support for NOOP is required (Section 4.5.1).

> The paragraph should include semantics that the NOOP command
> itself SHOULD be ignored, not just the string SHOULD be
> ignored.  Maybe:
> 
>     This command does not affect any parameters or previously
> entered
>     commands.  It specifies no action other than that the
> receiver send
>     an OK reply.  Server SHOULD ignore the NOOP command by
> issueing a
>     250 response.

This is just my personal opinion, but if we add specific text to
deal with every clear violation of the existing standard text,
we will make 2821bis twice as long and add approximately zero
new information.  I don't particularly object to this change but
wonder where things end if we start down that path.

And, of course, if the "Microsoft ESMTP server" issued the 421
because it intended to shut the connection down, then not only
is the response valid but I think we would prefer it to
"ignoring the command" by sending a 250 and suppressing that
information.

    john

Date: Wed, 28 Nov 2007 17:39:13 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: New Issue: NOOP clarification
Sender: [email-masked]

RFC 2821 (RFC 2821bis) says for the NOOP:

4.1.1.9 NOOP (NOOP)

    This command does not affect any parameters or previously entered
    commands.  It specifies no action other than that the receiver send
    an OK reply.

    This command has no effect on the reverse-path buffer, the forward-
    path buffer, or the mail data buffer and may be issued at any time.
    If a parameter string is specified, servers SHOULD ignore it.

    Syntax:
       "NOOP" [ SP String ] CRLF

I just came across a transaction where a Microsoft ESMTP server rejected 
  it with a 421 response.

The paragraph should include semantics that the NOOP command itself 
SHOULD be ignored, not just the string SHOULD be ignored.  Maybe:

    This command does not affect any parameters or previously entered
    commands.  It specifies no action other than that the receiver send
    an OK reply.  Server SHOULD ignore the NOOP command by issueing a
    250 response.

--
HLS

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject:  Re: Possible ambiguity in SMTP RFC2821 - your opinon please
Date:  Tue, 27 Nov 2007 18:04:53 +0100
Organization:  <http://purl.net/xyzzy>
Lines: 21
Reply-To: "Frank Ellermann" <[email-masked]>
Sender: [email-masked]

<[email-masked]> wrote:

> http://xn--fdbk5d8ap9b8a8d.xn--deba0ad/   is the actual target.
=20
> I see a dot between the 8d and the xn--, and my Firefox even
> renders it with a '.' (see attached .png).

Same here with IE6 on a box where I've not yet installed FF.  But
the zone cut is the TLD:

`nslookup -q=3Dns xn--fdbk5d8ap9b8a8d.xn--deba0ad`
| [...]
| xn--deba0ad
|         primary name server =3D [...]
|         [...]

RFC  2142 says "only the organization's top level domain name are
required to be valid."  For your example that would be xn--deba0ad,
resulting in a likely unusable abuse@xn--deba0ad address.

 Frank

Date: Tue, 27 Nov 2007 13:10:09 +0000
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: Chris Wright <[email-masked]>, [email-masked]
Subject: RE: Possible ambiguity in SMTP RFC2821 - your opinon please
Sender: [email-masked]

On Mon, 26 Nov 2007, John C Klensin wrote:
>
> So, while 2821bis clearly permits single-component (TLD-only)
> domains, my guess is that, to take advantage of it, you will
> need to implement it and then start a campaign to disseminate
> clues and allow at least a half-dozen years for it to be
> effective.

Even the people who try to set up TLDs with MXs mostly screw it up.
http://fanf.livejournal.com/72486.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BAILEY: NORTHWESTERLY 4 OR 5 BACKING SOUTHERLY 5 TO 7. ROUGH OR VERY ROUGH.
OCCASIONAL RAIN. MODERATE OR GOOD, OCCASIONALLY POOR.

To: Frank Ellermann <[email-masked]>
Cc: [email-masked]
Subject: Re: Possible ambiguity in SMTP RFC2821 - your opinon please
From: [email-masked]
Date: Tue, 27 Nov 2007 03:29:28 -0500
Sender: [email-masked]

--==_Exmh_1196152168_2790P
Content-Type: multipart/mixed ;
	boundary="==_Exmh_1196152057_27900"

This is a multipart MIME message.

--==_Exmh_1196152057_27900
Content-Type: text/plain; charset=us-ascii

On Tue, 27 Nov 2007 08:03:35 +0100, Frank Ellermann said:
> 
> <[email-masked]> wrote:
> 
> > I'd say that in *practical* terms you're just *looking* for
> > trouble if you publish an e-mail address that doesnt have a
> > '.' anywhere in it.
> 
> BTW, an interpretation of RFC 2142 about abuse@ addresses for
> any given FQDN takes the zone cut adding abuse@.  For the IDN
> test domains that ends up with no-dot abuse@ addresses, doesn't
> it ?  Should that go into an erratum for RFC 2142 ?  Of course
> RFC 2142 couldn't foresee what RFC 2821 says, but it's still
> an interesting detail for folks trying to implement RFC 2142,
> or even trying to promote it - the abuse@ part is important,
> the other mailbox names in RFC 2142 are less convincing.
> 
>  Frank
> 
http://idn.icann.org/ says:

The example test names:

Picking the Hebrew one at random:

http://xn--fdbk5d8ap9b8a8d.xn--deba0ad/   is the actual target.

I see a dot between the 8d and the xn--, and my Firefox even renders it
with a '.' (see attached .png).  The other languages *also* both have
a '.' between the two parts of the URL *and* render a dot in the middle
of the non-ascii string.  

--==_Exmh_1196152057_27900
Content-Type: image/png ; name="idn-has-dot.png"
Content-Description: idn-has-dot.png
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="idn-has-dot.png"

--==_Exmh_1196152057_27900--

--==_Exmh_1196152168_2790P
Content-Type: application/pgp-signature

--==_Exmh_1196152168_2790P--

To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject:  Re: Possible ambiguity in SMTP RFC2821 - your opinon please
Date:  Tue, 27 Nov 2007 08:03:35 +0100
Lines: 16
Reply-To: "Frank Ellermann" <[email-masked]>
Sender: [email-masked]

<[email-masked]> wrote:

> I'd say that in *practical* terms you're just *looking* for
> trouble if you publish an e-mail address that doesnt have a
> '.' anywhere in it.

BTW, an interpretation of RFC 2142 about abuse@ addresses for
any given FQDN takes the zone cut adding abuse@.  For the IDN
test domains that ends up with no-dot abuse@ addresses, doesn't
it ?  Should that go into an erratum for RFC 2142 ?  Of course
RFC 2142 couldn't foresee what RFC 2821 says, but it's still
an interesting detail for folks trying to implement RFC 2142,
or even trying to promote it - the abuse@ part is important,
the other mailbox names in RFC 2142 are less convincing.

 Frank

Date: Mon, 26 Nov 2007 22:06:58 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: FWD: Last Call: draft-klensin-rfc2821bis (Simple Mail Transfer Protocol) to Draft Standard
Sender: [email-masked]

And thank you all for the many comments everyone's provided to make this
a much better document.

	Tony

John C Klensin wrote:
> FYI...
> 
> 
> ---------- Forwarded Message ----------
> Date: Monday, 26 November, 2007 20:32 -0500
> From: The IESG <[email-masked]>
> To: IETF-Announce <[email-masked]>
> Subject: Last Call: draft-klensin-rfc2821bis (Simple Mail
> Transfer  Protocol) to Draft Standard
> 
> The IESG has received a request from an individual submitter to
> consider the following document:
> 
> - 'Simple Mail Transfer Protocol '
>    <draft-klensin-rfc2821bis-06.txt> as a Draft Standard
> 
> The IESG plans to make a decision in the next few weeks, and
> solicits final comments on this action.  Please send substantive
> comments to the [email-masked] mailing lists by 2007-12-24.
> Exceptionally,  comments may be sent to [email-masked] instead.
> In either case, please  retain the beginning of the Subject line
> to allow automated sorting.
> 
> The file can be obtained via
> http://www.ietf.org/internet-drafts/draft-klensin-rfc2821bis-06.
> txt
> 
> Implementation Report can be accessed at
> http://www.ietf.org/IESG/implementation.html
> 
> 
> IESG discussion can be tracked via
> https://datatracker.ietf.org/public/pidtracker.cgi?command=view_
> id&dTag=13354&rfc_flag=0
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf-announce
> 
> ---------- End Forwarded Message ----------
> 
> 
> 
> 

To: John C Klensin <[email-masked]>
Cc: Chris Wright <[email-masked]>, [email-masked]
Subject: Re: Possible ambiguity in SMTP RFC2821 - your opinon please
From: [email-masked]
Date: Mon, 26 Nov 2007 21:15:08 -0500
Sender: [email-masked]

--==_Exmh_1196129708_2838P
Content-Type: text/plain; charset=us-ascii

On Mon, 26 Nov 2007 10:32:57 EST, John C Klensin said:

> In practical terms, it would probably be better to not count on
> this working, or at least to provide an alternate address
> ([email-masked] would be traditional) for people to use when their
> MUAs balk.

"[email-masked]? That can't be right, that doesn't have a .com on the end" :)

Given that level of clue out there, I'd say that in *practical* terms
you're just *looking* for trouble if you publish an e-mail address that
doesnt have a '.' anywhere in it.

This is the sort of corner case that makes me wish that RFC2119 had included
a "Paging Randy Bush.." qualifier somewhere between 'SHOULD NOT' and 'MUST NOT'.

--==_Exmh_1196129708_2838P
Content-Type: application/pgp-signature

--==_Exmh_1196129708_2838P--

Date: Mon, 26 Nov 2007 20:38:47 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
Subject: FWD: Last Call: draft-klensin-rfc2821bis (Simple Mail Transfer  Protocol) to Draft Standard
Sender: [email-masked]

FYI...

---------- Forwarded Message ----------
Date: Monday, 26 November, 2007 20:32 -0500
From: The IESG <[email-masked]>
To: IETF-Announce <[email-masked]>
Subject: Last Call: draft-klensin-rfc2821bis (Simple Mail
Transfer  Protocol) to Draft Standard

The IESG has received a request from an individual submitter to
consider the following document:

- 'Simple Mail Transfer Protocol '
   <draft-klensin-rfc2821bis-06.txt> as a Draft Standard

The IESG plans to make a decision in the next few weeks, and
solicits final comments on this action.  Please send substantive
comments to the [email-masked] mailing lists by 2007-12-24.
Exceptionally,  comments may be sent to [email-masked] instead.
In either case, please  retain the beginning of the Subject line
to allow automated sorting.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-klensin-rfc2821bis-06.
txt

Implementation Report can be accessed at
http://www.ietf.org/IESG/implementation.html

IESG discussion can be tracked via
https://datatracker.ietf.org/public/pidtracker.cgi?command=view_
id&dTag=13354&rfc_flag=0

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

---------- End Forwarded Message ----------

Date: Mon, 26 Nov 2007 10:32:57 -0500
From: John C Klensin <[email-masked]>
To: Chris Wright <[email-masked]>
cc: [email-masked]
Subject: RE: Possible ambiguity in SMTP RFC2821 - your opinon please
Sender: [email-masked]

--On Tuesday, 27 November, 2007 02:13 +1100 Chris Wright
<[email-masked]> wrote:

> John,
> 
> Thank you for taking the time to reply, I have been
> experimenting and discovered that even most mail clients wont
> accept chris@au (for example) as a valid email address,
> outlook certainly wont, neither will hotmail or yahoo so this
> in itself is a problem that would almost be impossible to
> tackle. Even if the MTA software did router the mail, it would
> be useless with a large majority of email users being unable to
> send email to the address (and presumably unable to reply).
> 
> Oh well, at least I have my answer.
> 
> Once again thanks for replying.

Yes, that MUA problem would be hard to overcome.  We still have
MUAs and web sites that reject addresses in the museum, info,
aero, etc., TLDs, presumably on the grounds that we all know
that TLDs can only be two or three characters long unless they
are "arpa".   That constraint was never built into the DNS
protocols, while the "two or more labels" one is.  Many, perhaps
most of those clients have been fixed in the last half-dozen
years, but the number of bad actors has certainly not reached
zero.

So, while 2821bis clearly permits single-component (TLD-only)
domains, my guess is that, to take advantage of it, you will
need to implement it and then start a campaign to disseminate
clues and allow at least a half-dozen years for it to be
effective.

Worse, in these suspicious days, I'd expect spam detectors and
the often-lame SMTP implementations on firewalls to treat
messages containing single-component domains in
backward-pointing addresses as suspicious, perhaps suspicious
enough to drop or classify into "junk" folders.  My impression
has been that it is often harder to get such things updated than
to obtain improvements in actual MUAs.  So, even if people could
send mail to you, you might not be able to successfully reach
them with a reply.

In practical terms, it would probably be better to not count on
this working, or at least to provide an alternate address
([email-masked] would be traditional) for people to use when their
MUAs balk.

good luck,

    john

From: [email-masked]
Cc: Frank Ellermann <[email-masked]>, [email-masked]
Date: Sun, 25 Nov 2007 16:41:33 -0800 (PST)
Subject: Re: 2821bis and address rewriting
To: Hector Santos <[email-masked]>
Sender: [email-masked]

> Frank Ellermann wrote:
> > <[email-masked]> wrote:
> >
> >> you really *don't* want to have to poke around in the 822 headers
> >> to find a 821-style return address (especially since if you *do*
> >> that sort of grovelling, you will almost certainly Do The Wrong
> >> Thing if you ever get a bounce with an 822-From *other* than
> >> Mailer-Daemon).
> >
> > +1  Or as RFC 3834 put it:
> >
> >  If the response is to be generated after delivery, and there is no
> >  Return-Path field in the subject message, there is an implementation
> >  or configuration error in the SMTP server that delivered the message
> >  or gatewayed the message outside of SMTP.  A Personal or Group
> >  responder SHOULD NOT deliver a response to any address other than
> >  that in the Return-Path field, even if the Return-Path field is
> >  missing.  It is better to fix the problem with the mail delivery
> >  system than to rely on heuristics to guess the appropriate
> >  destination of the response.  Such heuristics have been known to
> >  cause problems in the past.

> Well, I think it goes back to "Final Destination"  distinction.

I'm not entirely clear on this whole "final destination" thing you're talking
about - do you perhaps mean "final delivery"? If so, then we're talking about
totally different things here. If not, I guess you're referring to some point
past which the RCPT TO address is no longer subject to change, and I completely
fail to see how that is in any way relevant.

The main sort of automated agent that's a concern here is autoforwarders. A
nasty and hard to detect loop can form If an aufoforwarder changes a NULL MAIL
FROM to point at the autoforwarder itself while redirecting the message to a
remote address that turns out to be invalid.

In this situation the autoforwarder is acting before final delivery has
occurred. Moreover, the autforwarder in general has no way to tell if the RCPT
TO address is the "final destination" or not, and even if it did know, I don't
see any way the information could be used to cause it to "do the right thing".

> Once final destination has been been established, there is no need for a
> return path, null or otherwise.

Not that it is relevant, but RFC 3798 is an obvious counterexample here -
proper MDN generation depends critically on return-path information and MDNs
are routinely generated long after the actual destination has been determined
and final delivery has occurred.

> It is quite possible that a gate will
> eliminate useless network control lines when final destination is a
> foregone conclusion.  The same is true for any post smtp mailbot.

> The key is the automation that is transparent of the policy reasons.

> All the above paragraph means is that the SMTP receivers needs to
> support adding the return-path,

Stuffing the MAIL FROM into a return-path header is part of final delivery.
We're talking about routing modifications made before final delivery takes
place and often without any information about whether further routing will
occur.

> and that its remains persistent through
> out the entire routing and/or mail processing until final destination is
> established, at which point it no longer applies.

Again, this appears to be at odds with MDNs semantics at a minimum.

It also seems apppropriate to caution against any more stringent restrictions
on MAIL FROM modifications. Changing a non-NULL MAIL FROM to some other value
isn't exactly uncommon.

> What happens after that is out of SMTP automation controls.

> Consider the automation logic:

> 1) If the return path is NULL, then no response related the current
> transaction is expected to be generated.  What an external mailbot does
> with transaction is out of the controls of SMTP, so all you can provide
> is insight for post smtp mail processing authors to understand that no
> response is expected as part of the automation process.

OK, but again, this all seems to concern actions taken after final delivery.

> 2) If the return path is NOT NULL, then responses are possible to this
> address. (AFIK, there is no known common user part that restricts a
> response to this non-null address)

> 3) The return path must remain persistent until the routing and final
> destination is completed.

MAIL FROM information is persistent but mutable. And that's the problem -
one particular type of change can cause serious problems and needs to be
restricted.

> 4) If final destination is negatively established (i.e, address or user
> dosn't exist), then the persistent return-path MUST be used for the
> non-delivery notification.

> 5) Once final destination is successfully established (user exist and if
> applicable acceptable by local or user policy), the return path value is
> no longer applicable - Not Needed.

It's no longer needed in the sense of reporting errors for the current message
transfer activity. However, the field's semantics are broader than message
transfer error reporting.

				Ned

Date: Sun, 25 Nov 2007 17:42:38 -0500
From: John C Klensin <[email-masked]>
To: Chris Wright <[email-masked]>
cc: [email-masked]
Subject: Re: Possible ambiguity in SMTP RFC2821 - your opinon please
Sender: [email-masked]

--On Monday, 26 November, 2007 02:42 +1100 Chris Wright
<[email-masked]> wrote:

> Hi John,
>  My name is Chris, I am CTO of AusRegistry the registry
> operator of the .au cctld. I have been doing some research on
> the SMTP protocol lately and have come up with something that
> I guess is a potential ambiguity in the SMTP protocol defined
> in RFC2821 you authored (if I am remembering my BNF
> correctly). Any clarification you can shed on the matter would
> be greatly appreciated.
> 
> In section 2.3.5 of the RFC you define the domain name part of
> an email address as being "one or more dot-separated
> components" (I have reproduced it below to save you having to
> go and find it). Further down in section 4.1.2 your grammar
> describes the domain as being:
> 
>    Domain = (sub-domain 1*("." sub-domain)) / address-literal
> 
> Which (If I am hopefully remembering correctly means a
> sub-domain followed by 1 or more "." sub-domain combinations.
> This appears to be in contradiction to the paragraphs of 2.3.5.
> 
> I have come up against it because I am looking at the
> possibilities of having email addresses under a TLD eg.
> john@au or chris@com, which would presumably be legal given
> that au or com is a fully qualified domain name (when you put
> the dot on the end). It appears most email software doesn't
> support this :(
> 
> Could you clarify firstly my interpretation and secondly the
> intent of the RFC, would be greatly appreciated.
> 
> Thanks
> 
> From Section 2.3.5 of RFC2821
> 
>    A domain (or domain name) consists of one or more
> dot-separated    components.  These components ("labels" in
> DNS terminology [22]) are    restricted for SMTP purposes to
> consist of a sequence of letters,    digits, and hyphens drawn
> from the ASCII character set [1].  Domain    names are used as
> names of hosts and of other entities in the domain    name
> hierarchy.  For example, a domain may refer to an alias (label
>    of a CNAME RR) or the label of Mail eXchanger records to be
> used to    deliver mail instead of representing a host name.
> See [22] and    section 5 of this specification.
> 
>    The domain name, as described in this document and in [22],
> is the    entire, fully-qualified name (often referred to as
> an "FQDN").  A    domain name that is not in FQDN form is no
> more than a local alias.    Local aliases MUST NOT appear in
> any SMTP transaction.
> 
> 
> From Section 4.1.2
> 
>    Domain = (sub-domain 1*("." sub-domain)) / address-literal

Hi Chris,

There is a revision of 2821 in progress.  The current draft is
draft-klensin-rfc2821bis-06.txt and discussion is occurring on
the ietf-smtp list (copied on this note to expose your question
and comments on the list).

The question of whether an address of the form username@au was
actively discussed some months ago, and the ICANN ccNSO and gNSO
were polled on the question.  The (few) responses we got back
from the latter can, I think, be fairly summarized as "no one
cared very much", something you can verify with Chris Disspain
if you like.

The difficulty with TLD-only addresses is that the mail
protocols do not permit that trailing dot.  Discussion over the
last year concluded that the backwards-compatibility issues that
would be raised by changing that are far more problematic than
the change might be worth.  So, if one has a one-component name,
there is some difficulty in distinguishing it lexically from an
abbreviation (not permitted on the wire, but common in local
systems).  For that reason and regardless of what is done with
the standard, you should anticipate difficulties with some mail
user agents in handling the address.

However, while one must be cautious, there seems to be no reason
to prohibit a TLD from being used as an MX record and receiving
mail, so the revision draft has been changed to explicitly
permit that form.  In the working draft I've further modified
that uncomfortable sentence in 2.3.5 but the text (and syntax)
of -06 clearly permit what you want to do.

 regards,
     john

Date: Thu, 22 Nov 2007 05:13:49 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Frank Ellermann <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

Hector Santos wrote:

> Receiver (preliminary: final destination decision made)
> 
>    C: RCPT TO:<some address>
> 
>    250 - USER FOUND
>    45x - USER FOUND (For some policy reason, temporary reject)
>    55x - USER NOT FOUND (domain is locally hosted)
>    55x - RELAY AUTHENTICATION REQUIRED (domain is remote)
> 
> The point is that in the newer models the Return-Path value increases 
> and is more reliable at this point.  Regardless if NULL or not NULL, in 
> this model, it is meaningless. It has no value.  In fact, adding the 
> Return-Path for some post smtp mailbot shouldn't be a consideration 
> because it no longer applies.

I said two different things. :)

Let me be more specific:

I meant it has no value as a "notification" address because it is not 
required with a dynamic SMTP receiver making final destination 
decisions. For a negative response, the message is not going to be 
accepted so there is no issue with post smtp mail bots.  This is ideal 
to prevent bounce attacks.

If the message is accepted, the value of the return path means 
"something." It has value.  If there are post smtp mail bots existing 
outside of the control of the main SMTP implementation that could create 
problems, it is quite possible that the return-path not be added or 
change to NULL.   I'm not recommending this or suggesting it be part of 
some automation logic, but that it is possible in order to minimize post 
smtp mailbot mishaps.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 22 Nov 2007 04:59:10 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Frank Ellermann <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

Frank Ellermann wrote:
> <[email-masked]> wrote:
> 
>> you really *don't* want to have to poke around in the 822 headers
>> to find a 821-style return address (especially since if you *do*
>> that sort of grovelling, you will almost certainly Do The Wrong
>> Thing if you ever get a bounce with an 822-From *other* than 
>> Mailer-Daemon).
> 
> +1  Or as RFC 3834 put it:
> 
>  If the response is to be generated after delivery, and there is no
>  Return-Path field in the subject message, there is an implementation
>  or configuration error in the SMTP server that delivered the message
>  or gatewayed the message outside of SMTP.  A Personal or Group
>  responder SHOULD NOT deliver a response to any address other than
>  that in the Return-Path field, even if the Return-Path field is
>  missing.  It is better to fix the problem with the mail delivery
>  system than to rely on heuristics to guess the appropriate
>  destination of the response.  Such heuristics have been known to
>  cause problems in the past.

Well, I think it goes back to "Final Destination"  distinction.

Once final destination has been been established, there is no need for a 
return path, null or otherwise.  It is quite possible that a gate will 
eliminate useless network control lines when final destination is a 
foregone conclusion.  The same is true for any post smtp mailbot.

The key is the automation that is transparent of the policy reasons.

All the above paragraph means is that the SMTP receivers needs to 
support adding the return-path, and that its remains persistent through 
out the entire routing and/or mail processing until final destination is 
established, at which point it no longer applies.

What happens after that is out of SMTP automation controls.

Consider the automation logic:

1) If the return path is NULL, then no response related the current 
transaction is expected to be generated.  What an external mailbot does 
with transaction is out of the controls of SMTP, so all you can provide 
is insight for post smtp mail processing authors to understand that no 
response is expected as part of the automation process.

2) If the return path is NOT NULL, then responses are possible to this 
address. (AFIK, there is no known common user part that restricts a 
response to this non-null address)

3) The return path must remain persistent until the routing and final 
destination is completed.

4) If final destination is negatively established (i.e, address or user 
dosn't exist), then the persistent return-path MUST be used for the 
non-delivery notification.

5) Once final destination is successfully established (user exist and if 
applicable acceptable by local or user policy), the return path value is 
no longer applicable - Not Needed.

Correct me if I wrong, but the above logic applies to all systems - 
transparently. I don't see any scenario that can alter this fundamental 
long time practice - it is all part of what makes it all works.

The only issue I think that is partially lost in this thread is *when* 
or at what point is *final destination* established.

This is very important design and implementation point because old mail 
models did not have the SMTP receiver did not make Final Destination 
decisions.  Typically it was the SMTP ROUTER that made that decision:

Older model:

Receiver (no final destination decision made)

    temp msg received --> msg stored in spool\queue

Router:

    Local mail -->  queue\msg is stored in spool\inbound
    Relay mail -->  queue\msg is stored in spool\outbound

And in more current models, the same may be true, but the Receiver may 
make a preliminary Final Destination decision as part of the logic to 
implement bounce attacks:

Current and more model model:

Receiver (preliminary: final destination decision made)

    C: RCPT TO:<some address>

    250 - USER FOUND
    45x - USER FOUND (For some policy reason, temporary reject)
    55x - USER NOT FOUND (domain is locally hosted)
    55x - RELAY AUTHENTICATION REQUIRED (domain is remote)

The point is that in the newer models the Return-Path value increases 
and is more reliable at this point.  Regardless if NULL or not NULL, in 
this model, it is meaningless. It has no value.  In fact, adding the 
Return-Path for some post smtp mailbot shouldn't be a consideration 
because it no longer applies.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject:  Re: 2821bis and address rewriting
Date:  Thu, 22 Nov 2007 08:04:50 +0100
Lines: 23
Reply-To: "Frank Ellermann" <[email-masked]>
Sender: [email-masked]

<[email-masked]> wrote:

> you really *don't* want to have to poke around in the 822 headers
> to find a 821-style return address (especially since if you *do*
> that sort of grovelling, you will almost certainly Do The Wrong
> Thing if you ever get a bounce with an 822-From *other* than=20
> Mailer-Daemon).

+1  Or as RFC 3834 put it:

 If the response is to be generated after delivery, and there is no
 Return-Path field in the subject message, there is an implementation
 or configuration error in the SMTP server that delivered the message
 or gatewayed the message outside of SMTP.  A Personal or Group
 responder SHOULD NOT deliver a response to any address other than
 that in the Return-Path field, even if the Return-Path field is
 missing.  It is better to fix the problem with the mail delivery
 system than to rely on heuristics to guess the appropriate
 destination of the response.  Such heuristics have been known to
 cause problems in the past.

Frank

Date: Wed, 21 Nov 2007 22:10:39 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Cc: Tony Finch <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>
Sender: [email-masked]

Tony Finch writes:
> On Wed, 21 Nov 2007, Arnt Gulbrandsen wrote:
>> But 3461 (bottom two paragraphs of page 13) says any message can have 
>> <> as sender when it reaches the recipient.
>
> Null return paths have never been particularly consistent or clearly 
> scoped :-) But still, RFC 3461 doesn't say that arbitrary programs 
> can send email from <> because they think it would be neat.

As I read it, it says that arbitrary SMTP clients may send email from <> 
because the sender of the message doesn't want to be told about the 
fate of the message. IMO that's much the same thing.

Arnt

Date: Wed, 21 Nov 2007 20:09:40 +0000
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Sender: [email-masked]

On Wed, 21 Nov 2007, Arnt Gulbrandsen wrote:
> Tony Finch writes:
> >
> > I don't think that is a valid use of a null return path. RFC 2821
> > section 4.5.5 says:
> >
> >    All other types of messages (i.e., any message which is not
> >    required by a standards-track RFC to have a null reverse-path)
> >    SHOULD be sent with with a valid, non-null reverse-path.
> >
> > and the list of standards-track specifications is just DSNs (RFC
> > 3461), MDNs (RFC 3798), and vacation messages (RFC 3834).
>
> But 3461 (bottom two paragraphs of page 13) says any message can have <>
> as sender when it reaches the recipient.

Null return paths have never been particularly consistent or clearly
scoped :-) But still, RFC 3461 doesn't say that arbitrary programs can
send email from <> because they think it would be neat.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MALIN HEBRIDES: NORTH VEERING NORTHEAST 5 TO 7, OCCASIONALLY GALE 8. MODERATE
OR ROUGH. SQUALLY SHOWERS. GOOD.

Date: Wed, 21 Nov 2007 20:53:53 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Cc: Tony Finch <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>
Sender: [email-masked]

Tony Finch writes:
> On Mon, 19 Nov 2007, [email-masked] wrote:
>>  Seems people have problems wrapping their tiny little brains around 
>>  the idea that <> is useful for other things - in this particular 
>>  case, it was a mailing list manager that sent out all the "Please 
>>  reply to this message to confirm your request" messages with <> 
>>  specifically because it did *not* want to hear back if there was a 
>>  problem (as the request would just time out all by itself anyhow).
>
> I don't think that is a valid use of a null return path. RFC 2821 
> section 4.5.5 says:
>
>    All other types of messages (i.e., any message which is not 
>    required by a standards-track RFC to have a null reverse-path) 
>    SHOULD be sent with with a valid, non-null reverse-path.
>
> and the list of standards-track specifications is just DSNs (RFC 
> 3461), MDNs (RFC 3798), and vacation messages (RFC 3834).

But 3461 (bottom two paragraphs of page 13) says any message can have <> 
as sender when it reaches the recipient.

Arnt

Date: Wed, 21 Nov 2007 19:20:25 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Sender: [email-masked]

On Mon, 19 Nov 2007, [email-masked] wrote:
>
> Seems people have problems wrapping their tiny little brains around the
> idea that <> is useful for other things - in this particular case, it
> was a mailing list manager that sent out all the "Please reply to this
> message to confirm your request" messages with <> specifically because
> it did *not* want to hear back if there was a problem (as the request
> would just time out all by itself anyhow).

I don't think that is a valid use of a null return path. RFC 2821 section
4.5.5 says:

   All other types of messages (i.e., any message which is not required
   by a standards-track RFC to have a null reverse-path) SHOULD be sent
   with with a valid, non-null reverse-path.

and the list of standards-track specifications is just DSNs (RFC 3461),
MDNs (RFC 3798), and vacation messages (RFC 3834).

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH FITZROY SOLE LUNDY FASTNET: NORTHEAST BACKING NORTHWEST 4 OR 5,
OCCASIONALLY 6. SLIGHT OR MODERATE, OCCASIONALLY ROUGH. SHOWERS. GOOD.

Date: Wed, 21 Nov 2007 10:06:41 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis and address rewriting
Cc: Hector Santos <[email-masked]>, Dave Crocker <[email-masked]>
Sender: [email-masked]

Hector Santos writes:
> Dave Crocker wrote:
>> Almost makes one wish that real bounce messages were distinguished by 
>> something other than the 2821.mailfrom address...
>
> Isn't this already done by most systems via the x822.From header 
> address user part having mainly 'mailer-daemon'?

There is a standards-track RFC saying that the way to distinguish them 
from other mail is

     Content-Type: multipart/report

Arnt

Date: Tue, 20 Nov 2007 23:36:06 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: Dave Crocker <[email-masked]>, [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

[email-masked] wrote:
> On Tue, 20 Nov 2007 19:02:01 EST, Hector Santos said:
>> Dave Crocker wrote:
>>
>>> Almost makes one wish that real bounce messages were distinguished by 
>>> something other than the 2821.mailfrom address...
>> Isn't this already done by most systems via the x822.From header address 
>> user part having mainly 'mailer-daemon'?
> 
> Yes, but that's a layering issue - you really *don't* want to have to poke
> around in the 822 headers to find a 821-style return address (especially
> since if you *do* that sort of grovelling, you will almost certainly Do The
> Wrong Thing if you ever get a bounce with an 822-From *other* than Mailer-Daemon).

Ok. I thought this 2821bis "update" concern was regarding automated bots 
altering a NULL return path (an implied post smtp concept).  But yes, I 
see your point.

Maybe an ESMTP I-D for a new MAIL FROM modifier?

   C: MAIL FROM: <> [EFA=Email From Address]

Also, isn't it a "requirement" (and also one of the "security" issues 
wrt spam), that a NULL return path must be accepted assuming RCPT TO 
passes some possible validation, if any?  This may make an I-D proposal 
a mute issue.

-- 
Hector Santos
http://www.santronics.com

To: Hector Santos <[email-masked]>
Cc: Dave Crocker <[email-masked]>, [email-masked]
Subject: Re: 2821bis and address rewriting
From: [email-masked]
Date: Tue, 20 Nov 2007 19:44:12 -0500
Sender: [email-masked]

--==_Exmh_1195605852_4794P
Content-Type: text/plain; charset=us-ascii

On Tue, 20 Nov 2007 19:02:01 EST, Hector Santos said:
> 
> Dave Crocker wrote:
> 
> > Almost makes one wish that real bounce messages were distinguished by 
> > something other than the 2821.mailfrom address...
> 
> Isn't this already done by most systems via the x822.From header address 
> user part having mainly 'mailer-daemon'?

Yes, but that's a layering issue - you really *don't* want to have to poke
around in the 822 headers to find a 821-style return address (especially
since if you *do* that sort of grovelling, you will almost certainly Do The
Wrong Thing if you ever get a bounce with an 822-From *other* than Mailer-Daemon).

--==_Exmh_1195605852_4794P
Content-Type: application/pgp-signature

--==_Exmh_1195605852_4794P--

Date: Tue, 20 Nov 2007 19:02:01 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Dave Crocker <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

Dave Crocker wrote:

> Almost makes one wish that real bounce messages were distinguished by 
> something other than the 2821.mailfrom address...

Isn't this already done by most systems via the x822.From header address 
user part having mainly 'mailer-daemon'?

For notifications, I've seen these x822.From user-part usages coupled 
with a null x821.MailFrom:

     postmaster
     abuse   (via your RFC 2142)
     root
     admin
     administrator
     sysadmin
     sysop

-- 
Hector Santos
http://www.santronics.com

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
To: Dave Crocker <[email-masked]>
Subject: Re: 2821bis and address rewriting
Date: Tue, 20 Nov 2007 10:21:42 -0800
Sender: [email-masked]

On Nov 19, 2007, at 4:36 PM, Dave Crocker wrote:

> [email-masked] wrote:
>>> There
>>> are reasons for not messing with null return paths, including  
>>> their being an important clue for various automated programs, that  
>>> has nothing to do with mail loops (at least directly).
>> Amen to that.  Over the years, I've had to explain to a *lot* of  
>> people that "No, that piece of mail with MAIL FROM:<> isn't a  
>> bounce message. Yes, it's correct".  Seems people have problems  
>> wrapping their tiny little brains around the idea that <> is useful  
>> for other things
>
> Or rather, it is useful for exactly its primary purpose, which is to  
> signal that error messages are not to be returned.
>
> The problem is that we have overloaded that construct with more  
> semantics, such as "this is a bounce message" and then done it  
> inconsistently.
>
> Almost makes one wish that real bounce messages were distinguished  
> by something other than the 2821.mailfrom address...

Indeed.  Despite possible loss of diagnostic information regarding use  
of RFC3464 where original message content is omitted (including  
subject line),  a spartan DSN would be a means to restore order.  Any  
exception to the omission of content could be exploited.

Currently tens of million of different servers each day are exploited  
to send messages where the 2821.mailfrom has been spoofed.  Adhering  
to RFC3464 and omitting original content is a safe means to mitigate  
this abuse, where adherence can be enforced through reputation lists  
filtering null 2821.mailfrom messages.  Such a practice would likely  
drive a need for MUAs able to recognize valid DSNs, while also  
lessening payloads associated with bad actors attempting this exploit.

-Doug

Date: Mon, 19 Nov 2007 18:36:32 -0600
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked]
CC: [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

[email-masked] wrote:
>>     There
>> are reasons for not messing with null return paths, including
>> their being an important clue for various automated programs,
>> that has nothing to do with mail loops (at least directly).
> 
> Amen to that.  Over the years, I've had to explain to a *lot* of people that
> "No, that piece of mail with MAIL FROM:<> isn't a bounce message. Yes, it's
> correct".  	Seems people have problems wrapping their tiny little brains
> around the idea that <> is useful for other things 

Or rather, it is useful for exactly its primary purpose, which is to signal 
that error messages are not to be returned.

The problem is that we have overloaded that construct with more semantics, 
such as "this is a bounce message" and then done it inconsistently.

Almost makes one wish that real bounce messages were distinguished by 
something other than the 2821.mailfrom address...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

To: John C Klensin <[email-masked]>
Cc: [email-masked], SM <[email-masked]>, [email-masked]
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
From: [email-masked]
Date: Mon, 19 Nov 2007 17:35:37 -0500
Sender: [email-masked]

--==_Exmh_1195511737_4539P
Content-Type: text/plain; charset=us-ascii

On Sun, 18 Nov 2007 22:13:35 EST, John C Klensin said:
> (2) I fear someone coming along and saying "well, I know how to
> avoid mail loops, so that statement doesn't apply to me".  There
> are reasons for not messing with null return paths, including
> their being an important clue for various automated programs,
> that has nothing to do with mail loops (at least directly).

Amen to that.  Over the years, I've had to explain to a *lot* of people that
"No, that piece of mail with MAIL FROM:<> isn't a bounce message. Yes, it's
correct".  	Seems people have problems wrapping their tiny little brains
around the idea that <> is useful for other things - in this particular case,
it was a mailing list manager that sent out all the "Please reply to this
message to confirm your request" messages with <> specifically because it
did *not* want to hear back if there was a problem (as the request would just
time out all by itself anyhow).

--==_Exmh_1195511737_4539P
Content-Type: application/pgp-signature

--==_Exmh_1195511737_4539P--

Date: Mon, 19 Nov 2007 09:29:14 -0500
From: John C Klensin <[email-masked]>
To: Pete Resnick <[email-masked]>, [email-masked]
cc: [email-masked], ietf-822 <[email-masked]>, [email-masked]
Subject: Re: Missing ABNF terms in 2821bis?
Sender: [email-masked]

--On Sunday, 18 November, 2007 22:48 -0600 Pete Resnick
<[email-masked]> wrote:

> 
> So, going back to my original questions:
> 
> On 11/18/07 at 5:36 PM -0800, [email-masked] wrote:
> 
>> [John Klensin wrote:]
>>> 
>>> --On Wednesday, 17 October, 2007 09:46 -0700 Pete Resnick 
>>> <[email-masked]> wrote:
>>> 
>>>> - Domain literals on the right hand side of addresses
>>> 
>>> This is the one that affects 2821bis. The control characters
>>> cannot  appear in either IPv4 address or IPv6 address domain
>>> literals. They  can, in principle, appear in General Address
>>> literal, but those  require standards track action, etc.,
>>> and one could easily cut them  off there. In other words,
>>> they are permitted in principle, but  prohibited in practice
>>> and are going to stay that way. It is  therefore probably
>>> safe to let this go.
>> 
>> Agreed. The issue is really with 2822. 2821's rules for
>> defining new  address literals offer sufficient protection
>> IMO.
> 
> So you and John are saying that we should pull NO-WS-CTL out
> of dtext?

I would be happy seeing it moved some sort of obsolete status,
e.g., moved to "accept only" and then attach to a statement that
it is enough of a source of trouble that one should be really
cautious about that.

> Should we allow control characters in quoted-pair inside
> domain-literal?

I think not.   Keep in mind that 2821bis (and 2821) prohibits
them in IPv4 and IPv6 address literals as well as in domain name
labels.  So the only possible case involves General (i.e.,
post-IPv6) literals.  No IESG in its collective right mind would
approve a syntax for those that included control characters so,
at least in the envelope or on the wire, this is a dead case.
 
> Should we allow quoted-pair at all in domain-literal?
> 
> (Remember that they all appear in the obsolete (i.e., must be
> able to handle for interpretation) syntax.)
> 
> The first is easy. The latter two are trickier, with the
> second one giving me severe heartburn.

Indeed.  I have trouble removing something that is even slightly
plausible from the list of what is already obsolete although I
could easily live with inserting a "MAY reject if this appears"
there.  To the extent to which they are tolerated in 821/2821, I
imagine it is a legacy of NVT, where some of these controls were
mistakenly (IMO) permitted as the only way (at the time) to
manage even minimal formatting.  I have no idea what they were
doing in 822 -- perhaps Dave remembers and can shed some light
on this.

>>>> - Quoted strings (which can be on the left hand size of
>>>> addresses  as well as lots of other structured header
>>>> fields)
>>> 
>>> The address part of this is a little more problematic than
>>> the case  above, but also easier, under the "you can screw
>>> only yourself by  putting one of those addresses up on your
>>> server" principle.
> 
> So John, you are saying that we should leave NO-WS-CTL in
> quoted-string?

I really do not know.  _Very_ mixed feelings.  I'd be happy to
see them go, but am a little concerned about unintended
consequences.  On the other hand, I would not lose any sleep
over keeping it because I don't mind helping stupid people enjoy
their stupidity.

>>>> - Comments (which can appear in lots of structured header
>>>> fields  including address fields)
>>>> - Unstructured text (in fields and in the body)
>>> 
>>> The rest of this is, indeed, a 2822 problem.
>> 
>> Yep, that's where it really needs to be dealt with.
> 
> And the answer to whether or not we should leave NO-WS-CTL in
> these is....?

I think you should at least have a really strong warning about
the difficulties invisible characters and characters whose
semantics may be interpreted different by the sender and
receiver can cause.  I don't have a strong opinion about how
much beyond that it is sensible to go. 

Sorry I'm unable to be more definitive this morning.

    john

To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject:  SRS and TBR (was: Fixing graylisting)
Date:  Mon, 19 Nov 2007 10:27:11 +0100
Lines: 19
Reply-To: "Frank Ellermann" <[email-masked]>
Sender: [email-masked]

SM wrote:
=20
> A side-effect of using SPF for the first part above is that sender=20
> address rewrite generally suggested as a solution for forwarding=20
> doesn't fit into TBR.

SRS is only one way how receivers can forward SPF FAIL protected
mails to third parties checking SPF "again" (i.e. at a point where
the "border" IPs permitted by the originator cannot help anymore).

Receivers can use other methods to get a similar effect, they could
whitelist "trusted forwarders" (e.g. in combination with an SPF HELO
PASS), they can "pull" their mails with POP3 from their old address,
they could also "pull" them with TBR if the originator, ex-forwarder,
and next hop (new address at third party) support TBR. =20

As far as SPF is concerned they could also use avian carriers... :-)

 Frank

Date: Sun, 18 Nov 2007 22:48:19 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: Re: Missing ABNF terms in 2821bis?
Cc: John C Klensin <[email-masked]>, [email-masked], ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

So, going back to my original questions:

On 11/18/07 at 5:36 PM -0800, [email-masked] wrote:

>[John Klensin wrote:]
>>
>>--On Wednesday, 17 October, 2007 09:46 -0700 Pete Resnick 
>><[email-masked]> wrote:
>>
>>>- Domain literals on the right hand side of addresses
>>
>>This is the one that affects 2821bis. The control characters cannot 
>>appear in either IPv4 address or IPv6 address domain literals. They 
>>can, in principle, appear in General Address literal, but those 
>>require standards track action, etc., and one could easily cut them 
>>off there. In other words, they are permitted in principle, but 
>>prohibited in practice and are going to stay that way. It is 
>>therefore probably safe to let this go.
>
>Agreed. The issue is really with 2822. 2821's rules for defining new 
>address literals offer sufficient protection IMO.

So you and John are saying that we should pull NO-WS-CTL out of dtext?

Should we allow control characters in quoted-pair inside domain-literal?

Should we allow quoted-pair at all in domain-literal?

(Remember that they all appear in the obsolete (i.e., must be able to 
handle for interpretation) syntax.)

The first is easy. The latter two are trickier, with the second one 
giving me severe heartburn.

>>>- Quoted strings (which can be on the left hand size of addresses 
>>>as well as lots of other structured header fields)
>>
>>The address part of this is a little more problematic than the case 
>>above, but also easier, under the "you can screw only yourself by 
>>putting one of those addresses up on your server" principle.

So John, you are saying that we should leave NO-WS-CTL in quoted-string?

>>>- Comments (which can appear in lots of structured header fields 
>>>including address fields)
>>>- Unstructured text (in fields and in the body)
>>
>>The rest of this is, indeed, a 2822 problem.
>
>Yep, that's where it really needs to be dealt with.

And the answer to whether or not we should leave NO-WS-CTL in these is....?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sun, 18 Nov 2007 22:15:10 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry  Strategy     recommendation
Sender: [email-masked]

--On Sunday, 18 November, 2007 18:37 -0600 Dave Crocker
<[email-masked]> wrote:

> John C Klensin wrote:
>> "Four" is in the spec because it is a plausible compromise
>> without getting bogged down in the details of multiple
>> countries and cultures and because, if one does get that
>> involved, one will often come up with four anyway.
> 
> As I recall, there was also an empirical basis for this, based
> on thinking about the notification "spikes" that were observed
> over some years.
> 
> For obvious and reasonable reasons, one can strongly argue for
> the time-out to be 1, 2, 3, 4, 5, 7 and even 10 days.  The
> latter are too long for effectiveness and the first few cause
> too many false notices.
> 
> That leaves, perhaps, 3, 4 and 5 to debate about and 4 does
> jump out as a reasonable compromise, doesn't it?

Exactly.
   john

Date: Sun, 18 Nov 2007 22:13:35 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], SM <[email-masked]>
cc: [email-masked]
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting   [was TBR])
Sender: [email-masked]

--On Sunday, 18 November, 2007 17:28 -0800
[email-masked] wrote:

>>      If a reverse-path is null, it SHOULD NOT be modified as
>>      that is likely to cause a mail loop.
> 
> I think citing loop prevention as the reason for this
> recommendation is a good
> idea, but think this overstates things a bit - it creates a
> condition where
> loops are possible, but not necessarily likely. So change it
> to say "SHOULD NOT
> be modified as this can cause mail loops" and I'm happy.

I just dropped 2821bis-06 into the posting queue.  It says
"SHOULD NOT be modified" but doesn't give a reason.

I had two reasons for that: if you (or others) disagree, it can
obviously still be changed.

(1) More often than not, we don't give reasons for rules of this
sort.  I usually favor having the explanations, but have been
regularly beaten up for it.

(2) I fear someone coming along and saying "well, I know how to
avoid mail loops, so that statement doesn't apply to me".  There
are reasons for not messing with null return paths, including
their being an important clue for various automated programs,
that has nothing to do with mail loops (at least directly).   So
it seems to me that, in this case, the right answer is "just say
'no'" rather than trying to justify that decision in the text.

    john

Date: Sun, 18 Nov 2007 21:49:35 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: SM <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Subject: Re: 2821bis and address rewriting
Sender: [email-masked]

[email-masked] wrote:
> 
>> I suggest the following change to Section 4.5.5 (Messages with a null
>> reverse-path):
> 
>>   If a reverse-path is null, it SHOULD NOT be modified as 
 >>   that is likely to cause a mail loop.
> 
> I think citing loop prevention as the reason for this recommendation is 
> a good idea, but think this overstates things a bit - it creates a 
 > condition where loops are possible, but not necessarily likely. So
 > change it to say "SHOULD NOT be modified as this can cause
 > mail loops" and I'm happy.

Hi Ned,

Question, is a NULL return path a final destination idea?

If so, then maybe a statement or clarification could be finessed:

    A null return path is considered a final destination
    transaction.  To avoid the potential of creating mail loops,
    if a reverse-path is null, SMTP accepted messages handled
    by automated email processors or gateways MUST|SHOULD NOT
    modified or create a return path after final destination
    and mail delivery decisions are firmly established.

I think the key is making that final destination distinction.

Some examples flows:

Example 1:  NON-NULL

    SMTP RECEIVER - mail accepted, adds Return-Path: <!null>
    mailbotA - does something, set Return-Path: <>
    mailbotB - does something else, restricted from sending bounces

Example 2: NULL

    SMTP RECEIVER - bounce mail accepted, adds Return-Path: <>
    mailbotA - does something, set Return-Path: <!NULL>, adds to queue

In example 1, final destination is not necessarily established, 
especially if the host in example 1 is acting as  relay or router.  But 
if mailbotA (gateway, local host importer) establishes final 
destination, it might want to set Return-Path: to prevent any other
subsequent bot from erroneously sending a bounce.  Note: There exist 
applications which will "prune" the unnecessary headers, including the 
return-path in order to make it presentable or useful to users in 
multi-device reading environments, and this is only allowed because the 
final destination has already been established.

In example 2, having a NULL return path signifies "final destination" 
and therefore no mailbot SHOULD NOT modified it if the message has the 
potential to be put back into an outbound queue mail stream.

So I think SM's other statement about proper POST SMTP handling is 
important.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

From: [email-masked]
Cc: Pete Resnick <[email-masked]>, [email-masked], ietf-822 <[email-masked]>, [email-masked]
Date: Sun, 18 Nov 2007 17:36:30 -0800 (PST)
Subject: Re: Missing ABNF terms in 2821bis?
To: John C Klensin <[email-masked]>
Sender: [email-masked]

> Dusting off this particular bit of ancient history...

> --On Wednesday, 17 October, 2007 09:46 -0700 Pete Resnick
> <[email-masked]> wrote:

> >
> > I've copied this conversation over to ietf-822, since I think
> > that will probably be where this issue is addressed.
> >
> > On 10/17/07 at 6:51 AM -0700, [email-masked] wrote:
> >
> >>> Peter J. Holzer wrote:
> >>>
> >>>> Which reminds me: What is the reason for allowing control
> >>>> characters in email addresses?
> >>>
> >>> Idiocy.
> >>
> >> I try hard to stay out of these sorts of debates, but in this
> >> case I  am forced to agree. We are way past the point where
> >> backwards  compatibility is an acceptable excuse for
> >> something like this. It  should be obvious why  allowing
> >> nonprinting control characters in  addresses is a bad idea on
> >> many levels.
> >>
> >> And if common sense isn't enough here, how about the fact
> >> that it is  pretty clear that these things have obvious and
> >> severe  interoperability problems that will make it hard to
> >> meet the  criterua for draft standard?
> >
> > Right now, control characters are allowed in the following:
> >
> > - Domain literals on the right hand side of addresses

> This is the one that affects 2821bis.  The control characters
> cannot appear in either IPv4 address or IPv6 address domain
> literals.   They can, in principle, appear in General Address
> literal, but those require standards track action, etc., and one
> could easily cut them off there.  In  other words, they are
> permitted in principle, but prohibited in practice and are going
> to stay that way.    It is therefore probably safe to let this
> go.

Agreed. The issue is really with 2822. 2821's rules for defining new address
literals offer sufficient protection IMO.

> I'm not going to do any for the version of 2821 that I intend to
> post later tonight, but, if there is consensus that it is time
> to do away with the puppies (and I am very much in agreement
> with Ned that they should go), it may be time to invent a new
> metalanguage rule for, e.g., "pcontent" (for printable) which
> consists of characters in the ASCII range from 20 to 7E (hex).

Seems like a reasonable thing to do to me. The interesting thing is that
this excludes tab, and I actually think that's the right thing to do.

> Looking ahead to i18n issues, it may be desirable to have a
> similar rule for Unicode characters, but the characters can't be
> enumerated (in practice and to preserve version-independence) so
> the rule would have to be defined in terms of character class
> and properties.

I haven't looked at this in much detail but I suspect it's going to be tricky
to get right. For example, the tab issue, if you'll pardon the pun, expands to
include the many other ways of representing whitespace provided by Unicode. Now
think of how this might interact with BIDI rules...

> > - Quoted strings (which can be on the left hand size of
> > addresses as well as lots of other structured header fields)

> The address part of this is a little more problematic than the
> case above, but also easier, under the "you can screw only
> yourself by putting one of those addresses up on your server"
> principle.

> The rest of this is, indeed, a 2822 problem.

Yep, that's where it really needs to be dealt with.

				Ned

From: [email-masked]
Cc: John C Klensin <[email-masked]>, [email-masked], [email-masked]
Date: Sun, 18 Nov 2007 17:28:47 -0800 (PST)
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting   [was TBR])
To: SM <[email-masked]>
Sender: [email-masked]

> Hi John,
> At 18:51 17-11-2007, [email-masked] wrote:
> >Agreed. But it needs to be clear that this applies to autoforwarders. How
> >about adding "and SHOULD NOT add a non-null reverse path to such a message
> >when forwarding".

> I suggest the following change to Section 4.5.5 (Messages with a null
> reverse-path):

>      If a reverse-path is null, it SHOULD NOT be modified as that is likely
>      to cause a mail loop.

I think citing loop prevention as the reason for this recommendation is a good
idea, but think this overstates things a bit - it creates a condition where
loops are possible, but not necessarily likely. So change it to say "SHOULD NOT
be modified as this can cause mail loops" and I'm happy.

				Ned

Date: Sun, 18 Nov 2007 20:28:47 -0500
From: John C Klensin <[email-masked]>
To: Pete Resnick <[email-masked]>, [email-masked]
cc: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: Missing ABNF terms in 2821bis?
Sender: [email-masked]

Dusting off this particular bit of ancient history...

--On Wednesday, 17 October, 2007 09:46 -0700 Pete Resnick
<[email-masked]> wrote:

> 
> I've copied this conversation over to ietf-822, since I think
> that will probably be where this issue is addressed.
> 
> On 10/17/07 at 6:51 AM -0700, [email-masked] wrote:
> 
>>> Peter J. Holzer wrote:
>>> 
>>>> Which reminds me: What is the reason for allowing control 
>>>> characters in email addresses?
>>> 
>>> Idiocy.
>> 
>> I try hard to stay out of these sorts of debates, but in this
>> case I  am forced to agree. We are way past the point where
>> backwards  compatibility is an acceptable excuse for
>> something like this. It  should be obvious why  allowing
>> nonprinting control characters in  addresses is a bad idea on
>> many levels.
>> 
>> And if common sense isn't enough here, how about the fact
>> that it is  pretty clear that these things have obvious and
>> severe  interoperability problems that will make it hard to
>> meet the  criterua for draft standard?
> 
> Right now, control characters are allowed in the following:
> 
> - Domain literals on the right hand side of addresses

This is the one that affects 2821bis.  The control characters
cannot appear in either IPv4 address or IPv6 address domain
literals.   They can, in principle, appear in General Address
literal, but those require standards track action, etc., and one
could easily cut them off there.  In  other words, they are
permitted in principle, but prohibited in practice and are going
to stay that way.    It is therefore probably safe to let this
go.

I'm not going to do any for the version of 2821 that I intend to
post later tonight, but, if there is consensus that it is time
to do away with the puppies (and I am very much in agreement
with Ned that they should go), it may be time to invent a new
metalanguage rule for, e.g., "pcontent" (for printable) which
consists of characters in the ASCII range from 20 to 7E (hex).
Looking ahead to i18n issues, it may be desirable to have a
similar rule for Unicode characters, but the characters can't be
enumerated (in practice and to preserve version-independence) so
the rule would have to be defined in terms of character class
and properties.

> - Quoted strings (which can be on the left hand size of
> addresses as well as lots of other structured header fields)

The address part of this is a little more problematic than the
case above, but also easier, under the "you can screw only
yourself by putting one of those addresses up on your server"
principle.

The rest of this is, indeed, a 2822 problem.

> - Comments (which can appear in lots of structured header
> fields including address fields)
> - Unstructured text (in fields and in the body)

   john

Date: Sun, 18 Nov 2007 18:37:57 -0600
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry  Strategy     recommendation
Sender: [email-masked]

John C Klensin wrote:
> "Four" is in the spec because it is a plausible compromise
> without getting bogged down in the details of multiple countries
> and cultures and because, if one does get that involved, one
> will often come up with four anyway.

As I recall, there was also an empirical basis for this, based on thinking 
about the notification "spikes" that were observed over some years.

For obvious and reasonable reasons, one can strongly argue for the time-out to 
be 1, 2, 3, 4, 5, 7 and even 10 days.  The latter are too long for 
effectiveness and the first few cause too many false notices.

That leaves, perhaps, 3, 4 and 5 to debate about and 4 does jump out as a 
reasonable compromise, doesn't it?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sun, 18 Nov 2007 19:14:01 -0500
From: John C Klensin <[email-masked]>
To: "Robert A. Rosenberg" <[email-masked]>, [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry  Strategy     recommendation
Sender: [email-masked]

--On Sunday, 18 November, 2007 16:14 -0500 "Robert A. Rosenberg"
<[email-masked]> wrote:

> 
> At 03:08 -0500 on 11/18/2007, John C Klensin wrote about Re:
> 2821bis consideration - New 2nd attempt Retry Strategy :
> 
>...
>> And that is exactly why the guidance in the spec says what it
>> does.
>> 
>>    john
> 
> My comment was specifically directed to the suggestion to trim
> the give-up period back to 2 days. I pointed out that weekends
> are not the only problem with 2 days. If someone DOES decide
> that 2 days is enough, that should be defined as 2 business
> days and automatically adjusted when 2 business days spans
> more than 2 calender days. IOW: Account for Holidays and
> Weekends that fall during the next 2 calender days (ie: Set
> the give-up date on a message to 2 calender days after
> submission). This still is US Centric about Holidays but it
> does allow for the shortened give-up period when it will not
> run into problems.

Sorry, I am rarely too succinct for people, but it happens.

Since I try to avoid being US Centric, and the protocol
certainly should not be, what I was trying to suggest was that
it is probably a bad idea to try to figure out and adjust for
"business days".  It is probably an exaggeration, but it often
seems as if every day is a holiday somewhere.  Since Friday is
the rest/ business break day in significant parts of the work,
any sort of international scheme has to consider that "weekends"
are three days long (Friday-Sunday) even before allowing for
differences that occur due to time zones and the date line.  

The consequence of this is that planning for "two business days"
will take one fairly close to a four calendar day window even
without allowing for national holidays.  If one does allow for
national holidays in a non-trivial number of countries, but
ignores multi-day holidays, one quickly gets to four calendar
days as well (although one then also has to sort out weekends
and non-weekends).

"Four" is in the spec because it is a plausible compromise
without getting bogged down in the details of multiple countries
and cultures and because, if one does get that involved, one
will often come up with four anyway.

I think there is one important way in which one might be
selective and that is on notification messages: it is just my
personal opinion, but I'm a lot more willing to dump a
notification message that cannot be delivered after a couple of
days than I am to lose a message that was sent out by a known
legitimate user or via an authorized relay operation.  For the
latter, I consider the specified four days to be a fairly bare
minimum for robustness.   

Again, just my opinion.

    john

Date: Sun, 18 Nov 2007 17:46:47 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "David F. Skoll" <[email-masked]>
CC: [email-masked]
Subject: Re: 2nd attempt Retry Time - Real World Examples
Sender: [email-masked]

David F. Skoll wrote:
> Hector Santos wrote:
> 
>> I wrote quick log analyzer to produce the domain vs 2nd retry time
>> result. I ran the analyzer for my November 2007 logs only.
> 
> [data removed]
> 
> RFC 2821 doesn't preclude the behaviour you've implemented (a quick
> second try with slower retries after that.)
> 
> So I don't think it's necessary to modify the RFC to suggest that
> maybe a quick second try is a good idea.  We don't want to codify
> something in an RFC that might be inappropriate a few years from now.

The behavior is inappropiate now. :-)

Anyway, if Mr. Skroll is now officially against it, we can nixed the 
suggestion.

I take it off the board now.  I won't bring it up again.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sun, 18 Nov 2007 16:14:20 -0500
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry  Strategy recommendation
Sender: [email-masked]

At 03:08 -0500 on 11/18/2007, John C Klensin wrote about Re: 2821bis 
consideration - New 2nd attempt Retry Strategy :

>--On Saturday, 17 November, 2007 21:54 -0500 "Robert A.
>Rosenberg" <[email-masked]> wrote:
>
>>
>>  At 00:20 -0800 on 11/17/2007, SM wrote about Re: 2821bis
>>  consideration - New 2nd attempt Retry Strategy :
>>
>>>  It's only a lot of retries over four or five days if you
>>>  retry too  frequently.  Some view the period as too long.  A
>>>  two-day give-up  period, for example, may result in delivery
>>>  issues being missed if  it falls over a weekend
>>
>>  You need at least 4 days to handle 3 day weekends (a holiday
>>  on Friday or Monday) although I've seen 4 day Holidays (Xmas
>>  and New Years on a Friday and the company shutting down at
>>  Noon on Thursday). A better idea is to use 3 days BUT adjust
>>  it to 4 or 5 days when there is a Friday/Monday holiday
>>  (although that can still run into problems since it is US
>>  Centric and can run afoul of Non-US Holidays).
>
>And that is exactly why the guidance in the spec says what it
>does.
>
>    john

My comment was specifically directed to the suggestion to trim the 
give-up period back to 2 days. I pointed out that weekends are not 
the only problem with 2 days. If someone DOES decide that 2 days is 
enough, that should be defined as 2 business days and automatically 
adjusted when 2 business days spans more than 2 calender days. IOW: 
Account for Holidays and Weekends that fall during the next 2 
calender days (ie: Set the give-up date on a message to 2 calender 
days after submission). This still is US Centric about Holidays but 
it does allow for the shortened give-up period when it will not run 
into problems.

Date: Sun, 18 Nov 2007 10:53:29 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Fixing Greylist - Its usage of 451 vs 450
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 21:44 17-11-2007, Hector Santos wrote:
>However, I believe you are saying the exception to the literal 
>reason rule would be the "catch all" 450 response code when the 
>reject reasons does not fit 451 or 452 because those two codes are 
>deemed sacred for those two "finer" specific server rejection 
>literal reasons only.

My reply is based on draft-klensin-rfc2821bis-05.  I'm not suggesting 
that the 450 reply code is a "catch all".  As you said, I tried to 
see where the reject reason fits in.  The aim is to convey the 
"message" clearly, within the guidelines of the RFC, to the other end 
instead of them second-guessing what might be wrong.

>  - 451 seems to be reserved for the server admission that
>    it just experienced a backend server problem but it is
>    deemed to be temporary, i.e, its not the sender's fault,
>    therefore it should try again,

Yes.

Section 4.3.2 (Command-Reply Sequences) of RFC 2821 lists each 
command with its usual possible replies.  It does not mention 450 in 
the DATA phase.  There is a change in rfc2821bis to include the 450 
reply code there.

Any further comments will be off-list.

Regards,
-sm 

Date: Sun, 18 Nov 2007 12:42:45 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: 2nd attempt Retry Time - Real World Examples
Sender: [email-masked]

Hector Santos wrote:

> I wrote quick log analyzer to produce the domain vs 2nd retry time
> result. I ran the analyzer for my November 2007 logs only.

[data removed]

RFC 2821 doesn't preclude the behaviour you've implemented (a quick
second try with slower retries after that.)

So I don't think it's necessary to modify the RFC to suggest that
maybe a quick second try is a good idea.  We don't want to codify
something in an RFC that might be inappropriate a few years from now.

Regards,

David.

Date: Sun, 18 Nov 2007 18:34:27 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Greylisting pitfalls
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--8t9RHnE3ZwKMSgU+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-18 11:28:25 -0500, David F. Skoll wrote:
> Peter J. Holzer wrote:
> > Just out of curiosity: Which version of Lotus was this?
>=20
> This behavior was pointed out to me by a customer who
> uses Lotus.  This is in his e-mail header:
>=20
> X-MIMETrack: Serialize by Router on xxxx(Release
>         7.0.2FP2|May 14, 2007)... rest deleted

Interesting. That's exactly the version we use. Of course we wouldn't
notice this in production (the Domino servers don't deliver directly and
the smarthost doesn't greylist them, of course) but the issue should
have come up during pre-production tests.

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--8t9RHnE3ZwKMSgU+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--8t9RHnE3ZwKMSgU+--

Date: Sun, 18 Nov 2007 11:28:25 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Greylisting pitfalls (was Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation)
Sender: [email-masked]

Peter J. Holzer wrote:

> Just out of curiosity: Which version of Lotus was this?

This behavior was pointed out to me by a customer who
uses Lotus.  This is in his e-mail header:

X-MIMETrack: Serialize by Router on xxxx(Release
        7.0.2FP2|May 14, 2007)... rest deleted

Regards,

David.

Date: Sun, 18 Nov 2007 10:24:06 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: 2nd attempt Retry Time - Real World Examples
Sender: [email-masked]

Folks, you might find this interesting.

I wrote quick log analyzer to produce the domain vs 2nd retry time 
result. I ran the analyzer for my November 2007 logs only.

I left the duplicate domains (separate transactions) in the result to 
see if there is a constant or varying time.  I didn't bother to separate 
good or bad, or if they were forged domains. I will say, they all pass 
the initial (RCPT and MAIL) filtering test such as RBL, SPF, CallBack 
Verification.  They are sorted by domain, so the history of the 
transactions is not shown.

All these domains were rejected at DATA by our Greylist and then passed 
with their eventual 2nd attempt:

263.net                             | 10 min(s) (0.10 hrs)
BBIns.com                           | 9 min(s) (0.9 hrs)
BigDirectories.info                 | 10 min(s) (0.10 hrs)
IRS.gov                             | 1 min(s) (0.1 hrs)
IRS.gov                             | 219 min(s) (3.39 hrs)
Teradata.Com                        | 2 min(s) (0.2 hrs)
aadvantage.info.aa.com              | 456 min(s) (7.36 hrs)
aadvantage.info.aa.com              | 763 min(s) (12.43 hrs)
alsafeeradv.ae                      | 7 min(s) (0.7 hrs)
aol.com                             | 10 min(s) (0.10 hrs)
apc-overnight.com                   | 6 min(s) (0.6 hrs)
apc-overnight.com                   | 7 min(s) (0.7 hrs)
artemida.amu.edu.pl                 | 10 min(s) (0.10 hrs)
auction123.com                      | 10 min(s) (0.10 hrs)
audioadvice.com                     | 10 min(s) (0.10 hrs)
bangor.com                          | 15 min(s) (0.15 hrs)
bank-of-china.com                   | 82 min(s) (1.22 hrs)
bayfrontdeals.com                   | 10 min(s) (0.10 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 5 min(s) (0.5 hrs)
bellsouth.net                       | 7 min(s) (0.7 hrs)
bellsouth.net                       | 9 min(s) (0.9 hrs)
bounce.exacttarget.com              | 15 min(s) (0.15 hrs)
bounce.exacttarget.com              | 15 min(s) (0.15 hrs)
bounce.exacttarget.com              | 15 min(s) (0.15 hrs)
bounce.exacttarget.com              | 15 min(s) (0.15 hrs)
bounce.fplemail.com                 | 15 min(s) (0.15 hrs)
bounce.fplemail.com                 | 15 min(s) (0.15 hrs)
bounce.mkt32.net                    | 2 min(s) (0.2 hrs)
bounce.mkt32.net                    | 5 min(s) (0.5 hrs)
bounce.myngp.com                    | 20 min(s) (0.20 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
bouncejohnedwards.plusthree.com     | 1 min(s) (0.1 hrs)
btconnect.com                       | 1 min(s) (0.1 hrs)
cadenacope.net                      | 10 min(s) (0.10 hrs)
canadianpublication.net             | 6 min(s) (0.6 hrs)
captaris.com                        | 1 min(s) (0.1 hrs)
captaris.com                        | 1 min(s) (0.1 hrs)
cc.juniper.com                      | 492 min(s) (8.12 hrs)
cc.juniper.com                      | 492 min(s) (8.12 hrs)
cc.juniper.com                      | 497 min(s) (8.17 hrs)
ccmail.firstequitycard.com          | 1 min(s) (0.1 hrs)
centrum.cz                          | 196 min(s) (3.16 hrs)
cert.org                            | 174 min(s) (2.54 hrs)
cert.org                            | 183 min(s) (3.3 hrs)
cert.org                            | 33 min(s) (0.33 hrs)
cexpert.org                         | 15 min(s) (0.15 hrs)
cfew.org                            | 15 min(s) (0.15 hrs)
choicepointprg.net                  | 15 min(s) (0.15 hrs)
cie-online.net                      | 1415 min(s) (23.35 hrs)
ciscinc.com                         | 60 min(s) (1.0 hrs)
comcast.net                         | 10 min(s) (0.10 hrs)
comcast.net                         | 2 min(s) (0.2 hrs)
comcast.net                         | 2 min(s) (0.2 hrs)
comcast.net                         | 5 min(s) (0.5 hrs)
comcast.net                         | 5 min(s) (0.5 hrs)
cornerstonehealthcare.com           | 10 min(s) (0.10 hrs)
dbc.mtview.ca.us                    | 21 min(s) (0.21 hrs)
dbc.mtview.ca.us                    | 39 min(s) (0.39 hrs)
dbc.mtview.ca.us                    | 54 min(s) (0.54 hrs)
dbc.mtview.ca.us                    | 76 min(s) (1.16 hrs)
derbycitybbs.com                    | 5 min(s) (0.5 hrs)
dessault.com                        | 10 min(s) (0.10 hrs)
dhl.com                             | 1 min(s) (0.1 hrs)
dialogic.com                        | 1 min(s) (0.1 hrs)
doubleclick.net                     | 10 min(s) (0.10 hrs)
dsr.kvl.dk                          | 10 min(s) (0.10 hrs)
duqar.com                           | 2 min(s) (0.2 hrs)
e.delta.com                         | 1 min(s) (0.1 hrs)
e.delta.com                         | 1 min(s) (0.1 hrs)
ebay.com                            | 12 min(s) (0.12 hrs)
ebay.com                            | 3 min(s) (0.3 hrs)
ebay.com                            | 42 min(s) (0.42 hrs)
eim.ae                              | 5 min(s) (0.5 hrs)
elabs3.com                          | 40 min(s) (0.40 hrs)
ema10.net                           | 60 min(s) (1.0 hrs)
email.ingrammicro.com               | 20 min(s) (0.20 hrs)
email.ingrammicro.com               | 20 min(s) (0.20 hrs)
email.vzwshop.com                   | 15 min(s) (0.15 hrs)
emdeon.com                          | 14 min(s) (0.14 hrs)
emdeon.com                          | 26 min(s) (0.26 hrs)
emirates.net.ae                     | 5 min(s) (0.5 hrs)
emirates.net.ae                     | 5 min(s) (0.5 hrs)
epix.net                            | 1429 min(s) (23.49 hrs)
facebook.com                        | 10 min(s) (0.10 hrs)
familiar-holidays.com               | 830 min(s) (13.50 hrs)
faxwhiz.net                         | 1 min(s) (0.1 hrs)
faxwhiz.net                         | 1 min(s) (0.1 hrs)
fcservicesnetwork.com               | 94 min(s) (1.34 hrs)
fedmail.prime-vendor.com            | 30 min(s) (0.30 hrs)
ff.com                              | 10 min(s) (0.10 hrs)
fideliscare.org                     | 21 min(s) (0.21 hrs)
fideliscare.org                     | 23 min(s) (0.23 hrs)
foxriver.net                        | 1379 min(s) (22.59 hrs)
foxriver.net                        | 6 min(s) (0.6 hrs)
foxriver.net                        | 60 min(s) (1.0 hrs)
foxriver.net                        | 60 min(s) (1.0 hrs)
foxriver.net                        | 60 min(s) (1.0 hrs)
foxriver.net                        | 60 min(s) (1.0 hrs)
foxriver.net                        | 60 min(s) (1.0 hrs)
fsuimail.ferris.edu                 | 1 min(s) (0.1 hrs)
fsuimail.ferris.edu                 | 1 min(s) (0.1 hrs)
fsuimail.ferris.edu                 | 2 min(s) (0.2 hrs)
galaxybbs.net                       | 11 min(s) (0.11 hrs)
galaxybbs.net                       | 14 min(s) (0.14 hrs)
genius-network.com                  | 7 min(s) (0.7 hrs)
geno.no                             | 15 min(s) (0.15 hrs)
gmail.com                           | 104 min(s) (1.44 hrs)
gmail.com                           | 200 min(s) (3.20 hrs)
gmail.com                           | 226 min(s) (3.46 hrs)
gmail.com                           | 85 min(s) (1.25 hrs)
gsaemail.com                        | 1 min(s) (0.1 hrs)
gsaemail.com                        | 1 min(s) (0.1 hrs)
hanan.com                           | 10 min(s) (0.10 hrs)
hcl.in                              | 10 min(s) (0.10 hrs)
herbalink.com                       | 10 min(s) (0.10 hrs)
hesbynett.no                        | 31 min(s) (0.31 hrs)
honorgardz.com                      | 45 min(s) (0.45 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 1 min(s) (0.1 hrs)
hotmail.com                         | 6 min(s) (0.6 hrs)
hotmail.fr                          | 1 min(s) (0.1 hrs)
hotmail.fr                          | 1 min(s) (0.1 hrs)
hotmail.fr                          | 1 min(s) (0.1 hrs)
hotmail.fr                          | 1 min(s) (0.1 hrs)
hr11.net                            | 60 min(s) (1.0 hrs)
hs.utc.com                          | 139 min(s) (2.19 hrs)
hughes.net                          | 59 min(s) (0.59 hrs)
imatech.com.au                      | 10 min(s) (0.10 hrs)
in.confirmedcc.com                  | 16 min(s) (0.16 hrs)
in.confirmedcc.com                  | 18 min(s) (0.18 hrs)
in.confirmedcc.com                  | 20 min(s) (0.20 hrs)
in.confirmedcc.com                  | 25 min(s) (0.25 hrs)
in.confirmedcc.com                  | 25 min(s) (0.25 hrs)
in.confirmedcc.com                  | 29 min(s) (0.29 hrs)
in.constantcontact.com              | 141 min(s) (2.21 hrs)
in.constantcontact.com              | 37 min(s) (0.37 hrs)
in10.mailcompufinder.com            | 12 min(s) (0.12 hrs)
in16.mailcompufinder.com            | 14 min(s) (0.14 hrs)
in19.mailcompufinder.com            | 4 min(s) (0.4 hrs)
in7.mailcompufinder.com             | 8 min(s) (0.8 hrs)
in8.mailcompufinder.com             | 4 min(s) (0.4 hrs)
in9.mailcompufinder.com             | 9 min(s) (0.9 hrs)
irscounsel.treas.gov                | 10 min(s) (0.10 hrs)
jangomail.com                       | 15 min(s) (0.15 hrs)
jangomail.com                       | 15 min(s) (0.15 hrs)
jeeves.archives.listbox.com         | 102 min(s) (1.42 hrs)
jeeves.archives.listbox.com         | 102 min(s) (1.42 hrs)
jeeves.archives.listbox.com         | 103 min(s) (1.43 hrs)
jeeves.archives.listbox.com         | 104 min(s) (1.44 hrs)
jeeves.archives.listbox.com         | 110 min(s) (1.50 hrs)
jeeves.archives.listbox.com         | 129 min(s) (2.9 hrs)
jeeves.archives.listbox.com         | 135 min(s) (2.15 hrs)
jeeves.archives.listbox.com         | 146 min(s) (2.26 hrs)
jeeves.archives.listbox.com         | 147 min(s) (2.27 hrs)
jeeves.archives.listbox.com         | 149 min(s) (2.29 hrs)
jeeves.archives.listbox.com         | 160 min(s) (2.40 hrs)
jeeves.archives.listbox.com         | 18 min(s) (0.18 hrs)
jeeves.archives.listbox.com         | 18 min(s) (0.18 hrs)
jeeves.archives.listbox.com         | 19 min(s) (0.19 hrs)
jeeves.archives.listbox.com         | 24 min(s) (0.24 hrs)
jeeves.archives.listbox.com         | 39 min(s) (0.39 hrs)
jeeves.archives.listbox.com         | 42 min(s) (0.42 hrs)
jeeves.archives.listbox.com         | 43 min(s) (0.43 hrs)
jeeves.archives.listbox.com         | 47 min(s) (0.47 hrs)
jeeves.archives.listbox.com         | 48 min(s) (0.48 hrs)
jeeves.archives.listbox.com         | 51 min(s) (0.51 hrs)
jeeves.archives.listbox.com         | 52 min(s) (0.52 hrs)
jeeves.archives.listbox.com         | 59 min(s) (0.59 hrs)
jeeves.archives.listbox.com         | 60 min(s) (1.0 hrs)
jeeves.archives.listbox.com         | 60 min(s) (1.0 hrs)
jeeves.archives.listbox.com         | 61 min(s) (1.1 hrs)
jeeves.archives.listbox.com         | 63 min(s) (1.3 hrs)
jeeves.archives.listbox.com         | 66 min(s) (1.6 hrs)
jeeves.archives.listbox.com         | 67 min(s) (1.7 hrs)
jeeves.archives.listbox.com         | 67 min(s) (1.7 hrs)
jeeves.archives.listbox.com         | 69 min(s) (1.9 hrs)
jeeves.archives.listbox.com         | 71 min(s) (1.11 hrs)
jeeves.archives.listbox.com         | 74 min(s) (1.14 hrs)
jeeves.archives.listbox.com         | 75 min(s) (1.15 hrs)
jeeves.archives.listbox.com         | 76 min(s) (1.16 hrs)
jeeves.archives.listbox.com         | 77 min(s) (1.17 hrs)
jeeves.archives.listbox.com         | 77 min(s) (1.17 hrs)
jeeves.archives.listbox.com         | 78 min(s) (1.18 hrs)
jeeves.archives.listbox.com         | 91 min(s) (1.31 hrs)
jgocf.com                           | 15 min(s) (0.15 hrs)
jgocf.com                           | 15 min(s) (0.15 hrs)
kaptest.com                         | 10 min(s) (0.10 hrs)
katamail.com                        | 294 min(s) (4.54 hrs)
katamail.com                        | 33 min(s) (0.33 hrs)
kayakfloridakeys.com                | 3 min(s) (0.3 hrs)
ktc.co.th                           | 10 min(s) (0.10 hrs)
list.windowsitpro.com               | 15 min(s) (0.15 hrs)
list.windowsitpro.com               | 29 min(s) (0.29 hrs)
lists.infradead.org                 | 36 min(s) (0.36 hrs)
lsinews.com                         | 71 min(s) (1.11 hrs)
mag-ias.com                         | 80 min(s) (1.20 hrs)
magnetdev.com                       | 27 min(s) (0.27 hrs)
magnetdev.com                       | 40 min(s) (0.40 hrs)
magnetdev.com                       | 7 min(s) (0.7 hrs)
magnetdev.com                       | 9 min(s) (0.9 hrs)
mail.moviments.net                  | 7 min(s) (0.7 hrs)
mail136.subscribermail.com          | 20 min(s) (0.20 hrs)
mail136.subscribermail.com          | 40 min(s) (0.40 hrs)
mail2.zogby.com                     | 6 min(s) (0.6 hrs)
mail2.zogby.com                     | 7 min(s) (0.7 hrs)
mail2.zogby.com                     | 7 min(s) (0.7 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mailer.hillaryclinton.com           | 15 min(s) (0.15 hrs)
mapurdy.com.au                      | 15 min(s) (0.15 hrs)
mapurdy.com.au                      | 15 min(s) (0.15 hrs)
mapurdy.com.au                      | 15 min(s) (0.15 hrs)
mastercard.com                      | 21 min(s) (0.21 hrs)
mastercard.com                      | 79 min(s) (1.19 hrs)
mcnally.com.au                      | 10 min(s) (0.10 hrs)
me19609.mailengine1.com             | 10 min(s) (0.10 hrs)
mexicoverde.com                     | 231 min(s) (3.51 hrs)
mgcomunicaciones.com.mx             | 154 min(s) (2.34 hrs)
monterreal.com                      | 10 min(s) (0.10 hrs)
montreal.servershost.net            | 215 min(s) (3.35 hrs)
moveher.com                         | 41 min(s) (0.41 hrs)
mpo.fr                              | 10 min(s) (0.10 hrs)
mtbone.no-ip.net                    | 5 min(s) (0.5 hrs)
mx2.ucg.com                         | 1 min(s) (0.1 hrs)
mx2.ucg.com                         | 1 min(s) (0.1 hrs)
mycybernet.net                      | 39 min(s) (0.39 hrs)
myjaring.net                        | 11 min(s) (0.11 hrs)
netmonks.ca                         | 30 min(s) (0.30 hrs)
netpractice.com                     | 1 min(s) (0.1 hrs)
newkirk.com                         | 1 min(s) (0.1 hrs)
newsletters.microsoft.com           | 1 min(s) (0.1 hrs)
newsletters.microsoft.com           | 1 min(s) (0.1 hrs)
objectwatch.com                     | 1 min(s) (0.1 hrs)
online.networkworld.info            | 30 min(s) (0.30 hrs)
online.networkworld.info            | 31 min(s) (0.31 hrs)
online.networkworld.info            | 31 min(s) (0.31 hrs)
online.networkworld.info            | 31 min(s) (0.31 hrs)
online.networkworld.info            | 32 min(s) (0.32 hrs)
online.networkworld.info            | 34 min(s) (0.34 hrs)
ontheflydj.com                      | 6 min(s) (0.6 hrs)
ontheflydj.com                      | 6 min(s) (0.6 hrs)
onyxgfx.com                         | 10 min(s) (0.10 hrs)
optonline.net                       | 30 min(s) (0.30 hrs)
p15105247.pureserver.info           | 340 min(s) (5.40 hrs)
paradise.net.nz                     | 15 min(s) (0.15 hrs)
paradise.net.nz                     | 15 min(s) (0.15 hrs)
playabeachrentals.com               | 10 min(s) (0.10 hrs)
pollackrosen.com                    | 5 min(s) (0.5 hrs)
positive-feeding.com                | 469 min(s) (7.49 hrs)
post.fukubiki.com                   | 10 min(s) (0.10 hrs)
prideinfamily.com                   | 31 min(s) (0.31 hrs)
rdb.ap-hop-paris.fr                 | 10 min(s) (0.10 hrs)
rdmo.com                            | 10 min(s) (0.10 hrs)
remotequity.com                     | 10 min(s) (0.10 hrs)
reply.newsmax.com                   | 10 min(s) (0.10 hrs)
reply.newsmax.com                   | 10 min(s) (0.10 hrs)
reply.newsmax.com                   | 10 min(s) (0.10 hrs)
reply.newsmax.com                   | 12 min(s) (0.12 hrs)
repsco.com                          | 612 min(s) (10.12 hrs)
returns.groups.yahoo.com            | 10 min(s) (0.10 hrs)
returns.groups.yahoo.com            | 1064 min(s) (17.44 hrs)
returns.groups.yahoo.com            | 1194 min(s) (19.54 hrs)
returns.groups.yahoo.com            | 24 min(s) (0.24 hrs)
returns.groups.yahoo.com            | 26 min(s) (0.26 hrs)
returns.groups.yahoo.com            | 798 min(s) (13.18 hrs)
reu.org                             | 60 min(s) (1.0 hrs)
reu.org                             | 60 min(s) (1.0 hrs)
reu.org                             | 60 min(s) (1.0 hrs)
reu.org                             | 60 min(s) (1.0 hrs)
sable.adelphi.edu                   | 74 min(s) (1.14 hrs)
sage.com                            | 5 min(s) (0.5 hrs)
salesdiesel.com                     | 1 min(s) (0.1 hrs)
san.rr.com                          | 61 min(s) (1.1 hrs)
sbcglobal.net                       | 6 min(s) (0.6 hrs)
solberglaw.com                      | 10 min(s) (0.10 hrs)
speedy.com.ar                       | 21 min(s) (0.21 hrs)
standardandpoors.com                | 1 min(s) (0.1 hrs)
stiedprmman1.ietf.org               | 19 min(s) (0.19 hrs)
stoncanyon.com                      | 20 min(s) (0.20 hrs)
stoncanyon.com                      | 39 min(s) (0.39 hrs)
sub4-net.net                        | 15 min(s) (0.15 hrs)
suboletinmx.com                     | 1 min(s) (0.1 hrs)
subscription.networkworld.info      | 31 min(s) (0.31 hrs)
subscription.networkworld.info      | 31 min(s) (0.31 hrs)
sumon.com                           | 10 min(s) (0.10 hrs)
suntecsingapore.com                 | 10 min(s) (0.10 hrs)
sunwestmanagement.com               | 10 min(s) (0.10 hrs)
takacho.com                         | 2 min(s) (0.2 hrs)
takacho.com                         | 2 min(s) (0.2 hrs)
talismaonline.com                   | 10 min(s) (0.10 hrs)
talismaonline.com                   | 10 min(s) (0.10 hrs)
telcos.net.ar                       | 22 min(s) (0.22 hrs)
terra.com.ar                        | 10 min(s) (0.10 hrs)
thedatabank.com                     | 2 min(s) (0.2 hrs)
thedatabank.com                     | 3 min(s) (0.3 hrs)
tiscomhosting.nl                    | 133 min(s) (2.13 hrs)
trackstick.com                      | 1 min(s) (0.1 hrs)
trackstick.com                      | 1 min(s) (0.1 hrs)
trackstick.com                      | 1 min(s) (0.1 hrs)
uasom.uab.edu                       | 10 min(s) (0.10 hrs)
ucla.edu                            | 0 min(s) (0.0 hrs)
ucla.edu                            | 1 min(s) (0.1 hrs)
uhifl.com                           | 10 min(s) (0.10 hrs)
univ-reunion.fr                     | 10 min(s) (0.10 hrs)
ups.com                             | 10 min(s) (0.10 hrs)
us.army.mil                         | 1 min(s) (0.1 hrs)
uscellular.com                      | 10 min(s) (0.10 hrs)
usxpress.com                        | 1 min(s) (0.1 hrs)
usxpress.com                        | 1 min(s) (0.1 hrs)
usxpress.com                        | 1 min(s) (0.1 hrs)
velcom.ca                           | 3 min(s) (0.3 hrs)
verizon.com                         | 48 min(s) (0.48 hrs)
verizon.net                         | 60 min(s) (1.0 hrs)
verizon.net                         | 60 min(s) (1.0 hrs)
ville-bagnolet.fr                   | 10 min(s) (0.10 hrs)
virginia.edu                        | 19 min(s) (0.19 hrs)
voyager.net                         | 19 min(s) (0.19 hrs)
voyager.net                         | 9 min(s) (0.9 hrs)
wackerwhippets.com                  | 9 min(s) (0.9 hrs)
wackerwhippets.com                  | 9 min(s) (0.9 hrs)
washpost.com                        | 1 min(s) (0.1 hrs)
wayne.com                           | 2 min(s) (0.2 hrs)
webmarketingdealsonline.com         | 22 min(s) (0.22 hrs)
webnonerononcam.superfoon.nl        | 164 min(s) (2.44 hrs)
weightonesec.com                    | 35 min(s) (0.35 hrs)
weightonesec.com                    | 42 min(s) (0.42 hrs)
weightonesec.com                    | 42 min(s) (0.42 hrs)
weightonesec.com                    | 43 min(s) (0.43 hrs)
weightonesec.com                    | 50 min(s) (0.50 hrs)
weightonesec.com                    | 60 min(s) (1.0 hrs)
weightonesec.com                    | 69 min(s) (1.9 hrs)
west.net                            | 18 min(s) (0.18 hrs)
wittendorp.nl                       | 10 min(s) (0.10 hrs)
xilinx.com                          | 1 min(s) (0.1 hrs)
xilinx.com                          | 1 min(s) (0.1 hrs)
xilinx.com                          | 1 min(s) (0.1 hrs)
yahoo.com                           | 10 min(s) (0.10 hrs)
yahoo.com                           | 2 min(s) (0.2 hrs)
yahoo.com                           | 2 min(s) (0.2 hrs)
yahoo.com                           | 35 min(s) (0.35 hrs)
yahoo.com                           | 6 min(s) (0.6 hrs)
yahoo.com                           | 6 min(s) (0.6 hrs)
yahoo.com.ar                        | 17 min(s) (0.17 hrs)
yahoo.com.mx                        | 1 min(s) (0.1 hrs)
yahoo.com.mx                        | 1 min(s) (0.1 hrs)
yahoo.it                            | 207 min(s) (3.27 hrs)
yourhostingaccount.com              | 15 min(s) (0.15 hrs)
yourhostingaccount.com              | 17 min(s) (0.17 hrs)
zombie.ab-space.de                  | 101 min(s) (1.41 hrs)

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Hector Santos wrote:
> 
> John C Klensin wrote:
> 
>> --On Saturday, 17 November, 2007 21:54 -0500 "Robert A.
> 
>>> You need at least 4 days to handle 3 day weekends (a holiday
>>> on Friday or Monday) although I've seen 4 day Holidays (Xmas
>>> and New Years on a Friday and the company shutting down at
>>> Noon on Thursday). A better idea is to use 3 days BUT adjust
>>> it to 4 or 5 days when there is a Friday/Monday holiday
>>> (although that can still run into problems since it is US
>>> Centric and can run afoul of Non-US Holidays).
>>
>> And that is exactly why the guidance in the spec says what it
>> does.
> 
> Yes.
> 
> Please keep in mind that my suggestion is specifically regarding the 2nd 
>  and possibly 3rd retry but mainly the 2nd one, in regards in minimizing 
> the impact related to the growth Challenge/Responses-based systems.
> 
> Going beyond begins to get into subjective and complex considerations, 
> and I don't believe you want to go there for 2821bis. Me too.
> 
> Even though I believe that most systems already have shorter initial 
> intervals, all I am suggesting is a simple clause or sentence to give 
> "hindsight" to RFC 2821 readers and future developers regarding a 
> shorter 2nd attempt.
> 
> Based on what I read in 2821bis (I have rev3), it goes into great length 
> regardng reasons why the stragety may be altered, in particular this 
> paragraph:
> 
>    Experience suggests that failures are typically transient (the target
>    system or its connection has crashed), favoring a policy of two
>    connection attempts in the first hour the message is in the queue,
>    and then backing off to one every two or three hours.
> 
> Well, I think there is adequate global experience today that failure may 
> includss challenge/response systems, not just necessarily downed systems 
> or connections.  I know anti-spam ideas is a taboo for you in RFC 
> 2821bis, so maybe someone can finesse some sentence there to include 
> this consideration without actually getting into a AVS relationships?
> 
> Maybe?
> 
>    In addition, experience suggest that a SMTP receiver may have
>    a local policy which will temporarily reject the first initial
>    delivery attempt by anonymous SMTP senders using a 451 response
>    code from with an expectation the SMTP client will
>    retry successfully.  For a 451 response, the SMTP client MAY
>    consider using a shorter 2nd retry attempt to minimize the
>    delivery impact when connecting  to these types of remote systems.
> 
> I was thinking of specifically saying 451 only. That way, it won't 
> effect 450, 452 operations. Otherwise, make it 45x.
> 
> Or something simpler:
> 
>    A SMTP client MAY consider shortening the 2nd attempt ONLY to
>    5-10 minutes for 45z responses to RCPT TO or DATA.  For failed
>    connections, the SMTP client SHOULD continue with the
>    suggestions outlined in this section.
> 

Date: Sun, 18 Nov 2007 07:08:28 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: "Robert A. Rosenberg" <[email-masked]>, [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy     recommendation
Sender: [email-masked]

John C Klensin wrote:

> --On Saturday, 17 November, 2007 21:54 -0500 "Robert A.

>> You need at least 4 days to handle 3 day weekends (a holiday
>> on Friday or Monday) although I've seen 4 day Holidays (Xmas
>> and New Years on a Friday and the company shutting down at
>> Noon on Thursday). A better idea is to use 3 days BUT adjust
>> it to 4 or 5 days when there is a Friday/Monday holiday
>> (although that can still run into problems since it is US
>> Centric and can run afoul of Non-US Holidays).
> 
> And that is exactly why the guidance in the spec says what it
> does.

Yes.

Please keep in mind that my suggestion is specifically regarding the 2nd 
  and possibly 3rd retry but mainly the 2nd one, in regards in 
minimizing the impact related to the growth Challenge/Responses-based 
systems.

Going beyond begins to get into subjective and complex considerations, 
and I don't believe you want to go there for 2821bis. Me too.

Even though I believe that most systems already have shorter initial 
intervals, all I am suggesting is a simple clause or sentence to give 
"hindsight" to RFC 2821 readers and future developers regarding a 
shorter 2nd attempt.

Based on what I read in 2821bis (I have rev3), it goes into great length 
regardng reasons why the stragety may be altered, in particular this 
paragraph:

    Experience suggests that failures are typically transient (the target
    system or its connection has crashed), favoring a policy of two
    connection attempts in the first hour the message is in the queue,
    and then backing off to one every two or three hours.

Well, I think there is adequate global experience today that failure may 
includss challenge/response systems, not just necessarily downed systems 
or connections.  I know anti-spam ideas is a taboo for you in RFC 
2821bis, so maybe someone can finesse some sentence there to include 
this consideration without actually getting into a AVS relationships?

Maybe?

    In addition, experience suggest that a SMTP receiver may have
    a local policy which will temporarily reject the first initial
    delivery attempt by anonymous SMTP senders using a 451 response
    code from with an expectation the SMTP client will
    retry successfully.  For a 451 response, the SMTP client MAY
    consider using a shorter 2nd retry attempt to minimize the
    delivery impact when connecting  to these types of remote systems.

I was thinking of specifically saying 451 only. That way, it won't 
effect 450, 452 operations. Otherwise, make it 45x.

Or something simpler:

    A SMTP client MAY consider shortening the 2nd attempt ONLY to
    5-10 minutes for 45z responses to RCPT TO or DATA.  For failed
    connections, the SMTP client SHOULD continue with the
    suggestions outlined in this section.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sun, 18 Nov 2007 12:01:53 +0000
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

[email-masked] wrote:

>> John Leslie writes:
>> >    It may be worth noting that sender address rewrite should not be
>> >    done when forwarding a message via TBR.
>> >
>> >    (I can't offhand think of a reason you'd actually want to...)
>
>> Sieve processors executing redirect statement are permitted to change
>> the MAIL FROM, or leave it.
>
> With one important exception: If the return address is empty you're not
> supposed to fill it in. If you do you can end up creating a bounce loop.
>
> I've been wondering if this restriction/concern needs to be mentioned in
> 2821bis.

I think it should.

Date: Sun, 18 Nov 2007 00:54:04 -0800
To: John C Klensin <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Cc: [email-masked], [email-masked]
Sender: [email-masked]

Hi John,
At 18:51 17-11-2007, [email-masked] wrote:
>Agreed. But it needs to be clear that this applies to autoforwarders. How
>about adding "and SHOULD NOT add a non-null reverse path to such a message
>when forwarding".

I suggest the following change to Section 4.5.5 (Messages with a null 
reverse-path):

     If a reverse-path is null, it SHOULD NOT be modified as that is likely
     to cause a mail loop.

     Implementors of automated email processors should ensure that
     the various kinds of messages with a null reverse-path are handled
     correctly, in particular such systems SHOULD NOT reply to messages
     with a null reverse-path.

Regards,
-sm 

Date: Sun, 18 Nov 2007 03:08:38 -0500
From: John C Klensin <[email-masked]>
To: "Robert A. Rosenberg" <[email-masked]>, [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy     recommendation
Sender: [email-masked]

--On Saturday, 17 November, 2007 21:54 -0500 "Robert A.
Rosenberg" <[email-masked]> wrote:

> 
> At 00:20 -0800 on 11/17/2007, SM wrote about Re: 2821bis
> consideration - New 2nd attempt Retry Strategy :
> 
>> It's only a lot of retries over four or five days if you
>> retry too  frequently.  Some view the period as too long.  A
>> two-day give-up  period, for example, may result in delivery
>> issues being missed if  it falls over a weekend
> 
> You need at least 4 days to handle 3 day weekends (a holiday
> on Friday or Monday) although I've seen 4 day Holidays (Xmas
> and New Years on a Friday and the company shutting down at
> Noon on Thursday). A better idea is to use 3 days BUT adjust
> it to 4 or 5 days when there is a Friday/Monday holiday
> (although that can still run into problems since it is US
> Centric and can run afoul of Non-US Holidays).

And that is exactly why the guidance in the spec says what it
does.

   john

Date: Sun, 18 Nov 2007 00:44:37 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Fixing Greylist - Its usage of 451 vs 450
Sender: [email-masked]

Hi SM,

All I am stating is, if 451 is presumed incorrect (wrt to GL) for 
literal reasons, then so would be the others as well for literal reasons.

However, I believe you are saying the exception to the literal reason 
rule would be the "catch all" 450 response code when the reject reasons 
does not fit 451 or 452 because those two codes are deemed sacred for 
those two "finer" specific server rejection literal reasons only.

If so, then, you're right. :-)

But I can see why maybe the GL author probably chose 451:

  - The RFC prohibits creating new reply codes,

  - Not all systems support extended codes, making it harder to
    justify using 450,

  - 450 and 452 have more specific reasons one can relate to,

  - 451 has an "open ended" non-specific interpretation,

  - 451 seems to be reserved for the server admission that
    it just experienced a backend server problem but it is
    deemed to be temporary, i.e, its not the sender's fault,
    therefore it should try again,

  - Maybe 451 "Error In Processing" is what he wanted to
    tell the bad guy what is going on, because the good guy
    system is going to try again, so no problem.

But who knows? :-)

--
HLS

SM wrote:
> Hi Hector,
> At 04:37 17-11-2007, Hector Santos wrote:
>> But who says the z=1 in 45z is "error in processing."?
> 
> Quoting RFC 2821:
> 
>   "The third digit gives a finer gradation of meaning in each category
>    specified by the second digit.  The list of replies illustrates this.
>    Each reply text is recommended rather than mandatory, and may even
>    change according to the command with which it is associated.  On the
>    other hand, the reply codes must strictly follow the specifications
>    in this section.  Receiver implementations should not invent new
>    codes for slightly different situations from the ones described here,
>    but rather adapt codes already defined."
> 
>> There is nothing in the specs that officially states z=1 (3rd digit)
> 
> See the above text about "finer graduation".
> 
>> technically means is a error in processing.  If we do that, then it has
>> to equally apply to all the other response codes and their various ways
>> they are uses.  For example:
>>
>>       551 User not local; please try <forward-path>
>>       421 <domain> Service not available, closing transmission channel
>>       251 User not local; will forward to <forward-path>
>>       551 User not local; please try <forward-path>
>>
>> All these commonly used z=1 responses are not use for "error in
>> processing" ideas.
> 
> I'm not saying that z=1 means "error in processing".  There's a subtile 
> difference between the theory of reply codes and practice.
> 
>> The only thing that is important are the first two digits.
> 
> Yes.
> 
>> To me, a sender, which I presume to mean a "SMTP client software",
>> doesn't read Japanese, Chinese, Spanish, Spanglish, and even English.
>> It reads SMTP and in terms of the state machine, the first two digit 
>> is what matters.
> 
> There's also SMTP operations.  The SMTP client software only deals with 
> machine state, the text is for us to decipher.
> 
>> So I am having a difficult time seeing how it can be "technically 
>> incorrect" if the design requirement for correct state machine 
>> automation is the 45x reading.  Nothing more, no extended codes, no 
>> literals.  That extra stuff is for humans. No?
> 
> The extra stuff is for humans.
> 
>> Also, to me, if we wanted to take it literally, error in processing 
>> should be a permanent rejection idea.  Why?  We had this discussion in 
>> length in regards when 55x or 45x is appropriate and I think we 
>> establish if the data is not going to change, providing the same input 
>> on new tries is not going change the state. So one can argue that the 
>> literal example for 451 is erroneous.  It should be for some realistic 
>> "temporary reason" other then error in processing. Error in processing 
>> what? Your data?  If so, it should not ask again to retry.
> 
> There are temporary and permanent error conditions.  If there is an 
> error in processing (451), it doesn't mean that the message won't be 
> processed successful on the next retry.  A permanent rejection causes a 
> breakdown in the communication channel.
> 
> Regards,
> -sm
> 
> 
> 

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sat, 17 Nov 2007 21:54:23 -0500
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy  recommendation
Sender: [email-masked]

At 00:20 -0800 on 11/17/2007, SM wrote about Re: 2821bis 
consideration - New 2nd attempt Retry Strategy :

>It's only a lot of retries over four or five days if you retry too 
>frequently.  Some view the period as too long.  A two-day give-up 
>period, for example, may result in delivery issues being missed if 
>it falls over a weekend

You need at least 4 days to handle 3 day weekends (a holiday on 
Friday or Monday) although I've seen 4 day Holidays (Xmas and New 
Years on a Friday and the company shutting down at Noon on Thursday). 
A better idea is to use 3 days BUT adjust it to 4 or 5 days when 
there is a Friday/Monday holiday (although that can still run into 
problems since it is US Centric and can run afoul of Non-US Holidays).

From: [email-masked]
Cc: John C Klensin <[email-masked]>, [email-masked], [email-masked]
Date: Sat, 17 Nov 2007 18:51:08 -0800 (PST)
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting  [was TBR])
To: SM <[email-masked]>
Sender: [email-masked]

> Hi John,
> At 08:53 17-11-2007, John C Klensin wrote:
> >I'm working this weekend on what I hope will be the last version of
> >2821bis prior to IETF Last Call, so, if you think such a mention
> >would be useful, please propose text and where it should go.

> I have not come across any cases where the empty return path is
> rewritten.

I've seen it happen quite a few times, although not so much recently. I'm
pretty sure Exchange used to do it unconditionally when autoforwarding. I'm not
sure if it still does.

> If Ned thinks it might be a problem, some text could be added.

Well, the only reason I even found out that some agents do it is because we've
had customers have meltdowns because of it. The sorts of loop this causes
usually have messages that grow continually in size and in many cases the
number of messages in the loop grows geometrically over time.

> In Section 4.1.1.2:

>    "In some types of reporting messages for which a reply is likely to cause a
>     mail loop (for example, mail delivery and nondelivery notifications), the
>     reverse-path may be null (see Section 3.6)."

> That provides some guidance on how to avoid a mail loop.

> In Section 4.5.5, Messages with a null reverse-path:

>    "Implementors of automated email processors should be careful to make
>     sure that the various kinds of messages with null reverse-path are
>     handled correctly, in particular such systems SHOULD NOT reply to
>     messages with null reverse-path."

> I read that as a warning about the null reverse-path, i.e. not to
> rewrite the address or else the message may generate a reply.

Agreed. But it needs to be clear that this applies to autoforwarders. How
about adding "and SHOULD NOT add a non-null reverse path to such a message
when forwarding".

I also note that there's a missing "a" in the above, it should read "with a
null reverse-path".

				Ned

Date: Sat, 17 Nov 2007 16:00:17 -0800
To: John C Klensin <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Cc: [email-masked]
Sender: [email-masked]

Hi John,
At 08:53 17-11-2007, John C Klensin wrote:
>I'm working this weekend on what I hope will be the last version of 
>2821bis prior to IETF Last Call, so, if you think such a mention 
>would be useful, please propose text and where it should go.

I have not come across any cases where the empty return path is 
rewritten.  If Ned thinks it might be a problem, some text could be added.

In Section 4.1.1.2:

   "In some types of reporting messages for which a reply is likely to cause a
    mail loop (for example, mail delivery and nondelivery notifications), the
    reverse-path may be null (see Section 3.6)."

That provides some guidance on how to avoid a mail loop.

In Section 4.5.5, Messages with a null reverse-path:

   "Implementors of automated email processors should be careful to make
    sure that the various kinds of messages with null reverse-path are
    handled correctly, in particular such systems SHOULD NOT reply to
    messages with null reverse-path."

I read that as a warning about the null reverse-path, i.e. not to 
rewrite the address or else the message may generate a reply.

Regards,
-sm 

Date: Sat, 17 Nov 2007 14:56:03 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 04:37 17-11-2007, Hector Santos wrote:
>But who says the z=1 in 45z is "error in processing."?

Quoting RFC 2821:

   "The third digit gives a finer gradation of meaning in each category
    specified by the second digit.  The list of replies illustrates this.
    Each reply text is recommended rather than mandatory, and may even
    change according to the command with which it is associated.  On the
    other hand, the reply codes must strictly follow the specifications
    in this section.  Receiver implementations should not invent new
    codes for slightly different situations from the ones described here,
    but rather adapt codes already defined."

>There is nothing in the specs that officially states z=1 (3rd digit)

See the above text about "finer graduation".

>technically means is a error in processing.  If we do that, then it has
>to equally apply to all the other response codes and their various ways
>they are uses.  For example:
>
>       551 User not local; please try <forward-path>
>       421 <domain> Service not available, closing transmission channel
>       251 User not local; will forward to <forward-path>
>       551 User not local; please try <forward-path>
>
>All these commonly used z=1 responses are not use for "error in
>processing" ideas.

I'm not saying that z=1 means "error in processing".  There's a 
subtile difference between the theory of reply codes and practice.

>The only thing that is important are the first two digits.

Yes.

>To me, a sender, which I presume to mean a "SMTP client software",
>doesn't read Japanese, Chinese, Spanish, Spanglish, and even English.
>It reads SMTP and in terms of the state machine, the first two digit 
>is what matters.

There's also SMTP operations.  The SMTP client software only deals 
with machine state, the text is for us to decipher.

>So I am having a difficult time seeing how it can be "technically 
>incorrect" if the design requirement for correct state machine 
>automation is the 45x reading.  Nothing more, no extended codes, no 
>literals.  That extra stuff is for humans. No?

The extra stuff is for humans.

>Also, to me, if we wanted to take it literally, error in processing 
>should be a permanent rejection idea.  Why?  We had this discussion 
>in length in regards when 55x or 45x is appropriate and I think we 
>establish if the data is not going to change, providing the same 
>input on new tries is not going change the state. So one can argue 
>that the literal example for 451 is erroneous.  It should be for 
>some realistic "temporary reason" other then error in processing. 
>Error in processing what? Your data?  If so, it should not ask again to retry.

There are temporary and permanent error conditions.  If there is an 
error in processing (451), it doesn't mean that the message won't be 
processed successful on the next retry.  A permanent rejection causes 
a breakdown in the communication channel.

Regards,
-sm

Date: Sat, 17 Nov 2007 22:48:48 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--6Vw0j8UKbyX0bfpA
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-13 18:32:49 -0800, Douglas Otis wrote:
> For ISP that host customers sending out a spam campaign, their =20
> servers may receive a temp error at that point in time.  =20
> Unfortunately, this may cause some desired email to be delayed.  Once =20
> spamming from that server ceases, normal acceptance is restored.  =20
> This dynamic reputation tactic blocks often more than twice the =20
> amount of spam stopped by static lists.  Graduated dynamic lists are =20
> available to suit the needs of different receivers.
>=20
> Blocking an influx of spam using the temp error tactic frees up =20
> resources needed to process other (cleaner) sources of email.  =20
> Simultaneously offering the TBR extension within the 451 4.7.8/9/10 =20
> temp error would provide the transmitter a means to immediately =20
> complete their transaction by reverting to the TBR mode.  (Fewer =20
> transactions and less delay.)

If I understood this correctly an SMTP server supporting TBR may return
a 451 return code with any of the 4.7.8-10 enhanced status codes if it
is currently unable (or unwilling) to accept email content (e.g.,
because of high load, a full queue, etc.) but is able to accept a TBR
command. A client supporting TBR may then store the mail content on a
webserver and send the TBR command with the URI:

C: EHLO client.example.com
S: 250-server.example.net
S: 250-SIZE 20971520
S: 250 TBR
C: MAIL FROM:<[email-masked]> SIZE=3D10000000
S: 250 Ok
C: RCPT TO:<[email-masked]>
S: 250 Ok
C: DATA
S: 451 4.7.10 Load too high - try TBR HTTPS mode

(client saves the 10 MB message on _tbr.example.com)

C: TBR 0 https://_tbr.example.com/1234567890abcdef
S: 250 Ok

I didn't understand this while reading the draft. You may want to
clarify section 6.

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--6Vw0j8UKbyX0bfpA
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--6Vw0j8UKbyX0bfpA--

Date: Sat, 17 Nov 2007 19:45:16 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--rqzD5py0kzyFAOWN
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-17 16:38:43 +0100, Arnt Gulbrandsen wrote:
> John Leslie writes:
> >   It was my understanding that sieve is a "language for filtering
> >e-mail messages at time of final delivery". If so, the URI fetching=20
> >and message regeneration will already have been done. So I don't see=20
> >a problem here.
>=20
> Redirect causes a second final delivery to take place, and may change=20
> the envelope sender between the first and second deliveries.

In any case the content of the mail need to be retrieved before the
first delivery, so the original URL cannot be used for the second
delivery. If the path between the first and second delivery also uses
TBR the mail needs to stored on a second server and the envelope sender
needs to be changed to reflect that. So there is no problem.

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--rqzD5py0kzyFAOWN
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--rqzD5py0kzyFAOWN--

Date: Sat, 17 Nov 2007 19:37:16 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--sHrvAb52M6C8blB9
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-17 07:13:02 -0500, Hector Santos wrote:
> Also, from what I have seen, top ISPs are also well below 30 mins 2nd=20
> retry. In fact for Bellsouth.net, it has a 5 minute 2nd retry. But AT&T=
=20
> took them over, so lets see what  att.net servers are doing. ..... Yup,=
=20
> exactly 5 mins 2nd retry which is what I saw (<30 mins) by the majority=
=20
>  when I was looking at the time.
>=20
> IMO, 30 minutes is really too high and IMO, it corresponds to the good=20
> old days of less reliable machines, more down systems, less reliable=20
> connections, slower bandwidths, less scaled systems, etc, etc. But not=20
> today. With high performance machines, multi-threading software, Virtual=
=20
> Machines, higher bandwidths, better scaling, more challenge/response=20
> systems, etc, may suggest the x821 30 minute retry "guideline" is=20
> outdated.  :-)

I would agree with that. Temporary error conditions (real or simulated)
which are fixed in far less than 30 minutes are rather frequent these
days, and combining a short initial delay of a few minutes with an
exponential backoff to an upper limit of a few hours seems to be rather
common.

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--sHrvAb52M6C8blB9
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--sHrvAb52M6C8blB9--

Date: Sat, 17 Nov 2007 19:26:53 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Greylisting pitfalls (was Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--UHN/qo2QbUvPLonB
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-17 09:37:27 -0500, David F. Skoll wrote:
> Tempfailing RCPT commands will cause interoperability problems with certa=
in
> broken SMTP implementations, specifically ones from Lotus.  Here's what
> happens.  S is the server doing greylisting and C is the broken Lotus cli=
ent:
>=20
> S: 220 foo.example.com ESMTP
> C: HELO busted.lotus.box
> S: 250 foo.example.com Hello busted.lotus.box
> C: MAIL FROM:<[email-masked]>
> S: 250 Go ahead
> C: RCPT TO:<[email-masked]>
> S: 451 Greylisting in effect, try again later
> C: RCPT TO:<[email-masked]>
> S: 451 Greylisting in effect, try again later
> C: DATA
> S: 503 Need valid recipient first
> C: QUIT
>=20
> at which point busted.lotus.box generates a non-delivery notification.

Just out of curiosity: Which version of Lotus was this? We've been using
Greylisting for more than 4 years now and Lotus Domino so far hasn't
made it onto my list of software which doesn't cope with greylisting
(Even though we use it ourselves). I do know behaviour such as you
describe from Novell Groupwise ...

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--UHN/qo2QbUvPLonB
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--UHN/qo2QbUvPLonB--

Date: Sat, 17 Nov 2007 19:20:45 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--envbJBWh7q8WU6mo
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-11-17 07:37:49 -0500, Hector Santos wrote:
> SM wrote:
> >It's technically incorrect.  It's not an incompatibility problem.  A=20
> >sender should to be able to differentiate between a temporary rejection=
=20
> >due to an error and one due to the receiver's policy decision.=20
>=20
> Hi SM,
>=20
> But who says the z=3D1 in 45z is "error in processing."?
>=20
> There is nothing in the specs that officially states z=3D1 (3rd digit)
> technically means is a error in processing.

The 3 digits in an SMTP response code aren't independent. z=3D1 in 45z
means "Requested action aborted: local error in processing" (RFC 2821
says so), but that doesn't mean that z=3D1 for any other xyz means the
same.=20

> IMO, log analyzers can not 100% depend on any 3rd digit to mean anything=
=20
> specific.
>=20
> Also, to me, if we wanted to take it literally, error in processing=20
> should be a permanent rejection idea.

Why? Temporary errors do occur.

> Why?  We had this discussion in length in regards when 55x or 45x is
> appropriate and I think we establish if the data is not going to
> change, providing the same input on new tries is not going change the
> state.

This is simply not true. Something on the server may have changed (maybe
somebody restarted the LDAP server which was down, or cleaned out the
full spool directory, or whatever) and then the same request by the
client can be processed successfully. This is exactly the difference
between 4xx and 5xx replies.

	hp

--=20
   _  | Peter J. Holzer    | It took a genius to create [TeX],
|_|_) | Sysadmin WSR       | and it takes a genius to maintain it.
| |   | [email-masked]         | That's not engineering, that's art.
__/   | http://www.hjp.at/ |	-- David Kastrup in comp.text.tex

--envbJBWh7q8WU6mo
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--envbJBWh7q8WU6mo--

Date: Sat, 17 Nov 2007 11:53:26 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], John Leslie <[email-masked]>, [email-masked]
Subject: 2821bis and address rewriting (was: Re: Fixing graylisting [was TBR])
Sender: [email-masked]

--On Saturday, November 17, 2007 6:57 AM -0800 
[email-masked] wrote:

>> Sieve processors executing redirect statement are permitted
>> to change the MAIL FROM, or leave it.
>
> With one important exception: If the return address is empty
> you're not
> supposed to fill it in. If you do you can end up creating a
> bounce loop.
>
> I've been wondering if this restriction/concern needs to be
> mentioned in 2821bis.

I'm working this weekend on what I hope will be the last version 
of 2821bis prior to IETF Last Call, so, if you think such a 
mention would be useful, please propose text and where it should 
go.

   john

Date: Sat, 17 Nov 2007 08:36:05 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 04:13 17-11-2007, Hector Santos wrote:
>Care to share a few?

We already discussed that in a previous thread some time back.  The 
sender's address or the IP address can be different on the retry.

>We do have a very aggressive system, and thats what they want. 
>Wouldn't have any other way.  We all don't have to be the same, right?

Right. :-)

>Its only high if its problem across the board. I'm not going to 
>worry about the few (Pareto principle).  Also, 4 vs 2 is not much a 
>difference, but what does high translate too? Overhead? For whom? 
>Are we dealing with a down system? a session level 45x reject?

It may require more resources, for the sender, if the host was 
down.  The system may have to catch up while keeping up with current 
messages being queued.  The sending strategy should take into account 
the impact it may have on the receiver.  That host can only service a 
limited number of connections.

Regards,
-sm 

Date: Sat, 17 Nov 2007 10:44:52 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John Leslie <[email-masked]>
CC: SM <[email-masked]>, [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John Leslie wrote:

>    I doubt there's any implementation of SPF that doesn't allow
> you to separate SPF processing from RFC2821MailFrom rewriting.
> If there is, _that_ is a bad idea. There are other things that
> can be broken by indiscriminate RFC2821MailFrom rewriting.

I morally, ethically, uncategorically hate, despise any protocol that 
tampers with mail, including and especially network control lines or 
headers.  It just invite even more tampering, kludges, complexity and 
potential foul ups.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sat, 17 Nov 2007 16:38:43 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Cc: John Leslie <[email-masked]>
Sender: [email-masked]

John Leslie writes:
>    It was my understanding that sieve is a "language for filtering
> e-mail messages at time of final delivery". If so, the URI fetching 
> and message regeneration will already have been done. So I don't see 
> a problem here.

Redirect causes a second final delivery to take place, and may change 
the envelope sender between the first and second deliveries.

Arnt

Date: Sat, 17 Nov 2007 10:32:40 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "David F. Skoll" <[email-masked]>
CC: [email-masked]
Subject: Re: Greylisting pitfalls (was Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation)
Sender: [email-masked]

David F. Skoll wrote:

> [Aside: I think "General Ledger" when I see GL... but anyway...]

Hi David,

If this was accounting forum, I would too. :-)

We already use BL as in RBL for Relay Black List, so why not GL for Grey 
List?  Maybe RGL is a better acroymn?

> Tempfailing RCPT commands will cause interoperability problems with certain
> broken SMTP implementations, specifically ones from Lotus.  Here's what
> happens.  S is the server doing greylisting and C is the broken Lotus client:
> 
> S: 220 foo.example.com ESMTP
> C: HELO busted.lotus.box
> S: 250 foo.example.com Hello busted.lotus.box
> C: MAIL FROM:<[email-masked]>
> S: 250 Go ahead
> C: RCPT TO:<[email-masked]>
> S: 451 Greylisting in effect, try again later
> C: RCPT TO:<[email-masked]>
> S: 451 Greylisting in effect, try again later
> C: DATA
> S: 503 Need valid recipient first
> C: QUIT
> 
> at which point busted.lotus.box generates a non-delivery notification.
> 
> Sad, isn't it?

Looks like it.  But this isn't a GL issue, right?

Is Lotus Notes not understanding 451? I mean, would it behave the same 
thing happen with 450?

Is Lotus Notes pipelining without permission?

Should a receiver who issues a temporary 45x at RCPT stick with the 45x 
at DATA to avoid issues with pipeling clients using the final code?   I 
guess not, if the reeciver did not advertise EHLO support for pipelining.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

From: [email-masked]
Cc: [email-masked], John Leslie <[email-masked]>, [email-masked]
Date: Sat, 17 Nov 2007 06:57:39 -0800 (PST)
Subject: Re: Fixing graylisting [was TBR]
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> John Leslie writes:
> >    It may be worth noting that sender address rewrite should not be
> >    done when forwarding a message via TBR.
> >
> >    (I can't offhand think of a reason you'd actually want to...)

> Sieve processors executing redirect statement are permitted to change
> the MAIL FROM, or leave it.

With one important exception: If the return address is empty you're not
supposed to fill it in. If you do you can end up creating a bounce loop.

I've been wondering if this restriction/concern needs to be mentioned in
2821bis.

> There are arguments in favour of both
> behaviours. If it's changed, it'll probably be to teh owner of the
> sieve script, or to <>.

Blanking the address is of course permissible, but as you say, there are 
arguments either way.

We now return you to the regularly scheduled greylisting/TBR debate...

				Ned

Date: Sat, 17 Nov 2007 09:57:00 -0500
From: John Leslie <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> wrote:
> John Leslie writes:
> 
>> It may be worth noting that sender address rewrite should not be 
>> done when forwarding a message via TBR.
>>
>> (I can't offhand think of a reason you'd actually want to...)
> 
> Sieve processors executing redirect statement are permitted to change 
> the MAIL FROM, or leave it. There are arguments in favour of both 
> behaviours. If it's changed, it'll probably be to the owner of the 
> sieve script, or to <>.

   It was my understanding that sieve is a "language for filtering
e-mail messages at time of final delivery". If so, the URI fetching
and message regeneration will already have been done. So I don't see
a problem here.

--
John Leslie <[email-masked]>

Date: Sat, 17 Nov 2007 09:51:35 -0500
From: John Leslie <[email-masked]>
To: SM <[email-masked]>
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

SM <[email-masked]> wrote:
> At 18:30 16-11-2007, John Leslie wrote:
> 
>> It may be worth noting that sender address rewrite should not
>> be done when forwarding a message via TBR.
> 
> That would rule out using SPF then.
> 
>> (I can't offhand think of a reason you'd actually want to...)
> 
> SPF recommends using SRS (Sender Rewriting Scheme) for forwarding.

   As you said, "recommends". (I don't see how that "rules out"
anything...

   Whether you apply SPF processing on an incoming email really
doesn't matter to TBR. The only case of interest is where you
receive a TBR message (which, of course, you can only do if your
MTA advertises TBR) _and_ pass it to the next hop via TBR (which
can only happen if that MTA advertises TBR). In that case,
rewriting the RFC2821MailFrom is a bad idea.

   I doubt there's any implementation of SPF that doesn't allow
you to separate SPF processing from RFC2821MailFrom rewriting.
If there is, _that_ is a bad idea. There are other things that
can be broken by indiscriminate RFC2821MailFrom rewriting.

   This issue emphasizes how complex things become if we apply
patches to try to make our pet ideas do things they were not
originally designed to do.

--
John Leslie <[email-masked]>

Date: Sat, 17 Nov 2007 09:46:26 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: SM <[email-masked]>, [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Sender: [email-masked]

Hi John,

Fair enough. I'm going to pencil in some work to get some feedback and 
tune the default frequency table and also add other factors as you and 
SM suggested.

But just note, I am not the only one with a 5 minute 2nd retry. I quick 
review showed ISP bellsouth.net servers had a 5 min retry.  AT&T took 
over BS  and it also uses 5 mins.  The largest telecomm company, and 
among the tops in inventing, understanding and optimizing data 
communications, packet switching, volume distribution ideas can't be 
that wrong? <g>

I can't say if they are using the same BS computers, but I see they are 
using different IP addresses. I'm going to take the time to get an 
average for 2nd retries from my logs for the past year, and also ask a 
few spread of customers to do the same.

--
HLS

John C Klensin wrote:
> 
> 
> 
> --On Friday, November 16, 2007 9:48 PM -0500 Hector Santos 
> <[email-masked]> wrote:
> 
>>
>> Hector Santos wrote:
>>>
>>> I probably should of use, 3 days since our original defaults
>>> (non-variable) was once per hour, 72 attempts or 3 days.  And
>>> if you  follow 2821 recommendation, it suggests 4-5 days.
>>> With 30 mins intervals  it yields an awful amount of 240-300
>>> retries.
>>
>> Sorry, bad math - 192-240 attempts!
> 
> Except that 2821 suggests rather rapid backoff ("...policy of two
> connection attempts in the first hour the message is in the queue, and 
> then backing off to one every two or three hours...").  One could stay 
> with 30 minutes if one wanted to, but the recommendation is more like
>    0, 30 minutes, 1 or 2 hours, 2 or 3 hours, 2 or 3 hours,...
> My personal experience is that
>    0, 30 minutes, 30 minutes, 1 hour, 2 hours, 2 hours, 4 hours, and 
> then continuing at four or even six hour intervals yields good results 
> and, at least in the absence of a deliberate delay policy on the server, 
> does not, on average, result in significantly slower delivery than a 
> "try every thirty minutes" plan.  That is especially true if the SMTP 
> sender queues are threaded s.t. as soon as a connection can be made to a 
> given SMTP receiver, all pending traffic for that receiver is moved to 
> the front of the queue and transmitted.   Of course, any sort of 
> per-message greylisting would foul that procedure up as well.
> 
>     john
> 
> 
> 
> 
> 
> 

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sat, 17 Nov 2007 09:37:27 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Greylisting pitfalls (was Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation)
Sender: [email-masked]

Hector Santos wrote:

> I have to see if our GL people are comfortable enough with GL to justify
> adding a RCPT TO rejection option.

[Aside: I think "General Ledger" when I see GL... but anyway...]

Tempfailing RCPT commands will cause interoperability problems with certain
broken SMTP implementations, specifically ones from Lotus.  Here's what
happens.  S is the server doing greylisting and C is the broken Lotus client:

S: 220 foo.example.com ESMTP
C: HELO busted.lotus.box
S: 250 foo.example.com Hello busted.lotus.box
C: MAIL FROM:<[email-masked]>
S: 250 Go ahead
C: RCPT TO:<[email-masked]>
S: 451 Greylisting in effect, try again later
C: RCPT TO:<[email-masked]>
S: 451 Greylisting in effect, try again later
C: DATA
S: 503 Need valid recipient first
C: QUIT

at which point busted.lotus.box generates a non-delivery notification.

Sad, isn't it?

Regards,

David.

Date: Sat, 17 Nov 2007 07:37:49 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Sender: [email-masked]

SM wrote:

> It's technically incorrect.  It's not an incompatibility problem.  A 
> sender should to be able to differentiate between a temporary rejection 
> due to an error and one due to the receiver's policy decision. 

Hi SM,

But who says the z=1 in 45z is "error in processing."?

There is nothing in the specs that officially states z=1 (3rd digit)
technically means is a error in processing.  If we do that, then it has
to equally apply to all the other response codes and their various ways
they are uses.  For example:

       551 User not local; please try <forward-path>
       421 <domain> Service not available, closing transmission channel
       251 User not local; will forward to <forward-path>
       551 User not local; please try <forward-path>

All these commonly used z=1 responses are not use for "error in
processing" ideas.

The only thing that is important are the first two digits.

> If you 
> are doing log analysis, you can categorize the different conditions by 
> reply code and take whatever action may be necessary.
> 
> I once came across a site in the Far East that was temporary rejecting a 
> message.  As I didn't understand the text (language), I had to rely on 
> the reply code to deduce whether they were explicitly rejecting messages 
> from the sending host.

Sm, I can honestly say this is little way over my head. :-)

To me, a sender, which I presume to mean a "SMTP client software",
doesn't read Japanese, Chinese, Spanish, Spanglish, and even English.
It reads SMTP and in terms of the state machine, the first two digit is 
what matters.

So I am having a difficult time seeing how it can be "technically 
incorrect" if the design requirement for correct state machine 
automation is the 45x reading.  Nothing more, no extended codes, no 
literals.  That extra stuff is for humans. No?

IMO, log analyzers can not 100% depend on any 3rd digit to mean anything 
specific.

Also, to me, if we wanted to take it literally, error in processing 
should be a permanent rejection idea.  Why?  We had this discussion in 
length in regards when 55x or 45x is appropriate and I think we 
establish if the data is not going to change, providing the same input 
on new tries is not going change the state. So one can argue that the 
literal example for 451 is erroneous.  It should be for some realistic 
"temporary reason" other then error in processing. Error in processing 
what? Your data?  If so, it should not ask again to retry.

If we speaking in term of "Sender" the human, I guess I can could see 
how one might say it is functionally incorrect based on the literal 
interpretation, but not technical.

If we did take it literally, consider we would have only 9 possible 
unique temporary events for it. Are they only 9 functional reasons  for 
a 45z response?

If we follow the literal, we can't use 450 because the literal is not
true in the same vain the 451 error in processing is not true.  Neither 
is the literal for 452, insufficient system storage, although one could 
reason that "we don't have storage for spammers at this moment, so try 
again while we clear some space for you." for the 452 code. But that 
isn't true either. <g>

Anyway, to me what is more important is SMTP reliability and 45x is all
that is important.  In my opinion, the author of GL, really had no
choice but to use 451, probably because it is least use one.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sat, 17 Nov 2007 07:13:02 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy    recommendation
Sender: [email-masked]

SM wrote:

> A sender not familiar with Greylisting may find the rejects strange at 
> first and might blame the software.

We get blame for everything. <g>

> I use Greylisting in the DATA phase.  Some people prefer it to be done 
> at RCPT TO to reduce processing overhead.

I have to see if our GL people are comfortable enough with GL to justify 
adding a RCPT TO rejection option.

> Capturing content is not workable for us due to the additional disk 
> space required.  

Thats probably the only common comment I see now. Deleting this stuff 
more frequently. My answer?  "erase wcgl\msg*.dat" :-)

> I have encountered problems with Greylisting.  I came across odd cases. 

Care to share a few?

> Note that the following is merely a comment.  You are better placed than 
> me to decide what's best for your environment.

Well, its the default and it was pretty much a SWAG <g>  I'm sure 
operators have fine tuned them, and other SMTP related options, if they 
had to, but we haven't heard of any issues or concerns with mail 
delivery so thats most important. And if there were an issue, it would 
addressed pronto.

We do have a very aggressive system, and thats what they want. Wouldn't 
have any other way.  We all don't have to be the same, right?

> You are doing more than three retries over an hour period.  That's quite 
> high in my opinion.  

Its only high if its problem across the board. I'm not going to worry 
about the few (Pareto principle).  Also, 4 vs 2 is not much a 
difference, but what does high translate too? Overhead? For whom? Are we 
dealing with a down system? a session level 45x reject?

I'm of the opinion, that if it reaches the session, the odds are very 
hight it will succeed in 1 try or 2 tries for most GL systems, maybe the 
3rd try with you. :-)  Those with higher block times, well, are just 
playing games with spammers and could less about business customers.

Also, from what I have seen, top ISPs are also well below 30 mins 2nd 
retry. In fact for Bellsouth.net, it has a 5 minute 2nd retry. But AT&T 
took them over, so lets see what  att.net servers are doing. ..... Yup, 
exactly 5 mins 2nd retry which is what I saw (<30 mins) by the majority 
  when I was looking at the time.

IMO, 30 minutes is really too high and IMO, it corresponds to the good 
old days of less reliable machines, more down systems, less reliable 
connections, slower bandwidths, less scaled systems, etc, etc. But not 
today. With high performance machines, multi-threading software, Virtual 
Machines, higher bandwidths, better scaling, more challenge/response 
systems, etc, may suggest the x821 30 minute retry "guideline" is 
outdated.  :-)

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Sat, 17 Nov 2007 00:20:10 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 17:01 16-11-2007, Hector Santos wrote:
>Please note that I am not disagreeing with your points. With 
>skepticism, upon customer "wish list" request and the tracking of 
>how that request did not die, I explored GL and found it to be 
>"do-able." With the fine tuning to minimize impact, it can work 
>without disrupting operations.

We sometimes have to agree to disagree. :-)

>Once GL was part of the picture, it was fairly obvious now why 
>operators were previously reporting these strange rejects with no 
>explanation and confused observation of the eventual 
>delivery.  Hence the variable table was added, not just for the GL 
>operators but for those who were hitting GL systems.

A sender not familiar with Greylisting may find the rejects strange 
at first and might blame the software.

>As for turning on the GL feature for our own support system, that 
>was a even tougher decision since as a small company, we can't 
>afford any missed sales or customer support emails.  But it was 
>turned on and carefully followed. In fact, since there was 
>unsureness of false positives, rather than reject at RCPT TO as I 
>believe many do, we implemented it as part of the DATA filter hook 
>system with a dynamic response at that point.

I use Greylisting in the DATA phase.  Some people prefer it to be 
done at RCPT TO to reduce processing overhead.

>Anyway, with a web-base GL tool, this gave the operator an easy way 
>to view stats and check all current GL 1st rejects message content 
>to help give them  (and us) confidence

Capturing content is not workable for us due to the additional disk 
space required.  It can be turned on for debugging.  I have 
encountered problems with Greylisting.  I came across odd cases. I 
adapted to them as telling the receiver that it's the sender's fault 
won't solve the problem.

>The 5 mins was carefully decided upon, mainly because I don't 
>particular like the idea of going against 2821 recommendations.  But 
>the market overrule that issue.  In the end, our default variable table is:
>
>[Attempts]
>Default=60
>Attempt1=5
>Attempt2=5
>Attempt3=15
>Attempt5=30
>Attempt10=120
>Attempt21=5
>Attempt22=5
>Attempt23=15
>Attempt25=30
>Attempt30=120
>Attempt40=60
>Attempt72=60
>
>Note: attempt1 is really the 2nd attempt, since the rescheduling 
>code is  based off the current count, "msgQueue->nTotalAttempts"

Note that the following is merely a comment.  You are better placed 
than me to decide what's best for your environment.

You are doing more than three retries over an hour period.  That's 
quite high in my opinion.  Your backoff period could be 
incremental.  I see that you are using five minutes from attempt 
22.  Once you go over the first two attempts, you can move from 30 
minutes, one hour, then two hours etc.  You should take into account 
the impact frequent reruns can have if the queue gets quite 
large.  Also, consider the receiving hosts.  If the SMTP connection 
to that host failed, I would not retry the other messages addressed 
to that host within the same time frame.

>Finally, on the GL receiver side, our default is a 55 second block 
>and a  2 day grace period to send the retry.

I have a default block of over five minutes which means that your 
retry won't get through on the second attempt.  But then, not all 
connections are greylisted.

>I probably should of use, 3 days since our original defaults 
>(non-variable) was once per hour, 72 attempts or 3 days.  And if you 
>follow 2821 recommendation, it suggests 4-5 days. With 30 mins 
>intervals it yields an awful amount of 240-300 retries.

It's only a lot of retries over four or five days if you retry too 
frequently.  Some view the period as too long.  A two-day give-up 
period, for example, may result in delivery issues being missed if it 
falls over a weekend.

>Finally, for the 451 code itself, yeah, I didn't think it was ideal, 
>but I do think that given all our choices, the GL author made the 
>right decision.  Assuming the author is an operator mostly, reading 
>RFC 2821, he sees three examples of 45x with literals:

I commented on that in my previous email.

>But it should not matter from an SMTP technical standpoint because 
>the SMTP sender must use 45z for its retry considerations, 
>regardless of what z is.

Yes.

>I will say, that I did consider using 451 as a trigger for the 
>altered shorter 2nd attempt interval.  But our outbound mail code a 
>45x response  and I didn't want to change for reasons that it might 
>not be 451 but 450, 452 or some other 45x value.

The 451 reply code is used for other conditions as well.

Regards,
-sm 

Date: Sat, 17 Nov 2007 08:52:58 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Cc: John Leslie <[email-masked]>, [email-masked]
Sender: [email-masked]

John Leslie writes:
>    It may be worth noting that sender address rewrite should not be 
>    done when forwarding a message via TBR.
>
>    (I can't offhand think of a reason you'd actually want to...)

Sieve processors executing redirect statement are permitted to change 
the MAIL FROM, or leave it. There are arguments in favour of both 
behaviours. If it's changed, it'll probably be to teh owner of the 
sieve script, or to <>.

Arnt

Date: Fri, 16 Nov 2007 22:46:20 -0800
To: John Leslie <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Cc: [email-masked]
Sender: [email-masked]

Hi John,
At 18:30 16-11-2007, John Leslie wrote:
>    It may be worth noting that sender address rewrite should not
>be done when forwarding a message via TBR.

That would rule out using SPF then.

>    (I can't offhand think of a reason you'd actually want to...)

SPF recommends using SRS (Sender Rewriting Scheme) for forwarding.

Regards,
-sm 

Date: Fri, 16 Nov 2007 22:36:17 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 13:23 16-11-2007, Hector Santos wrote:
>Incorrect in what way? Inappropriate perhaps from a 
>"operator/policy" statement?  Functionally or Technically? 
>Compatibility?  If its a compatibility problem, then it needs to be 
>reconsidered.

It's technically incorrect.  It's not an incompatibility problem.  A 
sender should to be able to differentiate between a temporary 
rejection due to an error and one due to the receiver's policy 
decision.  If you are doing log analysis, you can categorize the 
different conditions by reply code and take whatever action may be necessary.

I once came across a site in the Far East that was temporary 
rejecting a message.  As I didn't understand the text (language), I 
had to rely on the reply code to deduce whether they were explicitly 
rejecting messages from the sending host.

Regards,
-sm 

Date: Sat, 17 Nov 2007 00:16:01 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, SM <[email-masked]>
cc: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Sender: [email-masked]

--On Friday, November 16, 2007 9:48 PM -0500 Hector Santos 
<[email-masked]> wrote:

>
> Hector Santos wrote:
>>
>> I probably should of use, 3 days since our original defaults
>> (non-variable) was once per hour, 72 attempts or 3 days.  And
>> if you  follow 2821 recommendation, it suggests 4-5 days.
>> With 30 mins intervals  it yields an awful amount of 240-300
>> retries.
>
> Sorry, bad math - 192-240 attempts!

Except that 2821 suggests rather rapid backoff ("...policy of two
connection attempts in the first hour the message is in the 
queue, and then backing off to one every two or three 
hours...").  One could stay with 30 minutes if one wanted to, 
but the recommendation is more like
    0, 30 minutes, 1 or 2 hours, 2 or 3 hours, 2 or 3 hours,...
My personal experience is that
    0, 30 minutes, 30 minutes, 1 hour, 2 hours, 2 hours, 4 
hours, and then continuing at four or even six hour intervals 
yields good results and, at least in the absence of a deliberate 
delay policy on the server, does not, on average, result in 
significantly slower delivery than a "try every thirty minutes" 
plan.  That is especially true if the SMTP sender queues are 
threaded s.t. as soon as a connection can be made to a given 
SMTP receiver, all pending traffic for that receiver is moved to 
the front of the queue and transmitted.   Of course, any sort of 
per-message greylisting would foul that procedure up as well.

     john

Date: Fri, 16 Nov 2007 21:48:50 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Sender: [email-masked]

Hector Santos wrote:
> 
> I probably should of use, 3 days since our original defaults 
> (non-variable) was once per hour, 72 attempts or 3 days.  And if you 
> follow 2821 recommendation, it suggests 4-5 days. With 30 mins intervals 
> it yields an awful amount of 240-300 retries.

Sorry, bad math - 192-240 attempts!

-- 
HLS

Date: Fri, 16 Nov 2007 21:30:12 -0500
From: John Leslie <[email-masked]>
To: SM <[email-masked]>
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

SM <[email-masked]> wrote:
> 
> A side-effect of using SPF for the first part above is that sender 
> address rewrite generally suggested as a solution for forwarding 
> doesn't fit into TBR.

   It may be worth noting that sender address rewrite should not
be done when forwarding a message via TBR.

   (I can't offhand think of a reason you'd actually want to...)

--
John Leslie <[email-masked]>

Date: Fri, 16 Nov 2007 21:12:56 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Nature of spammers (was Re: DoS attacks (was Re: SMTP Transferred-By-Reference))
Sender: [email-masked]

John C Klensin wrote:

> (off list)

Actually, it was on-list.

> Hector, based on a semi-private FBI briefing two weeks ago, the vast
> majority of spammers in the phishing, stock fraud, and related
> businesses are not amateurs or snake-oil sales people, but very
> sophisticated and organized criminals.

I agree.  There are still plenty of amateur spammers, but the
professional ones are the most dangerous.  I recently saw a phishing
scam that asked you to call a phone number rather than click a link.
When you called the number, you got a reasonably-convincing IVR menu
that prompted you through entering your bank card number, expiry date
and PIN.

That level of sophistication (getting a VoIP North American number;
setting up the IVR system to collect data) doesn't indicate stupidity
or amateurishness to me.

Regards,

David.

Date: Fri, 16 Nov 2007 20:19:09 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, Glenn Anderson <[email-masked]>
cc: [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

(off list)

--On Friday, November 16, 2007 6:16 PM -0500 Hector Santos 
<[email-masked]> wrote:

> Spammers get more credit than they deserve.
>
> The majority are small organization snake-oil sales people
> dealing in a very small margin/high volume business model.
> They don't wish to waste expense on any R&D SMTP cracking
> team.  They use free ware, brain-dead bulk mailers, scripts
> engines, some of which are ironically written by career IETF
> participants.   The better ones might get specialized bulk
> mailer/statistic software, but overall, their best offense is
> no offense - do nothing - work in legacy mode and/or compliant
> with the expectations.

Hector, based on a semi-private FBI briefing two weeks ago, the 
vast majority of spammers in the phishing, stock fraud, and 
related businesses are not amateurs or snake-oil sales people, 
but very sophisticated and organized criminals.  One can, of 
course, debate the fraction of total spam the phishers and stock 
fraud perpetrators represent (it is a large majority of what I'm 
seeing these days, but I may not be normal), but the days of 
spamming as a completely amateur activity seem to be waning 
quickly.

    john

Date: Fri, 16 Nov 2007 20:01:53 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Sender: [email-masked]

Hi SM,

Please note that I am not disagreeing with your points. With skepticism, 
upon customer "wish list" request and the tracking of how that request 
did not die, I explored GL and found it to be "do-able." With the fine 
tuning to minimize impact, it can work without disrupting operations.

Once GL was part of the picture, it was fairly obvious now why operators 
were previously reporting these strange rejects with no explanation and 
confused observation of the eventual delivery.  Hence the variable table 
was added, not just for the GL operators but for those who were hitting 
GL systems.

As for turning on the GL feature for our own support system, that was a 
even tougher decision since as a small company, we can't afford any 
missed sales or customer support emails.  But it was turned on and 
carefully followed. In fact, since there was unsureness of false 
positives, rather than reject at RCPT TO as I believe many do, we 
implemented it as part of the DATA filter hook system with a dynamic 
response at that point.

That allowed us to store a copy of the message for review to see how 
effective it was and/or more importantly, to see if "good messages" were 
lost due to the "good sender" not retrying again.

I can tell ya that the latter was a non-issue and that was sold me on 
this GL concept.  If there was even a small percentage showing that 
"good intention" systems had broken SMTP retry logic, odds are very 
high, I would have nixed this project and explained to our customers 
why.  This is not to say there were individual incidences where a "good 
intention" message system did not retry.  But that soon became a funny 
moral reason for supporters to yell at those: "FIX YOUR SMTP SOFTWARE - 
YOU ARE ACTING LIKE A SPAMMER."  If I recall, this was mostly an issue 
with systems old PHP scripts with one shot mail send or notification 
logic, but were failing not a GL, but with not properly handling 
multiple response lines. So in most cases, it wasn't GL itself, but some 
other reason, but they looked at GL as the reason.

Anyway, with a web-base GL tool, this gave the operator an easy way to 
view stats and check all current GL 1st rejects message content to help 
give them  (and us) confidence of this obscure idea working or not. 
This helped sell it.  Its funny I should note, remember, these are 
operators, early on some suggested that we add a click button to move 
the current message into the accepted mail inbound quue for import. 
But I explain, thats would be a good idea if we saw good systems not 
retrying.  I think today, they are convince of that. Just let it run and 
forget about it. Don't see there looking at the web GL stats and rejects 
table listings and begin to doubt if a partiticulr new mail that looks 
good would eventually come in again and get delivery.  Guaranteed! It 
will drive you nuts. :)

The 5 mins was carefully decided upon, mainly because I don't particular 
like the idea of going against 2821 recommendations.  But the market 
overrule that issue.  In the end, our default variable table is:

[Attempts]
Default=60
Attempt1=5
Attempt2=5
Attempt3=15
Attempt5=30
Attempt10=120
Attempt21=5
Attempt22=5
Attempt23=15
Attempt25=30
Attempt30=120
Attempt40=60
Attempt72=60

Note: attempt1 is really the 2nd attempt, since the rescheduling code is 
  based off the current count, "msgQueue->nTotalAttempts"

Finally, on the GL receiver side, our default is a 55 second block and a 
  2 day grace period to send the retry.

I probably should of use, 3 days since our original defaults 
(non-variable) was once per hour, 72 attempts or 3 days.  And if you 
follow 2821 recommendation, it suggests 4-5 days. With 30 mins intervals 
it yields an awful amount of 240-300 retries.

But I can't recall off hand the reason two days was selected for the 
default GL grace period.  Maybe I was thinking that if spammers were 
using the RFC as a guideline or the GL specs of 4 days, then all they 
had to do is wait 3 days to retry.

Finally, for the 451 code itself, yeah, I didn't think it was ideal, but 
I do think that given all our choices, the GL author made the right 
decision.  Assuming the author is an operator mostly, reading RFC 2821, 
he sees three examples of 45x with literals:

       450 Requested mail action not taken: mailbox unavailable
          (e.g., mailbox busy)
       451 Requested action aborted: local error in processing
       452 Requested action not taken: insufficient system storage

With the possible erroneous presumption the literals are set in stone 
for the reject reason, then among the three, 451 is arguably preferred 
over 450 and 452.

But it should not matter from an SMTP technical standpoint because the 
SMTP sender must use 45z for its retry considerations, regardless of 
what z is.

I will say, that I did consider using 451 as a trigger for the altered 
shorter 2nd attempt interval.  But our outbound mail code a 45x response 
  and I didn't want to change for reasons that it might not be 451 but 
450, 452 or some other 45x value.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Hector Santos wrote:
> 
> SM wrote:
>>
>>> the Greylist specs shows  a 4.7.1 extended code:
>>>
>>>       451 4.7.1 Please try again later
>>>       http://projects.puremagic.com/greylisting/whitepaper.html
>>
>> I believe that the reply code mentioned in that whitepaper is 
>> incorrect.  The extended code is correct.  I recommend using "450 
>> 4.7.1 Text" when the temporary failure is due to a policy decision.
> 
> Incorrect in what way? Inappropriate perhaps from a "operator/policy" 
> statement?  Functionally or Technically? Compatibility?  If its a 
> compatibility problem, then it needs to be reconsidered.
> 
>> As a general rule, I would use 30 minutes as receivers reading RFC 
>> 2821 will expect that.
> 
> Sure, but all receivers need to be ready for anything, including the 
> possibility of "more sophisticated and variable strategies" as it was 
> insightfully stated in 2821. :-)
> 
> So I don't think its would be a technical problem.
> 

Date: Fri, 16 Nov 2007 18:16:52 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Glenn Anderson <[email-masked]>
CC: [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

My opinion:

Spammers get more credit than they deserve.

The majority are small organization snake-oil sales people dealing in a 
very small margin/high volume business model.  They don't wish to waste 
expense on any R&D SMTP cracking team.  They use free ware, brain-dead 
bulk mailers, scripts engines, some of which are ironically written by 
career IETF participants.   The better ones might get specialized bulk 
mailer/statistic software, but overall, their best offense is no offense 
- do nothing - work in legacy mode and/or compliant with the expectations.

Also, in lieu of bugs, everyone with serious server systems already has 
a DoS, smtp related or otherwise, loading limit and/or self-regulating 
controls in their software.

I see these loading-based DoS points as a non-issue.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Glenn Anderson wrote:
> 
> At 12:01 am -0500 16/11/2007, [email-masked] wrote:
>> On Fri, 16 Nov 2007 14:46:29 +1300, Glenn Anderson said:
>>
>>>  I don't believe it to be a simple problem for the spammers and bot
>>>  architects to solve. Not only do they need to start keeping track of
>>>  state, they need to keep track of a lot of state (the volume of
>>>  messages they are trying to send is huge), which requires a lot more
>>>  resources, and they need to manage CPU and network resources such
>>>  that they are available for them to retry later.
>>
>> Oh, hogwash.
>>
>> Years ago (1995 timeframe), I was able to do a million RCPT TO's/day 
>> from a
>> listserv machine, using Sendmail on an IBM RS6000-250.  Little pizza 
>> box, 256M
>> of memory, a whole whopping 66mz 601E processor.
> 
> I don't follow. Your saying that because you could send a million RCPT 
> TOs on a 66MHz 601 back in 1995, the extra overhead of having to track 
> state information won't negatively impact the rate at which bots can 
> attempt to send spam?
> 
>> If spammers can't write code
>> tight enough to do so un-noticed on today's consumer boxes, there's 
>> something
>> *seriously* wrong.
> 
> I don't see how that makes any difference, unless you have some reason 
> to believe that current bots that don't track state are very 
> inefficient, or aren't utilizing as much CPU and network bandwidth as 
> they could be.
> 
> Glenn.
> 
> 
> 

Date: Sat, 17 Nov 2007 11:31:51 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

At 12:01 am -0500 16/11/2007, [email-masked] wrote:
>On Fri, 16 Nov 2007 14:46:29 +1300, Glenn Anderson said:
>
>>  I don't believe it to be a simple problem for the spammers and bot
>>  architects to solve. Not only do they need to start keeping track of
>>  state, they need to keep track of a lot of state (the volume of
>>  messages they are trying to send is huge), which requires a lot more
>>  resources, and they need to manage CPU and network resources such
>>  that they are available for them to retry later.
>
>Oh, hogwash.
>
>Years ago (1995 timeframe), I was able to do a million RCPT TO's/day from a
>listserv machine, using Sendmail on an IBM RS6000-250.  Little pizza box, 256M
>of memory, a whole whopping 66mz 601E processor.

I don't follow. Your saying that because you could send a million 
RCPT TOs on a 66MHz 601 back in 1995, the extra overhead of having to 
track state information won't negatively impact the rate at which 
bots can attempt to send spam?

>If spammers can't write code
>tight enough to do so un-noticed on today's consumer boxes, there's something
>*seriously* wrong.

I don't see how that makes any difference, unless you have some 
reason to believe that current bots that don't track state are very 
inefficient, or aren't utilizing as much CPU and network bandwidth as 
they could be.

Glenn.

Date: Fri, 16 Nov 2007 16:23:19 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy   recommendation
Sender: [email-masked]

SM wrote:
> 
>> the Greylist specs shows  a 4.7.1 extended code:
>>
>>       451 4.7.1 Please try again later
>>       http://projects.puremagic.com/greylisting/whitepaper.html
> 
> I believe that the reply code mentioned in that whitepaper is 
> incorrect.  The extended code is correct.  I recommend using "450 4.7.1 
> Text" when the temporary failure is due to a policy decision.

Incorrect in what way? Inappropriate perhaps from a "operator/policy" 
statement?  Functionally or Technically? Compatibility?  If its a 
compatibility problem, then it needs to be reconsidered.

> As a general rule, I would use 30 minutes as receivers reading RFC 2821 
> will expect that.

Sure, but all receivers need to be ready for anything, including the 
possibility of "more sophisticated and variable strategies" as it was 
insightfully stated in 2821. :-)

So I don't think its would be a technical problem.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Fri, 16 Nov 2007 08:49:14 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy  recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 06:53 16-11-2007, Hector Santos wrote:
>I believe the z digit in x5z is undefined:
>
>The 1st paragraph, last sentence in 4.2.1, says:
>
>    "The third digit and any supplemental information that may be
>    present is reserved for the finest gradation of information."
>
>I see the example literal text for 451 says "processing error", but 
>the literals are meaningless to SMTP.  Offer no flow control logic. Right?

The literals in there serve as guidance to when we would use such a 
reply code.  There's no different flow control logic on the 
literals.  From an operator's point of view the reply code is helpful 
to troubleshoot a problem.  I believe that we should differentiate 
between policy and processing errors.

>Yes, I agree, it would ideal if a special code can be detected.  Hmmm,
>the Greylist specs shows  a 4.7.1 extended code:
>
>       451 4.7.1 Please try again later
>
>
>       http://projects.puremagic.com/greylisting/whitepaper.html

I believe that the reply code mentioned in that whitepaper is 
incorrect.  The extended code is correct.  I recommend using "450 
4.7.1 Text" when the temporary failure is due to a policy decision.

>We use the 5 minute 2nd retry interval because anything else is 
>deemed too long for these deliberate 1st time only rejects - an 
>action that should not occur under normal circumstances.   I have 
>not seen or heard of any negative impact due to a shorter 2nd retry 
>interval. But there were an awful amount of reports from confused 
>operators before we added the variable frequency table to address 
>the increasing hits on remote GL systems.

The decision on what interval to choose for the second retry is 
guided by technical specifications but one also has to take the 
market into account.  It's a "SHOULD" which I read as unless you have 
a good reason not to follow it.  If you have not seen or heard of any 
negative impact due to the shorter second retry, then it's not a 
problem.  That setting should be customizable though.

As a general rule, I would use 30 minutes as receivers reading RFC 
2821 will expect that.    This question came up on the mailing list 
some time back.

Regards,
-sm  

Cc: John Leslie <[email-masked]>, [email-masked]
From: Douglas Otis <[email-masked]>
To: John C Klensin <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference // Alternative to Temp Error resource protection
Date: Fri, 16 Nov 2007 08:25:30 -0800
Sender: [email-masked]

On Nov 16, 2007, at 7:04 AM, John C Klensin wrote:

> The user of a commercial email service that provides free mailboxes,  
> or the user of an ISP's mail system, has _no_ practical control over  
> the behavior of the relevant servers, despite sharing an ADMD.    
> Yes, I control my mail servers and someone receiving mail through  
> the servers of an enterprise for which she works could be reasonably  
> though of as being within the same narrowly-defined ADMD, but, in  
> the general case, no.
>
> And, for that general case, one either
>
> 	* needs to retrieve or fetch the TBR message onto user
> 	equipment and then do the retrieval -- a process that
> 	cannot guarantee that the relevant machines are
> 	well-connected

Any transmitting MTA that is not well connected should not use the TBR  
extension.  There are other situations where the TBR extension would  
not be used, such as for MUA message submission to an MSA.  This  
extension provides an efficient alternate method to complete message  
transmissions that might otherwise be momentarily blocked by a  
receiver's Temp error.   Only the MTA transmitter would need to offer  
a well-connected service when the benefit of the TBR extension is  
desired.

The TBR extension does not involve any end user interaction.  An email  
user may benefit by having their email transmitted more expediently  
and with greater integrity when their provider supports the TBR  
extension, but there is no necessity that this extension be supported  
thoughout.  Good things still come to those that wait. : )

-Doug

  

Date: Fri, 16 Nov 2007 10:04:59 -0500
From: John C Klensin <[email-masked]>
To: John Leslie <[email-masked]>
cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

--On Friday, 16 November, 2007 09:33 -0500 John Leslie
<[email-masked]> wrote:

> John C Klensin <[email-masked]> wrote:
>> 
>> ... including my usual problem of not liking to have to be
>> online to read and reply to messages
> 
>    Evidently there is something in the draft which led John K
> to believe this problem would arise. That was not our
> intention, and it would help to point out the text in question
> so we can fix it.
> 
>    In the normal case, the TBR URI would be retrieved by a
> well- connected MTA on the receiver's Administrative
> Management Domain, and the (reconstructed) email would be
> forwarded as a non-TBR message to the recipient's message
> store.

Sorry, John.

I believe that delegating that sort of authority to the
"well-connected MTA" in the general case constitutes a far worse
operational or security compromise  than anything the TBR
procedure  could help with.  An ADMD is a useful convenience for
modeling but, for the Internet, has little actual meaning in the
general case.  The user of a commercial email service that
provides free mailboxes, or the user of an ISP's mail system,
has _no_ practical control over the behavior of the relevant
servers, despite sharing an ADMD.   Yes, I control my mail
servers and someone receiving mail through the servers of an
enterprise for which she works could be reasonably though of as
being within the same narrowly-defined ADMD, but, in the general
case, no.  

And, for that general case, one either 

	* needs to retrieve or fetch the TBR message onto user
	equipment and then do the retrieval -- a process that
	cannot guarantee that the relevant machines are
	well-connected
	
	* or needs to engage in filtering in the middle of the
	network, outside the user's practical (as distinct from
	theoretical) administrative control.

the second is a show-stopper for me, as well as a security
problem, etc., etc.  And the first raises the problems with poor
connections and fast-synch requirements.

    john

Date: Fri, 16 Nov 2007 09:56:11 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy  recommendation
Sender: [email-masked]

Hi SM,

Isn't the z digit in x5z is undefined?  The 1st paragraph, last sentence 
in 4.2.1, says:

    "The third digit and any supplemental information that may be
    present is reserved for the finest gradation of information."

I see the example literal text for 451 says "processing error", but the 
literals are meaningless to SMTP.  Offer no flow control logic. Right?

Yes, I agree, it would ideal if a special code can be detected.  Hmmm,
the Greylist specs shows  a 4.7.1 extended code:

       451 4.7.1 Please try again later

       http://projects.puremagic.com/greylisting/whitepaper.html

We use the 5 minute 2nd retry interval because anything else is deemed 
too long for these deliberate 1st time only rejects - an action that 
should not occur under normal circumstances.   I have not seen or heard 
of any negative impact due to a shorter 2nd retry interval. But there 
were an awful amount of reports from confused operators before we added 
the variable frequency table to address the increasing hits on remote GL 
systems.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

SM wrote:
 >
 > Hi Hector,
 > At 15:59 15-11-2007, Hector Santos wrote:
 >> But I want to make sure I wasn't mis-read.  To be clear, the 
suggestion really has nothing to do with anti-spam or even mentioning 
it.  We already have it saving:
 >>
 >> 4.5.4.1 Sending Strategy
 >>
 >>    ...
 >>
 >>    The sender MUST delay retrying a particular destination after one
 >>    attempt has failed.  In general, the retry interval SHOULD be at
 >>    least 30 minutes; however, more sophisticated and variable strategies
 >>    will be beneficial when the SMTP client can determine the reason for
 >>    non-delivery.
 >>
 >> This was excellent insight done at the time. Maybe that is enough, 
but I do believe a word smith can finesse a new clause or sentence 
indicating why a short 2nd attempt may be applicable and highly 
desirable for improved outbound mail operations.  Maybe something like:
 >>
 >>    Due to the increase of new SMTP challenge/response receiver
 >>    systems, if the SMTP client sees 451 temporary response code
 >>    for the RCPT or DATA state, it MAY consider using a shorter
 >>    2nd attempt interval, recommended 5-10 minutes.  This SHOULD
 >>    be for the 2nd attempt only.
 >
 > The "451" reply code means error in processing.  As it's a policy 
decision, a "450" reply code is more appropriate.
 >
 > With the advent of greylisting, a shorter second attempt may seem 
desirable to ensure timely delivery of message.  However, we should take 
into account the impact it will have on the receiver.  Such a short 
interval conflates the problem for the receiving system if it's under 
heavy load.  We are also assuming that the greylisting period will be 
under five or ten minutes.  That may not be the case.  For the proposed 
strategy to be effective, the sender would have to determine whether the 
temporary error was due to greylisting (status code instead of reply 
code) and the greylisting delay.
 >
 > Regards,
 > -sm
 >
 >

Date: Fri, 16 Nov 2007 09:33:36 -0500
From: John Leslie <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John C Klensin <[email-masked]> wrote:
> 
> ... including my usual problem of not liking to have to be online
> to read and reply to messages

   Evidently there is something in the draft which led John K to
believe this problem would arise. That was not our intention, and
it would help to point out the text in question so we can fix it.

   In the normal case, the TBR URI would be retrieved by a well-
connected MTA on the receiver's Administrative Management Domain,
and the (reconstructed) email would be forwarded as a non-TBR
message to the recipient's message store.

   There is text, about an error condition where a TBR MTA needs
to pass a TBR mesage to a non-TBR MTA, which allowed for an error
notification to go forward: this text was not well-thought-out
and needs to be fixed -- but this assuredly is not a normal case.

> But, in fairness to the proposal, the general idea has one
> advantage.  If one is concerned about source / originator
> identity and authentication, having to make a real-time direct
> connection back to the sender's repository permits thinking
> about much stronger methods than, e.g., header signatures.

   Kind words, John. Thank you.

--
John Leslie <[email-masked]>

Date: Fri, 16 Nov 2007 09:17:22 -0500
From: John Leslie <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> wrote:
> [email-masked] writes:
> 
>> One of the unintended advantages of TBR over greylisting is that it 
>> can be used for a message recall feature. :-)
> 
> So can greylisting if you can reach into the queue. Which noone can 
> today, but if you're going to implement something new you might as well 
> implement a reach-into-the-queue feature as TBR.

   Actually, reach-into-the-queue isn't a dependable recall. The
message my be queued becase of a 4xx error _after_ DATA while waiting
for content-analysis. To _deliver_ the mesage before the SMTP retry
would be a violation of protocol, but the message is clearly out there
and could be retrieved.

   (We could start a thread on message recall if folks think this is
worth further discussion...)

--
John Leslie <[email-masked]>

Date: Fri, 16 Nov 2007 09:11:02 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

[email-masked] <[email-masked]> wrote:
> On Tue, 13 Nov 2007 13:09:34 PST, SM said:
> 
>> One of the unintended advantages of TBR over greylisting is that it 
>> can be used for a message recall feature. :-)
> 
> Not really, unless you *mandate* that a "cooling-off" period of (say)
> 5 minutes be enforced before anybody can fetch the message. Otherwise
> you encounter a race condition between you finding 'recall message'
> in the menu, and another site fetching the message.

   I don't see that "mandating" a "cooling-off" period would actually
help. Do note, however, that information is available to the sender
whether the message was fetched before the recall.

> (Yes, I see the smiley - I'm just filling in the blanks for less-clued
> people who may someday be reading the list archives....)

   Fair enough...

--
John Leslie <[email-masked]>

Cc: [email-masked], [email-masked]
From: Douglas Otis <[email-masked]>
To: John C Klensin <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Fri, 16 Nov 2007 05:35:22 -0800
Sender: [email-masked]

On Nov 16, 2007, at 2:57 AM, John C Klensin wrote:

> I expected to dislike it because I consider "synchronize and run for  
> the airplane" and "synchronize before the stinking wireless link  
> goes down again and then actually read the mail offline" to be  
> _very_ important mail-use modes.  I am also, to put it mildly  
> unwilling to lose multihop relaying to reach locations with less- 
> than-perfect or some-times-of-day-only connectivity.   Consequently,  
> I see it as costly and burdensome on the receivers (and other "good  
> guys") while providing the bad guys only a temporary impediment.

John,

Sorry for the confusion in how the TBR Extension is intended to  
operate.  The TBR extension is to be used between well connected MTAs  
and not at the MUAs.  The end-user will not receive their email any  
differently.  You can still read your email at 30k feet.  End-User  
firewalls do not change.  DDoS concerns related to the reach-back  
mechanism would be at lower gains and easier to mitigate than would  
direct threats from bots.

A Temp error is the _only_ means available to prioritize resources  
when MTAs are overrun.  The TBR extension provides an alternative for  
completing an SMTP transaction, rather than the increasingly used "go  
away for now" mode of operation.  Using the TBR extension should allow  
email to arrive with _less_ delay, and not more.  This should involve  
_fewer_ transactions and not more.  Even if all connections used SSL  
certificates, the need to protect resources would not change.   
Receivers can still be inundated.  Certificates can be purchased with  
fraudulent credentials.  Resources must still be protected where "go  
away for now" makes the situation worse.

Message delivery would occur only after content had been retrieved.   
Whenever a situation limits an ability to reach the origination, the  
message can be carried in-band, but only after a delay when a Temp  
error is needed.  A greater portion of the effort to prioritize  
resources must be placed upon the transmitter.  The TBR extension  
permits a graceful transition toward this strategy.

The reach-back mechanism can be any protocol.  If imap, a proprietary  
concern should be alleviated.  SMTP is already being stressed, so it  
really should be a different protocol or on a different port.  These  
details are really less important.

-Doug

Date: Fri, 16 Nov 2007 05:57:31 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

--On Wednesday, 14 November, 2007 05:16 -0800 Dave Crocker
<[email-masked]> wrote:

> John C Klensin wrote:
>> But, in fairness to the proposal, the general idea has one
>> advantage.  If one is concerned about source / originator
>> identity and authentication, having to make a real-time
>> direct connection back to the sender's repository permits
>> thinking about much stronger methods than, e.g., header
>> signatures.
> 
> I'm pretty sure it permits nothing stronger at all.
> 
> So I need you to provide some detail.
> 
> I'll also note that a mechanism with this much cost --
> especially at the
> infastructure -- needs a proof-positive guarantee that it will
> provide
> benefits, and not just that it makes benefits possible.

Dave,

To take your comments in reverse order, I dislike this
mechanism.  I strongly encouraged Doug to write it up because I
didn't think repeated references to it without any details were
doing any of us any good.   Now that I've seen and thought about
the details --and read the comments of others -- I like it even
less than I expected to.   I expected to dislike it because I
consider "synchronize and run for the airplane" and "synchronize
before the stinking wireless link goes down again and then
actually read the mail offline" to be _very_ important mail-use
modes.  I am also, to put it mildly unwilling to lose multihop
relaying to reach locations with less-than-perfect or
some-times-of-day-only connectivity.   Consequently, I see it as
costly and burdensome on the receivers (and other "good guys")
while providing the bad guys only a temporary impediment.  

Before the "temporary" is questioned, I have every confidence
that, if we were to establish something like this as the primary
way of transferring mail, the designers of bot and zombie
software would observe that a great many machines run, either
without firewalls or without adequate restrictions on incoming
ports.  They would notice that most others are running
"personal" firewalls with well-known interfaces.  They would
then set up mail databases on the compromised machines, open
ports and compromise firewalls, and then go merrily on their
way.    It is obvious to me that this is possible and that the
hard problem is only in figuring out the details of how to do
it.   Once one does figure it out, the marginal effort to
compromise a network's worth of machines with this newer,
fancier, and smarter software rather than older and more
simplistic versions is trivial.  

Yes, I can also imagine relatively simple countermeasures to the
countermeasures, but getting users, unlike bots, to install and
maintain such countermeasures has historically been hard.  After
all, were prophylactic technology that we understand reasonable
well widely deployed and properly configured and maintained, we
would have no botnets.   So we need to assume that a machine
that can be compromised today using low-quality malware can be
compromised six months hence using high-quality software.
Statistically, the higher threshold -- or the greater use of
machine resources and higher odds of being noticed by the user
-- would drop some machines out of the bot-candidate pool, but,
to the spammer or botnet operator, even losing a few tens or
thousands, or even a few hundreds of thousands, of candidate
machines is trivial.

So please don't take any of my comments as supportive of this
proposal.

Now, all of that said, and repeating my belief that giving up
relaying is too high a price to pay and that making SMTP
sessions much longer...   The critical authentication problem
with techniques based on header signatures is that they require
either a widely-deployed, well-managed, and robust PKI (and we
know where _that_ condition gets us) or they require reliance on
a PKI-alternative that is less robust.  Putting keys or other
authorization credentials in the DNS isn't bad, but DNS spoofing
of various flavors is not exactly unheard of (and I dread the
effects of telling the bad guys that they have to learn that
particular skill and apply it in the vicinity of ISP and
enterprise DNS forwarders).  That problem presumably gets _lots_
better in the presence of DNSSEC, but only if "presence" means
very widespread deployment in resolvers, signatures up and down
the tree, and a really good strategy for dealing with DNSSEC
validation failures at the user end (e.g., we know that popping
up little boxes that say

	Warning: <incomprehensible gibberish> happened, do you
	want to continue?

is not part of such a strategy).  For some reason, I'm not
expecting that this month or even this year.

By contrast, suppose we were to abandon relaying and deploy
IPSec or some other technique that guards against mid-session
TCP hijacking if that is necessary.  Then we suddenly have the
full range of multiple-handshake and key exchange mechanisms,
validation checks with external servers, etc., available to us.
If desired to validate message integrity, one could also use
different and less tricky signature mechanisms, since the
delivered message would be known and one could more easily
bypass the effects of valid in-transit modifications (such as
addition of trace fields).

Tedious?  Worth the trouble for "ordinary" messages?  Almost
certainly not, even if one ignores the considerable costs of
abandoning relaying.  And, as I tried to say, if this is what
one wanted to do, then TBR is far too complicated and expensive.
But the security checks would, IMO, be slightly more secure.

     john

Date: Fri, 16 Nov 2007 09:45:37 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Cc: "David F. Skoll" <[email-masked]>
Sender: [email-masked]

David F. Skoll writes:
> Yeah, I know, but spammers can do that with or without 
> content-filtering. Sure, they'll overwhelm you a bit sooner if you 
> use content-filtering than if you don't, but that's about it.

Absolutely.

How big is the "a bit" you mention, btw? I don't remember, was it about 
a factor of 10 or 20 or so?

Arnt

Date: Thu, 15 Nov 2007 23:32:40 -0800
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 15:59 15-11-2007, Hector Santos wrote:
>But I want to make sure I wasn't mis-read.  To be clear, the 
>suggestion really has nothing to do with anti-spam or even 
>mentioning it.  We already have it saving:
>
>4.5.4.1 Sending Strategy
>
>    ...
>
>    The sender MUST delay retrying a particular destination after one
>    attempt has failed.  In general, the retry interval SHOULD be at
>    least 30 minutes; however, more sophisticated and variable strategies
>    will be beneficial when the SMTP client can determine the reason for
>    non-delivery.
>
>This was excellent insight done at the time. Maybe that is enough, 
>but I do believe a word smith can finesse a new clause or sentence 
>indicating why a short 2nd attempt may be applicable and highly 
>desirable for improved outbound mail operations.  Maybe something like:
>
>    Due to the increase of new SMTP challenge/response receiver
>    systems, if the SMTP client sees 451 temporary response code
>    for the RCPT or DATA state, it MAY consider using a shorter
>    2nd attempt interval, recommended 5-10 minutes.  This SHOULD
>    be for the 2nd attempt only.

The "451" reply code means error in processing.  As it's a policy 
decision, a "450" reply code is more appropriate.

With the advent of greylisting, a shorter second attempt may seem 
desirable to ensure timely delivery of message.  However, we should 
take into account the impact it will have on the receiver.  Such a 
short interval conflates the problem for the receiving system if it's 
under heavy load.  We are also assuming that the greylisting period 
will be under five or ten minutes.  That may not be the case.  For 
the proposed strategy to be effective, the sender would have to 
determine whether the temporary error was due to greylisting (status 
code instead of reply code) and the greylisting delay.

Regards,
-sm 

To: Glenn Anderson <[email-masked]>
Cc: [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
From: [email-masked]
Date: Fri, 16 Nov 2007 00:01:26 -0500
Sender: [email-masked]

--==_Exmh_1195189286_3222P
Content-Type: text/plain; charset=us-ascii

On Fri, 16 Nov 2007 14:46:29 +1300, Glenn Anderson said:

> I don't believe it to be a simple problem for the spammers and bot 
> architects to solve. Not only do they need to start keeping track of 
> state, they need to keep track of a lot of state (the volume of 
> messages they are trying to send is huge), which requires a lot more 
> resources, and they need to manage CPU and network resources such 
> that they are available for them to retry later.

Oh, hogwash.

Years ago (1995 timeframe), I was able to do a million RCPT TO's/day from a
listserv machine, using Sendmail on an IBM RS6000-250.  Little pizza box, 256M
of memory, a whole whopping 66mz 601E processor.  If spammers can't write code
tight enough to do so un-noticed on today's consumer boxes, there's something
*seriously* wrong.

--==_Exmh_1195189286_3222P
Content-Type: application/pgp-signature

--==_Exmh_1195189286_3222P--

Date: Thu, 15 Nov 2007 21:20:23 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

John C Klensin wrote:

> Unfortunately, graylisting is one of those techniques that works
> well as long as sufficiently few people use it that the spammers
> and bot architects don't feel motivated to go to the extra work
> to overcome it.

No, I don't agree.  Greylisting is useful to allow RBLs time to catch
up.  Forcing a sender to send from the same IP address for 20-30 minutes
or so can be useful.

> My guess is that we have passed at least the
> first version of that point: I'm seeing a rapidy increasing
> number of spam messages arriving in a one-two sequence from the
> same putative source.  First one message is sent, then a second
> is sent a few minutes later.  That doesn't even require that the
> bot maintain state, although graylisting that actually keeps
> track of message headers or signatures will.  

We keep a hash of some message content and we find it to be quite
effective against ratware that mutates the message with each retry.
Unfortunately, this means we can't greylist until post-DATA, but
that's a tradeoff we're willing to make.

> This brings us back to the point I tried to make to Hector:
> making these folks smarter may be unwise, especially when doing
> so consumes more resources on our and and, with botnets, they
> have essentially unlimited resources for which the costs to them
> are trivial.

Except that pinning them to the same IP address for a while lets
RBLs catch up so you can reject connection attempts very cheaply.

> And don't ask that we change the standards to make them more
> friendly to anti-spam techniques that can reasonably expected to
> have a relatively short lifespan.

I agree with that.  I don't think greylisting deserves official
recognition in an RFC.  As much as I like it, it is at the end of
the day a hack. :-)

Regards,

David.

Date: Fri, 16 Nov 2007 14:46:29 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

At 6:51 pm -0500 15/11/2007, John C Klensin wrote:
>Unfortunately, graylisting is one of those techniques that works
>well as long as sufficiently few people use it that the spammers
>and bot architects don't feel motivated to go to the extra work
>to overcome it.

I don't believe it to be a simple problem for the spammers and bot 
architects to solve. Not only do they need to start keeping track of 
state, they need to keep track of a lot of state (the volume of 
messages they are trying to send is huge), which requires a lot more 
resources, and they need to manage CPU and network resources such 
that they are available for them to retry later. To some extent they 
also need to make sure the receiving end has enough resources to 
accept their connections, some other spammer overloading a server 
will prevent them from sending spam to it. I consider resource 
management to be the fundamental problem for spammers, as the bulk of 
the is spam coming from bots using hijacked resources. They don't 
have total control over those resources (eg: user can switch of the 
machine, dynamic IP can change), and most importantly if they were 
able to hijack those resources, someone else can. The later turns 
this in to a Tragedy of the Commons type problem for the spammers.

>My guess is that we have passed at least the
>first version of that point: I'm seeing a rapidy increasing
>number of spam messages arriving in a one-two sequence from the
>same putative source.  First one message is sent, then a second
>is sent a few minutes later.  That doesn't even require that the
>bot maintain state, although graylisting that actually keeps
>track of message headers or signatures will.

I'm using a graylisting delay of 25 minutes. Tens of thousands of 
bots keeping sufficient state for more than 25 minutes, and then 
coordinating things such that they can get through to my server on 
the 24 connections allocated to them, would require quite a radical 
change in behavior.

Of the ~150,000 unique IPs that have connected to my server in the 
last two weeks, just over 70% of them only connected once.

>This brings us back to the point I tried to make to Hector:
>making these folks smarter may be unwise, especially when doing
>so consumes more resources on our and and, with botnets, they
>have essentially unlimited resources for which the costs to them
>are trivial.

I consider spammers to have the kind of selfish mentality that could 
never avoid being trapped by a Tragedy of the Commons. I can't see 
spammers not fighting over bots, not hogging all the resources on 
hijacked machines, and not hogging resources on receiving servers.

>So, since you are graylisting already, by all means enjoy the
>advantages as long as they last.  But, given what I think we are
>seeing already, don't expect them to last for a long time.  And
>don't ask that we change the standards to make them more
>friendly to anti-spam techniques that can reasonably expected to
>have a relatively short lifespan.

I definitely don't think the standards should change, I'm quite happy 
with the current recommendation of 30 minutes to retry.

Glenn.

Date: Thu, 15 Nov 2007 19:50:23 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: Glenn Anderson <[email-masked]>, [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

John C Klensin wrote to Glen:

> This brings us back to the point I tried to make to Hector:
> making these folks smarter may be unwise, especially when doing
> so consumes more resources on our and and, with botnets, they
> have essentially unlimited resources for which the costs to them
> are trivial.   

Ok, then you did mis-read me.

#1, I am against any I-D or IETF proposal for GL SMTP implementation. 
Maybe "against" the wrong word - rather waste of time, partially for the 
reasons you stated.  Mostly because it is "fundamentally unnatural"
to reject an email on this basis. I am fundamentally oppose to the idea.

#2, Reality Check.  They exist in the market, so whether I, you and
the your IETF peers don't agree with it, it is a fact of life. GL and 
other challenge/response systems exist which are based on checking for a 
compliant SMTP retry system, and with a blind consideration for whether 
the anonymous sender is bad or good - they are treated all the same.

So we already have insights regarding the possibility of "sophisticated 
variable retry strategies," all I am suggesting is that it is modernized
to highlight that there is a real reason today for this "sophistication" 
to exist with a recommended shorter 2nd attempt in order to improve 
outbound delivery operations or to minimize the impact of "living" in 
the real world.

If you believe that world is short live, which I don't 100% disagree 
(Medicine cures) then why even have the suggestion about the possibility 
for variable retry methods in the first place? :-)

I think it is solid engineering consideration that has nothing to do 
with introducing that evil term "anti-spam solution" into SMTP or 
specific to GL.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 15 Nov 2007 19:16:30 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Glenn Anderson <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Sender: [email-masked]

Hi Glen,

Thats certainly possible.  But just to be clear, I wasn't referring to 
the GL blocking delays, but the outbound sending retry frequency.

The facts are real: GL systems exist and growing every day. So even if a 
system is not interested in implementing it on the receiving side, there 
is a delayed delivery impact due to the increasing odds of hitting a 
remote GL compliant receiver. A shorted 2nd attempt will minimize this 
impact.

I personally think your GL block of 25 minutes is too high for most 
systems with a higher expectation of business or customer transactions.

But thats ok? It works for you. :-)

Hmmmm, I wonder if there could be an algoritm or heuristic to make the 
block time variable too based on who is the sender?  But I do think the 
beauty of GL is that it treats all anonymous senders the same.

--
HLS

Glenn Anderson wrote:

>>
>> I recommend a 5 min retry on the 2nd attempt, with a fall back to 
>> their normal interval after that.
> 
> My MTA has a default graylisting delay of 25 minutes. 25 minutes was 
> chosen because it wouldn't unnecessarily delay for hosts that use the 
> recommended 30 minute retry, and because at the time I saw a significant 
> number of what appeared to be bots that were retrying 5 times 5 minutes 
> apart (~20 minutes between the first and last retry).
> 
> Glenn.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 15 Nov 2007 18:59:43 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Sender: [email-masked]

Hi John,

Ok, I'm not going to waste time with an I-D, so if Tony or others want 
to address it, great, otherwise, I'm not going to worry about it. :-)

But I want to make sure I wasn't mis-read.  To be clear, the suggestion 
really has nothing to do with anti-spam or even mentioning it.  We 
already have it saving:

4.5.4.1 Sending Strategy

    ...

    The sender MUST delay retrying a particular destination after one
    attempt has failed.  In general, the retry interval SHOULD be at
    least 30 minutes; however, more sophisticated and variable strategies
    will be beneficial when the SMTP client can determine the reason for
    non-delivery.

This was excellent insight done at the time. Maybe that is enough, but I 
do believe a word smith can finesse a new clause or sentence indicating 
why a short 2nd attempt may be applicable and highly desirable for 
improved outbound mail operations.  Maybe something like:

    Due to the increase of new SMTP challenge/response receiver
    systems, if the SMTP client sees 451 temporary response code
    for the RCPT or DATA state, it MAY consider using a shorter
    2nd attempt interval, recommended 5-10 minutes.  This SHOULD
    be for the 2nd attempt only.

If not, never mind. :-)

--
HLS

John C Klensin wrote:
> Hector,
> 
> Tony may take a different position, but my response to this
> suggestion is the same as my response to all sorts of other
> anti-spam or spammer-conditioned proposed changes to 2821: 
> 
> Write a separate proposal, as an I-D, to make the change.  Get
> it approved as a Proposed Standard.  Get it implemented and
> deployed.  When the timer runs out, get it approved as a Draft
> Standard.  Do all of this efficiently enough that, by the time
> discussions of 2821ter come up, your proposal will be of full
> Standard maturity and deployment so that it can be folded in.
> 
> For 2821bis, this would be a significant change to a
> long-established piece of protocol.   It is too late, both
> procedurally and, IMO, technically.
> 
>     john
> 
> 
> --On Thursday, 15 November, 2007 17:58 -0500 Hector Santos
> <[email-masked]> wrote:
> 
>> John,
>>
>> Is there worth to this?
>>
>> My recommendation for 2821bis is to include some insights
>> about or how variable strategies" plays a bigger role these
>> days and in fact, might be almost an "necessity" for improved
>> modern operations these days.
>>
>> Even if one didn't want to incorporate GL into their SMTP
>> receiver, the default sending retry strategy (if it matches
>> the current specs) has to be reconsidered due to increased
>> remote GL systems and the growing probability of hitting such
>> a system.
>>
>> In short, they might be need to reconsider the recommended "30
>> minute" retry interval, at least for the 2nd attempt.
>>
>> I recommend a 5 min retry on the 2nd attempt, with a fall back
>> to their normal interval after that.
>>
>> --
>> HLS
> 
> 
> 
> 
> 
> 

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 15 Nov 2007 18:51:08 -0500
From: John C Klensin <[email-masked]>
To: Glenn Anderson <[email-masked]>, [email-masked]
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

--On Friday, 16 November, 2007 12:13 +1300 Glenn Anderson
<[email-masked]> wrote:

>...
> Connection prioritization helps keep mail flowing from hosts
> that aren't being graylisted. My MTA has two thresholds for
> how many SMTP connections are remaining, when the first one is
> exceeded connections from new hosts or hosts that are still
> graylisted are not accepted. When the second one is exceeded
> connections are only accepted from hosts that have been
> whitelisted.
> 
> The minor delays due to graylisting are far outweighed by it's
> useful anti-DDoS features. Anti-spam solutions that want to
> replace graylisting are of no interest to me if they can't
> replace that functionality.

Glen,

Unfortunately, graylisting is one of those techniques that works
well as long as sufficiently few people use it that the spammers
and bot architects don't feel motivated to go to the extra work
to overcome it.  My guess is that we have passed at least the
first version of that point: I'm seeing a rapidy increasing
number of spam messages arriving in a one-two sequence from the
same putative source.  First one message is sent, then a second
is sent a few minutes later.  That doesn't even require that the
bot maintain state, although graylisting that actually keeps
track of message headers or signatures will.  

This brings us back to the point I tried to make to Hector:
making these folks smarter may be unwise, especially when doing
so consumes more resources on our and and, with botnets, they
have essentially unlimited resources for which the costs to them
are trivial.   

So, since you are graylisting already, by all means enjoy the
advantages as long as they last.  But, given what I think we are
seeing already, don't expect them to last for a long time.  And
don't ask that we change the standards to make them more
friendly to anti-spam techniques that can reasonably expected to
have a relatively short lifespan.

Just my opinion, of course...

   john

Date: Thu, 15 Nov 2007 18:35:02 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>
cc: [email-masked]
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Sender: [email-masked]

Hector,

Tony may take a different position, but my response to this
suggestion is the same as my response to all sorts of other
anti-spam or spammer-conditioned proposed changes to 2821: 

Write a separate proposal, as an I-D, to make the change.  Get
it approved as a Proposed Standard.  Get it implemented and
deployed.  When the timer runs out, get it approved as a Draft
Standard.  Do all of this efficiently enough that, by the time
discussions of 2821ter come up, your proposal will be of full
Standard maturity and deployment so that it can be folded in.

For 2821bis, this would be a significant change to a
long-established piece of protocol.   It is too late, both
procedurally and, IMO, technically.

    john

--On Thursday, 15 November, 2007 17:58 -0500 Hector Santos
<[email-masked]> wrote:

> John,
> 
> Is there worth to this?
> 
> My recommendation for 2821bis is to include some insights
> about or how variable strategies" plays a bigger role these
> days and in fact, might be almost an "necessity" for improved
> modern operations these days.
> 
> Even if one didn't want to incorporate GL into their SMTP
> receiver, the default sending retry strategy (if it matches
> the current specs) has to be reconsidered due to increased
> remote GL systems and the growing probability of hitting such
> a system.
> 
> In short, they might be need to reconsider the recommended "30
> minute" retry interval, at least for the 2nd attempt.
> 
> I recommend a 5 min retry on the 2nd attempt, with a fall back
> to their normal interval after that.
> 
> --
> HLS

Date: Fri, 16 Nov 2007 12:29:23 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Sender: [email-masked]

At 5:58 pm -0500 15/11/2007, Hector Santos wrote:
>John,
>
>Is there worth to this?
>
>My recommendation for 2821bis is to include some insights about or 
>how variable strategies" plays a bigger role these days and in fact, 
>might be almost an "necessity" for improved modern operations these 
>days.
>
>Even if one didn't want to incorporate GL into their SMTP receiver, 
>the default sending retry strategy (if it matches the current specs) 
>has to be reconsidered due to increased remote GL systems and the 
>growing probability of hitting such a system.
>
>In short, they might be need to reconsider the recommended "30 
>minute" retry interval, at least for the 2nd attempt.
>
>I recommend a 5 min retry on the 2nd attempt, with a fall back to 
>their normal interval after that.

My MTA has a default graylisting delay of 25 minutes. 25 minutes was 
chosen because it wouldn't unnecessarily delay for hosts that use the 
recommended 30 minute retry, and because at the time I saw a 
significant number of what appeared to be bots that were retrying 5 
times 5 minutes apart (~20 minutes between the first and last retry).

Glenn.

Date: Fri, 16 Nov 2007 12:13:27 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

At 4:48 pm -0500 15/11/2007, David F. Skoll wrote:
>Arnt Gulbrandsen wrote:
>
>[...]
>
>>  There was an excellent article in the magazine iX on the subject earlier
>>  this year, illustrated with a number of depressing rrdtool/mrtg graphs.
>>  Dreary reading. Very brief summary: 1. Spammers whose botnets span n
>>  hundred thousand boxes can shovel more mail onto your MXes than they can
>>  handle. 2. Sometimes they do. 3. When they do, that's no fun at all for
>  > the sysadmins on the receiving end.

Having been at the receiving end for more than a year now, it 
definitely isn't fun. I've been seeing tens of thousands of bots a 
day hit my server for quite a while now, although the volume of 
connections from the bots has dropped off, and the volume of bots has 
dropped to about 10,000/day. And this is quite a modest server that 
just handles email for me, if spammers are throwing that much bot 
traffic at me, I'd hate to think what they might throw at a larger 
site.

>Yeah, I know, but spammers can do that with or without content-filtering.
>Sure, they'll overwhelm you a bit sooner if you use content-filtering
>than if you don't, but that's about it.

I've found graylisting combined with connection prioritization to be 
quite effective at minimizing the impact of DDoS attacks from 
bot-nets. Graylisting gives me the following benefits that I find to 
be useful against DDoS attacks:
* Transactions from new hosts are kept very short.
* My server gets to look at the HELO or EHLO name and MAIL FROM and 
RCPT TO addresses, and reject many hosts based on those.
* My server doesn't hand out any information on validity of RCPT TO 
addresses to hosts that are graylisted.

Connection prioritization helps keep mail flowing from hosts that 
aren't being graylisted. My MTA has two thresholds for how many SMTP 
connections are remaining, when the first one is exceeded connections 
from new hosts or hosts that are still graylisted are not accepted. 
When the second one is exceeded connections are only accepted from 
hosts that have been whitelisted.

The minor delays due to graylisting are far outweighed by it's useful 
anti-DDoS features. Anti-spam solutions that want to replace 
graylisting are of no interest to me if they can't replace that 
functionality.

Glenn.

Date: Thu, 15 Nov 2007 15:11:22 -0800
To: Douglas Otis <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Cc: [email-masked]
Sender: [email-masked]

At 12:19 15-11-2007, Douglas Otis wrote:
>Only to HTTP servers located within the "_tbr." subdomain.  This will
>not impact other websites.  In addition, fairly straight forward 
>anti- abuse measures can be applied, and are recommended in the draft.

Some sites may use the same infrastructure to host their other web 
services.  That angle may have to be explored further.

>>>5) protect valid email-address confidentiality
>>
>>See Section 7.6 of RFC 2821 about information disclosure in message
>>forwarding.
>
>This was not the concern.  An attempt to send to an invalid recipient
>is likely to return an error which may indicate their non-existence.

When a message is forwarded, the sender doesn't know the final 
recipient.  I was pointing out at disclosure of information about the 
final recipient.

A side-effect of using SPF for the first part above is that sender 
address rewrite generally suggested as a solution for forwarding 
doesn't fit into TBR.

Regards,
-sm  

Date: Thu, 15 Nov 2007 17:58:46 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: 2821bis consideration - New 2nd attempt Retry Strategy recommendation
Sender: [email-masked]

John,

Is there worth to this?

My recommendation for 2821bis is to include some insights about or how 
variable strategies" plays a bigger role these days and in fact, might 
be almost an "necessity" for improved modern operations these days.

Even if one didn't want to incorporate GL into their SMTP receiver, the 
default sending retry strategy (if it matches the current specs) has to 
be reconsidered due to increased remote GL systems and the growing 
probability of hitting such a system.

In short, they might be need to reconsider the recommended "30 minute" 
retry interval, at least for the 2nd attempt.

I recommend a 5 min retry on the 2nd attempt, with a fall back to their 
normal interval after that.

--
HLS

John C Klensin wrote:
> Hector,
> 
> I disagree.   I would construe this situation as one in which we
> are making the bad guys smarter, driving up our costs for
> detecting them and their products.  I have trouble considering
> that a benefit.
> 
> A medicine that makes a bacterium or virus treatment-resistant,
> especially when we lack treatments for the resistant forms
> and/or when the resistant forms are move virulent and deadly
> than the originals, is not a completely desirable treatment or
> mechanism.
> 
>     john
> 
> 
> --On Thursday, 15 November, 2007 15:44 -0500 Hector Santos
> <[email-masked]> wrote:
> 
>> Douglas Otis wrote:
>>  >
>>  > Bulk emailers are often anxious to complete their campaigns
>> and are
>>  > dubious about retry delays of 30 minutes.  The initial
>> benefit of
>>  > grey-listing was as a test for whether a transmitting MTA
>> carried
>>  > state since many bots did not.  This is changing where now
>> many
>>  > bots retry the same message.  The benefits from grey-listing
>>  > are fading, while bulk emailers have become more aggressive
>> in
>>  > their retries.
>>
>> That's good Doug.
>>
>> Good medicines are the ones where you won't eventually need it
>> anymore.
>>
>> Good medicines also helps separate the bad from the good from
>> a compliance standpoint, so if the Bulk mailers are getting
>> better at what they were suppose to do in the first place,
>> then thats good. Not bad.
>>
>> In the mean time, for the systems who don't follow the specs,
>> GL and other SMTP compliance-based ideas will continue to be
>> effective.
>>
>> So your statement about "fading", true or not, I see that as a
>> positive, not a negative sign that well it worked.
> 
> 
> 
> 
> 
> 

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 15 Nov 2007 17:37:07 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: Douglas Otis <[email-masked]>, [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John,

I can see your point, and I don't disagree.

My philosophy is such we have the same problem regardless of the 
"add-on" protocol.   Of course, if our goal is to reduce the bad guy 
issues, then enforcement ideas are required.  But as long as legacy 
operations is fundamental requirement (on the public PORT 25), we will 
always have a "bad guy" problem exploiting the legacy designs including 
those with add-ons.

In regards specifically with GL, I too was skeptical and I too did not 
want to introduce a "mickey mouse" kludge in our system, but if there 
are customers demands for it, well, you have to provide it and when we 
did, as I mentioned, everyone jaws dropped at how well it worked.

In my view, I can attribute the "acceptance" to the idea that we don't 
cater to the public market place, our software is very much dedicated to 
  customers who have two types of message - good and bad and the rates 
are that the BAD is really BAD and the GOOD is really GOOD with very low 
"impact" (reports of issues).  Anything that causes or increase our 
support cost is quickly reconsidered and addressed.

But as I also said, it was immediately apparent you need to change to 
the variable retry frequency with a shorter 2nd retry and most 
definitely you need to incorporate an auto whitelist concept to quickly 
get your "network" of good guy customers on the list, etc.  If you send 
an email to someone, it is automatically white list. That includes 
announcements.

I can agree that any protocol can help induce an increase of bulk 
senders, but I wouldn't blame it on the protocol. I would venture to say 
they are also trying to pressure a system of lowering their guards.

I feared the same exact increase issues with 822 based protocols such as 
DKIM especially when its becomes widely adopted. I already stated this a 
few times.  Bad guys can begin to make the payload MUCH higher to cause 
scalability issues and with increase sending, that can become a problem. 
  If there isn't a solution, an operator may be forced to turn off DKIM 
verification. I haven't seen any consideration or discussion in the DKIM 
WG regarding the hashing of large attachments and the overhead 
associated with that.

But is that going to be a show stopper? I doubt it and in fact, mostly 
completely ignored, specially if I am the one making this statement. :-)

--
HLS

John C Klensin wrote:
> Hector,
> 
> I disagree.   I would construe this situation as one in which we
> are making the bad guys smarter, driving up our costs for
> detecting them and their products.  I have trouble considering
> that a benefit.
> 
> A medicine that makes a bacterium or virus treatment-resistant,
> especially when we lack treatments for the resistant forms
> and/or when the resistant forms are move virulent and deadly
> than the originals, is not a completely desirable treatment or
> mechanism.
> 
>     john
> 
> 
> --On Thursday, 15 November, 2007 15:44 -0500 Hector Santos
> <[email-masked]> wrote:
> 
>> Douglas Otis wrote:
>>  >
>>  > Bulk emailers are often anxious to complete their campaigns
>> and are
>>  > dubious about retry delays of 30 minutes.  The initial
>> benefit of
>>  > grey-listing was as a test for whether a transmitting MTA
>> carried
>>  > state since many bots did not.  This is changing where now
>> many
>>  > bots retry the same message.  The benefits from grey-listing
>>  > are fading, while bulk emailers have become more aggressive
>> in
>>  > their retries.
>>
>> That's good Doug.
>>
>> Good medicines are the ones where you won't eventually need it
>> anymore.
>>
>> Good medicines also helps separate the bad from the good from
>> a compliance standpoint, so if the Bulk mailers are getting
>> better at what they were suppose to do in the first place,
>> then thats good. Not bad.
>>
>> In the mean time, for the systems who don't follow the specs,
>> GL and other SMTP compliance-based ideas will continue to be
>> effective.
>>
>> So your statement about "fading", true or not, I see that as a
>> positive, not a negative sign that well it worked.
> 
> 
> 
> 
> 
> 

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
To: "David F. Skoll" <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference, changed delivery requirement
Date: Thu, 15 Nov 2007 14:34:23 -0800
Sender: [email-masked]

On Nov 15, 2007, at 12:10 PM, David F. Skoll wrote:

>>
>> By providing the TBR Extension, the outbound serve is able to  
>> complete their transaction using the TBR verb now rather than the  
>> DATA verb at some point later.  This eliminates any need to retry.
>
> But it still means the sender has to hang on to the message.  And  
> worse, it cannot possibly know when its safe to relinquish it  
> because mailing-list exploders can cause the number of required  
> fetches to be indeterminate.

Perhaps retraction of the alternative mode specified in section 3.2  
needs to be strengthened.

3.2.  TBR Transaction
...

,---
| As an alternative, instead of retrieving the replacement message
| content, it MAY prepend any additional trace headers to a
| notification sent to the recipient containing the eXAM-URI itself,
| along with any other appropriate information.
'___

Changed to:
,~~~
: An alternative notification containing the eXAM-URI and not the
: retrieved message content SHOULD NOT be used. The message is not
: delivered unless the message content is retrieved.  Exploding the
: number of recipients MUST NOT occur without first retrieving
: message content.
'~~~

-Doug

Date: Thu, 15 Nov 2007 16:54:07 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, Douglas Otis <[email-masked]>
cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Hector,

I disagree.   I would construe this situation as one in which we
are making the bad guys smarter, driving up our costs for
detecting them and their products.  I have trouble considering
that a benefit.

A medicine that makes a bacterium or virus treatment-resistant,
especially when we lack treatments for the resistant forms
and/or when the resistant forms are move virulent and deadly
than the originals, is not a completely desirable treatment or
mechanism.

    john

--On Thursday, 15 November, 2007 15:44 -0500 Hector Santos
<[email-masked]> wrote:

> 
> Douglas Otis wrote:
>  >
>  > Bulk emailers are often anxious to complete their campaigns
> and are
>  > dubious about retry delays of 30 minutes.  The initial
> benefit of
>  > grey-listing was as a test for whether a transmitting MTA
> carried
>  > state since many bots did not.  This is changing where now
> many
>  > bots retry the same message.  The benefits from grey-listing
>  > are fading, while bulk emailers have become more aggressive
> in
>  > their retries.
> 
> That's good Doug.
> 
> Good medicines are the ones where you won't eventually need it
> anymore.
> 
> Good medicines also helps separate the bad from the good from
> a compliance standpoint, so if the Bulk mailers are getting
> better at what they were suppose to do in the first place,
> then thats good. Not bad.
> 
> In the mean time, for the systems who don't follow the specs,
> GL and other SMTP compliance-based ideas will continue to be
> effective.
> 
> So your statement about "fading", true or not, I see that as a
> positive, not a negative sign that well it worked.

Date: Thu, 15 Nov 2007 16:48:34 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: DoS attacks (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

Arnt Gulbrandsen wrote:

[...]

> There was an excellent article in the magazine iX on the subject earlier
> this year, illustrated with a number of depressing rrdtool/mrtg graphs.
> Dreary reading. Very brief summary: 1. Spammers whose botnets span n
> hundred thousand boxes can shovel more mail onto your MXes than they can
> handle. 2. Sometimes they do. 3. When they do, that's no fun at all for
> the sysadmins on the receiving end.

Yeah, I know, but spammers can do that with or without content-filtering.
Sure, they'll overwhelm you a bit sooner if you use content-filtering
than if you don't, but that's about it.

Regards,

David.

Date: Thu, 15 Nov 2007 16:17:26 -0500
From: "David F. Skoll" <[email-masked]>
To: Douglas Otis <[email-masked]>, [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Douglas Otis wrote:

>> It may eliminate backscatter but it introduces a new problem, i.e. it
>> can be used to stage  attacks using HTTP.

> Only to HTTP servers located within the "_tbr." subdomain.  This will
> not impact other websites.

Oh yes?

Then are you OK with my publishing the following DNS record?

     _tbr.roaringpenguin.com 86400 IN A 168.61.5.26

Regards,

David.

Date: Thu, 15 Nov 2007 15:44:57 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Douglas Otis <[email-masked]>
CC: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Douglas Otis wrote:
 >
 > Bulk emailers are often anxious to complete their campaigns and are
 > dubious about retry delays of 30 minutes.  The initial benefit of
 > grey-listing was as a test for whether a transmitting MTA carried
 > state since many bots did not.  This is changing where now many
 > bots retry the same message.  The benefits from grey-listing
 > are fading, while bulk emailers have become more aggressive in
 > their retries.

That's good Doug.

Good medicines are the ones where you won't eventually need it anymore.

Good medicines also helps separate the bad from the good from a 
compliance standpoint, so if the Bulk mailers are getting better at what 
they were suppose to do in the first place, then thats good. Not bad.

In the mean time, for the systems who don't follow the specs, GL and 
other SMTP compliance-based ideas will continue to be effective.

So your statement about "fading", true or not, I see that as a positive, 
not a negative sign that well it worked.

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Thu, 15 Nov 2007 21:42:01 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Cc: "David F. Skoll" <[email-masked]>, Douglas Otis <[email-masked]>
Sender: [email-masked]

David F. Skoll writes:
> Douglas Otis wrote:
>> Understandable. The current levels of spam, which in some cases is 
>> exceeding 99%, simply overwhelms any ability to filter all messages.
>
> Do you have evidence to support that?
>
>>  Content filtering alone can not keep up,
>
> Do you have evidence to support that?

If you can read German.

There was an excellent article in the magazine iX on the subject earlier 
this year, illustrated with a number of depressing rrdtool/mrtg graphs. 
Dreary reading. Very brief summary: 1. Spammers whose botnets span n 
hundred thousand boxes can shovel more mail onto your MXes than they 
can handle. 2. Sometimes they do. 3. When they do, that's no fun at all 
for the sysadmins on the receiving end.

Arnt

To: "David F. Skoll" <[email-masked]>
Cc: Douglas Otis <[email-masked]>, [email-masked]
Subject: Re: SMTP Transferred-By-Reference
From: [email-masked]
Date: Thu, 15 Nov 2007 15:38:30 -0500
Sender: [email-masked]

--==_Exmh_1195159110_3222P
Content-Type: text/plain; charset=us-ascii

(re-adding ietf-smtp, it appeared to get bcc:ed so 'reply' didnt see it..)

On Thu, 15 Nov 2007 14:20:48 EST, "David F. Skoll" said:
> 
> Douglas Otis wrote:
> 
> > To support this effort, the TBR Extension offers a low overhead
> > means to defer a formal obligation to deliver, while also avoiding
> > the exchange of undesired data.
> 
> You still haven't explained this:  What's in it for the sender?
> Why would an e-mail sender be interested in deferring the receiving SMTP
> server's obligation to deliver?  As an e-mail sender, I want the e-mail
> out of my hair as soon as possible!  I don't want to have to hang on to it!
> 
> And since ultimately the sender chooses whether or not to use TBR, you
> have to explain...
> 
>      What's in it for the sender?

Even *more* important - what's in it for the sender *and* is a *disincentive*
to nefarious-minded senders?  Remember that spammers were among the *first*
to deploy SPF (yes, I *know* that case is an abuse of the difference between
what SPF *really* is and what most people *think* it is - but that's exactly
the space that spammers live in: abusing the corner cases). 

--==_Exmh_1195159110_3222P
Content-Type: application/pgp-signature

--==_Exmh_1195159110_3222P--

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
To: SM <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Date: Thu, 15 Nov 2007 12:19:34 -0800
Sender: [email-masked]

On Nov 15, 2007, at 11:41 AM, SM wrote:

> At 10:02 15-11-2007, Douglas Otis wrote:
> The TBR extension can:
>
>> 1) without burdening the receiver
>>
>> a- provide a valid identity of origination
>>
>> b- eliminate back-scatter
>
> It may eliminate backscatter but it introduces a new problem, i.e.  
> it can be used to stage  attacks using HTTP.

Only to HTTP servers located within the "_tbr." subdomain.  This will  
not impact other websites.  In addition, fairly straight forward anti- 
abuse measures can be applied, and are recommended in the draft.

>> 5) protect valid email-address confidentiality
>
> See Section 7.6 of RFC 2821 about information disclosure in message  
> forwarding.

This was not the concern.  An attempt to send to an invalid recipient  
is likely to return an error which may indicate their non-existence.   
Dropping any acknowledged message will have the effect of lowering  
email's delivery integrity.  For those messages being handled by  
incoming filtering services, RFC8221 section 4.2.5 Reply Codes After  
DATA and the Subsequent <CRLF>.<CRLF> requires either the message be  
delivered, or a DSN be made.  These services will either produce a  
flurry of backscatter, cause messages to be lost, or permit valid  
recipient addresses to be discovered.  The TBR extensions is better  
able to obfuscate whether a message was refused due an opinion of the  
origination, or due to an invalid recipient.  Seeing a series of  
invalid recipients would be a clear indication of this being abusive.   
Any lucking guesses can be expunged when a short hold is placed upon  
suspicious points of origination.

-Doug

Date: Thu, 15 Nov 2007 15:10:20 -0500
From: "David F. Skoll" <[email-masked]>
To: Douglas Otis <[email-masked]>, [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Douglas Otis wrote:
> On Nov 15, 2007, at 11:20 AM, David F. Skoll wrote:
>> You still haven't explained this:  What's in it for the sender?

> Imagine the outbound server is that of a bulk emailer or that of a large
> ISP.  Our company issues realtime temp error blocks resulting from
> ongoing spam runs emerging from a server.  These blocks protect
> resources needed to content filter incoming email and prevent an
> otherwise global denial of service.  Currently, the affected server will
> receive a Temp 451 error and needs to retry the message at a later point
> in time.  To discourage rapid retries, dispositions will not change in
> less than an hour.

Right.  So anyone who runs a large e-mail setup knows that you shuffle
tempfailed mail off to a fallback MX host.  Your primary hosts' queues
stay nice and small and the fallback MX host can be optimized for deep
queues.

> By providing the TBR Extension, the outbound serve is able to complete
> their transaction using the TBR verb now rather than the DATA verb at
> some point later.  This eliminates any need to retry.

But it still means the sender has to hang on to the message.  And worse,
it cannot possibly know when its safe to relinquish it because mailing-list
exploders can cause the number of required fetches to be indeterminate.

>> Why would an e-mail sender be interested in deferring the receiving
>> SMTP server's obligation to deliver?  As an e-mail sender, I want the
>> e-mail out of my hair as soon as possible!  I don't want to have to
>> hang on to it!

> Understandable.  The current levels of spam, which in some cases is
> exceeding 99%, simply overwhelms any ability to filter all messages.

Do you have evidence to support that?

> Content filtering alone can not keep up,

Do you have evidence to support that?

> Content filtering is also, in the long run, the wrong approach.

That's your opinion, not a fact.  I happen to believe the opposite:
Content-filtering is the only long-run viable approach because in the end,
the spammer can fake everything but he/she still has to get the sales
pitch across.

> A bad actor is always able to find new ways to obfuscate their
> message.

A bad actor won't use TBR.  And a bad actor will subvert TBR, including
abusing it to *create* denial-of-service attacks as I illustrated in another
message.

[...]

> Whether you are holding the message in an outbound queue, or make it
> reachable through a reference, servers being flagged as currently
> sending spam will need to hold the message.  Nothing is able to scale in
> a manner that could allow an unlimited amount of abuse.  The TBR
> extension provides an effective alternative that should make life easier
> for both the transmitter and the receiver.

I don't see how it makes it easier for the transmitter.

> The TBR extension provides the sender an alternative to continuously
> retrying to send the same message.  The TBR extension allows these
> otherwise blocked senders a means to complete the SMTP portion of the
> exchange.

How many senders currently see this as a problem in need of addressing?
A good greylisting implementation will have a fairly small impact on
valid senders.

Regards,

David.

Date: Thu, 15 Nov 2007 11:41:51 -0800
To: Douglas Otis <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Cc: [email-masked]
Sender: [email-masked]

At 10:02 15-11-2007, Douglas Otis wrote:
The TBR extension can:

>1) without burdening the receiver
>
>  a- provide a valid identity of origination
>
>  b- eliminate back-scatter

It may eliminate backscatter but it introduces a new problem, i.e. it 
can be used to stage  attacks using HTTP.

>5) protect valid email-address confidentiality

See Section 7.6 of RFC 2821 about information disclosure in message forwarding.

Regards,
-sm 

Date: Thu, 15 Nov 2007 14:20:48 -0500
From: "David F. Skoll" <[email-masked]>
To: Douglas Otis <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Douglas Otis wrote:

> To support this effort, the TBR Extension offers a low overhead
> means to defer a formal obligation to deliver, while also avoiding
> the exchange of undesired data.

You still haven't explained this:  What's in it for the sender?
Why would an e-mail sender be interested in deferring the receiving SMTP
server's obligation to deliver?  As an e-mail sender, I want the e-mail
out of my hair as soon as possible!  I don't want to have to hang on to it!

And since ultimately the sender chooses whether or not to use TBR, you
have to explain...

     What's in it for the sender?

-- David.

Date: Thu, 15 Nov 2007 14:10:29 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Douglas Otis wrote:

> The benefits from grey-listing are fading,

I dispute that.  Our statistics show that greylisting has remained very
effective.  For example, login demo/demo at the following URL:

http://www.roaringpenguin.com/canit/statistics.php?pe=1&r=daily-greylisting&domain=&cur_stream_only=0&num_days=30

> while bulk emailers have become more aggressive in their retries.

That's probably true, but not much of a bother.

> I agree.  The TBR Extension should satisfy the desires of bulk emailers,
> while also providing an absolutely essential mechanism needed to protect
> exhaustive content filtering, the Temp error.

TBR is not needed.  You can write an SMTP server that accepts the
message and then just sits on it for any desired amount of time.  It can
then do whatever reputation-checking it wants to decide whether or not
to deliver the mail.

> The TBR extension can:
> 1) without burdening the receiver
No, the receiver has to add support for TBR.

>  a- provide a valid identity of origination
No, how so?  Anyone can register domains and set up DNS.  That
proves nothing.

>  b- eliminate back-scatter
Maybe.

> 2) conserve limited content assessment resources
At some point, you've got to decide on all your mail.  So just deferring
processing doesn't help if you can't keep up with the flood.

> 3) improve delivery integrity
How so?

> 4) eliminate bulk emailer's aggressive reties
Bulk emailers won't adopt TBR so this is moot.

> 5) protect valid email-address confidentiality
There are many other ways to do this without SMTP extensions.

> 6) defer and enhance assessments of questionable messages
Greylisting does that now without SMTP extensions.

> 7) avoid the DATA phase for abusive sources
Can be done already anyway.

> 8) avoid unintended DDoS effects
Or magnify them.  Look, suppose I decide I want to hurt "example.com".
I register a domain "example.net" and make my example.net URL point at
the IP address of example.com's Web server.  I then merrily send out
millions of e-mails and poor old example.com's Web server is DoS'd
as TBR implementations attempt to fetch a URL on it.

The fatal flaw is that there's no linkage between the owner of a domain
name and the owner of the IP address its A record points to.

--
David.

Cc: [email-masked], [email-masked]
From: Douglas Otis <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Date: Thu, 15 Nov 2007 10:53:04 -0800
Sender: [email-masked]

On Nov 15, 2007, at 9:27 AM, [email-masked] wrote:

> On Tue, 13 Nov 2007 18:32:49 PST, Douglas Otis said:
>
>> Message content offers little assurance of its origination.  Where  
>> to reach for the message is more significant.  The TBR extension  
>> offers both an assured last hop IP address and a domain of  
>> origination.  Often the origination of content is more important  
>> than the results of a scanning process.  Receivers would be foolish  
>> to trust the results of content scanning alone.
>
> OK. Let me get this straight.  We get an inbound connection from a  
> possibly dodgy source address, like a probably compromised  
> cablemodem box, and we then take the address *they provide us* and  
> use that as some sort of "verification"?

Yes, and at the MTA's leisure, as accepting a reference does not  
formally create an obligation.

> What's wrong with this picture? (Hint - what are the chances that a  
> hypothetical connection from said abused cablemodem, if it uses TBR,  
> will point at either the *same* address, or yet another abused  
> cablemodem?  Consider the use of fast-flux DNS changers when you  
> ponder this - there's no reason why you won't find that tbr- 
> server.fast-flux-r-us.com *won't* point at a cablemodem.)

You are right about the problem created by fast-flux DNS changers,  
especially when coupled with the noise created by odious domain  
tasting.  Even DKIM can be abused in this manner.

This problem can be tracked, but results may require a bit more time  
than allowed to return Okay to a DATA command.  Over a brief period of  
time, such abusive sources can be flagged.  The database is perhaps  
too extensive to distribute for real-time results (largely due to  
domain tasting), however near-real-time results are practical.  To  
support this effort, the TBR Extension offers a low overhead means to  
defer a formal obligation to deliver, while also avoiding the exchange  
of undesired data.

-Doug

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
To: Hector Santos <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Date: Thu, 15 Nov 2007 10:02:41 -0800
Sender: [email-masked]

On Nov 15, 2007, at 12:50 AM, Hector Santos wrote:

>
> Dave Crocker wrote:
>
>> John,
>> Although only a near-term, tactical benefit, greylisting directly  
>> impacts mail from bad actors.  It's serious downside is that it  
>> also impacts first-time mail from good actors.
>
> Which can be minimized, to a near "unfelt" impact by simple fine  
> tunning of SMTP router/sender frequency tables, in particular the  
> 2nd attempt.
>
> RFC-2821 currently recommends/states:
>
> 4.5.4.1 Sending Strategy
>
>   ...
>
>   The sender MUST delay retrying a particular destination after one
>   attempt has failed.  In general, the retry interval SHOULD be at
>   least 30 minutes; however, more sophisticated and variable  
> strategies
>   will be beneficial when the SMTP client can determine the reason for
>   non-delivery.
>
> The insights were laid out for variable (time/retry) strategies,  
> making it possible for Greylisting to work with minimal to no  
> significant impact.

Bulk emailers are often anxious to complete their campaigns and are  
dubious about retry delays of 30 minutes.  The initial benefit of grey- 
listing was as a test for whether a transmitting MTA carried state  
since many bots did not.  This is changing where now many bots retry  
the same message.  The benefits from grey-listing are fading, while  
bulk emailers have become more aggressive in their retries.

> I was really skeptical of GL, but when it was implemented right with  
> proper fine tuning of SMTP to minimize impact, it is really one of  
> those Ronco's "Set it and Forget it" ideas. :-)

Perhaps "Set it and regret it" might be the ultimate outcome when bulk  
emailers become even more aggressive as a result.

> My recommendation for 2821bis is to include some insights about or  
> how "variable strategies" plays a bigger roles these days and in  
> fact, might be almost an "necessity" for improved modern operations  
> these days - thats dealing with the realities we have out there.   
> Even if I didn't want to incorporate GL into our own system, the  
> sending retry strategies had to be reconsidered due to increased  
> remote GL systems.

I agree.  The TBR Extension should satisfy the desires of bulk  
emailers, while also providing an absolutely essential mechanism  
needed to protect exhaustive content filtering, the Temp error.  The  
EHLO parameter and TBR Temp error offers a valuable alternative to "Go  
away and retry later."

The TBR extension can:

1) without burdening the receiver

  a- provide a valid identity of origination

  b- eliminate back-scatter

2) conserve limited content assessment resources

3) improve delivery integrity

4) eliminate bulk emailer's aggressive reties

5) protect valid email-address confidentiality

6) defer and enhance assessments of questionable messages

7) avoid the DATA phase for abusive sources

8) avoid unintended DDoS effects

While DKIM offers a good mechanism to minimize false positives in anti- 
phishing filters, as with any content filtering, resources used must  
be protected.  To be effective, anti-phishing must examine message  
content for deceptive appearance.  Deceptive appearances are not  
something the IETF can hope to eliminate with signatures or path  
registration alone.  An anti-malware and anti-phishing effort requires  
the availability of resources needed for detailed analysis.  The TBR  
Extension offers receivers the means to request transmitters assist in  
this vital effort.

-Doug

To: Douglas Otis <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: SMTP Transferred-By-Reference
From: [email-masked]
Date: Thu, 15 Nov 2007 12:27:40 -0500
Sender: [email-masked]

--==_Exmh_1195147660_3222P
Content-Type: text/plain; charset=us-ascii

On Tue, 13 Nov 2007 18:32:49 PST, Douglas Otis said:

> Message content offers little assurance of its origination.  Where to  
> reach for the message is more significant.  The TBR extension offers  
> both an assured last hop IP address and a domain of origination.   
> Often the origination of content is more important than the results  
> of a scanning process.  Receivers would be foolish to trust the  
> results of content scanning alone.

OK. Let me get this straight.  We get an inbound connection from a possibly
dodgy source address, like a probably compromised cablemodem box, and we
then take the address *they provide us* and use that as some sort of
"verification"?

What's wrong with this picture? (Hint - what are the chances that a hypothetical
connection from said abused cablemodem, if it uses TBR, will point at either
the *same* address, or yet another abused cablemodem?  Consider the use of
fast-flux DNS changers when you ponder this - there's no reason why you won't
find that tbr-server.fast-flux-r-us.com *won't* point at a cablemodem.)

--==_Exmh_1195147660_3222P
Content-Type: application/pgp-signature

--==_Exmh_1195147660_3222P--

Date: Thu, 15 Nov 2007 03:50:36 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Dave Crocker wrote:

> John,
> 
> Although only a near-term, tactical benefit, greylisting directly 
> impacts mail from bad actors.  It's serious downside is that it also 
> impacts first-time mail from good actors.

Which can be minimized, to a near "unfelt" impact by simple fine tunning 
of SMTP router/sender frequency tables, in particular the 2nd attempt.

RFC-2821 currently recommends/states:

4.5.4.1 Sending Strategy

    ...

    The sender MUST delay retrying a particular destination after one
    attempt has failed.  In general, the retry interval SHOULD be at
    least 30 minutes; however, more sophisticated and variable strategies
    will be beneficial when the SMTP client can determine the reason for
    non-delivery.

The insights were laid out for variable (time/retry) strategies, making 
it possible for Greylisting to work with minimal to no significant impact.

It is when you (speaking in general) are setting there waiting or 
expecting a message,  and it doesn't arrive is what raises the hair of 
people.  But most people just press the "send" button and while they 
might expect "instant" send off because they know how their own mail 
system works, they won't be surprise if there isn't instant reception 
because they can't presume to know how the remote end works.  The same 
concept applies in telecommunications flow controls designs.

I was really skeptical of GL, but when it was implemented right with 
proper fine tuning of SMTP to minimize impact, it is really one of those 
Ronco's "Set it and Forget it" ideas. :-)

My recommendation for 2821bis is to include some insights about or how 
"variable strategies" plays a bigger roles these days and in fact, might 
be almost an "necessity" for improved modern operations these days - 
thats dealing with the realities we have out there.  Even if I didn't 
want to incorporate GL into our own system, the sending retry strategies 
had to be reconsidered due to increased remote GL systems.

My 2 pennies.

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Cc: [email-masked], [email-masked]
From: Douglas Otis <[email-masked]>
To: John C Klensin <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Wed, 14 Nov 2007 10:22:01 -0800
Sender: [email-masked]

On Nov 14, 2007, at 3:47 AM, John C Klensin wrote:

> Dave,
>
> I agree, and dislike this proposal for this (primarily) and other  
> reasons, including my usual problem of not liking to have to be  
> online to read and reply to messages (I'm reading this thread and  
> writing this answer at circa 30K feet over the Atlantic).

A minor change to the last sentence in the last paragraph of Section  
3.2 should eliminate this concern.  This sentence was added by John.   
It made sense to specify how to employ this alternative, however it  
also raised this concern, and possibly deprived recipients of content  
filtering that really should be done by their provider. : )

3.2.  TBR Transaction
...

,---
|As an alternative, instead of retrieving the replacement message
|content, it MAY prepend any additional trace headers to a
|notification sent to the recipient containing the eXAM-URI itself,
|along with any other appropriate information.
'___

Can be changed to:

,~~~
:An alternative notification containing the eXAM-URI and not the
:retrieved message content SHOULD NOT be used.
'~~~

> But, in fairness to the proposal, the general idea has one  
> advantage.  If one is concerned about source / originator identity  
> and authentication, having to make a real-time direct connection  
> back to the sender's repository permits thinking about much stronger  
> methods than, e.g., header signatures.

Header signatures represent yet another burden placed upon receiver  
resources.   Receiver side processes are being inundated by a rising  
level of spam.  The TBR method offers a means to shift the  
authentication burden to the transmitter of the message, while also  
providing the receiver a means to schedule and prioritize use of their  
resources without incurring an obligation to issue problematic DSNs.

> On the other hand, if we were willing to say "if you can't get email  
> unless you have stable end-to-end connectivity to the sender", one  
> could presumably get equally strong identification and  
> authentication with an enhanced SASL method and no new SMTP  
> extension protocol bits.

Without the TBR extension, point-to-point security would burden every  
stage of delivery.

-Doug

Date: Wed, 14 Nov 2007 10:02:39 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John Leslie <[email-masked]>
CC: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John Leslie wrote:
> 
>    What would others like to see in a fix for graylisting?

I am not sure how much can be done.  As you know, the reason why it 
"appears" to work (or it does it job) is that its based on legacy 
considerations (standard expectations and behavior of smtp senders).

But I may have a few items to consider:

Maybe this is more for 2821bis, is to emphasize proper or retry methods, 
in particular GL compliant sender to reconsider how their 2nd retry is 
done.  That was one of the first things we had to change in our SMTP 
client. Basically providing a variable frequency table. Otherwise, that 
2nd attempt was delayed to the old once per hour default and we did 
experiences an increase in reports about the rejects and not seeing 
email until much later. We hardly see reports or hear of them since the 
new frequency table as provided.

Possible some guidelines on the response code and text - making it official.

Suggestions about considering auto-whitelisting with local outbound mail 
flagging outgoing addresses to help auto-whitelist an anticipated 
response from them.   Today, its normal practice during a support 
incident for us to ask a customer for his email address to send an email 
if we are planning to ask them to send us something.  A 'permission 
email' per se. That way they get auto-whitelisting.

I guess also some guidelines on the fine tuning of typical GL 
parameters, such as using a class-c subnet (1.2.3.0) masking as part of 
the triplet hash calculation.  As you know, this addresses the sender 
with outbound farms on the same subnet.   In addition, other factors 
such as a general rule of thumb for retaining records, etc, etc.

One other option to help reduce issues, is to only apply GL for incoming 
addresses in a group of local domains, a spec or all.  For example, for 
our support system we only apply it to mail coming into to our corp or 
support domains. Not all other locally hosted domains.

Note sure if this is direct SMTP material but may be useful as design 
guidelines to consider for future GL implementations to maybe help 
minimize impact.

Other than that, it just runs, and it does its job which an amazing 
constant rejection rate of 60% - which I guess is the near standard rate 
of bombardment by these braindead bulk spammers.

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Wed, 14 Nov 2007 05:16:43 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John C Klensin wrote:
> But, in fairness to the proposal, the general idea has one advantage.  If
> one is concerned about source / originator identity and authentication,
> having to make a real-time direct connection back to the sender's
> repository permits thinking about much stronger methods than, e.g., header
> signatures.

I'm pretty sure it permits nothing stronger at all.

So I need you to provide some detail.

I'll also note that a mechanism with this much cost -- especially at the
infastructure -- needs a proof-positive guarantee that it will provide
benefits, and not just that it makes benefits possible.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 14 Nov 2007 06:47:52 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

--On Tuesday, 13 November, 2007 10:03 -0800 Dave Crocker
<[email-masked]> wrote:

>> How is that any different than sending a 5xx in response to
>> RCPT TO:, which is something we can (and should) do now?
> 
> 
> I'm feeling kinda dumb for having missed the ultimate flaw in
> the proposal.
> 
> The goal of the proposal is to permit deferred filtering
> analysis, or at least reputation analysis.
> 
> As someone else noted, most of this work requires the message
> header and/or content.  That requires message transfer.
> 
> The information that is transferred during the retained SMTP
> exchange is minimally helpful, except for previous-hop IP
> Address.  Everything else requires access to the actual
> message.  This means reaching across the net to get the
> message for inspection.
> 
> And this is better than transferring the message during a
> regular SMTP session how?
> 
> We don't save cross-net transfers.  We add transaction
> overhead and delay.
> 
> The issue of hand-off responsibility is changed, but I have
> not heard that asserted as a problem amidst anti-abuse efforts.

Dave,

I agree, and dislike this proposal for this (primarily) and
other reasons, including my usual problem of not liking to have
to be online to read and reply to messages (I'm reading this
thread and writing this answer at circa 30K feet over the
Atlantic).

But, in fairness to the proposal, the general idea has one
advantage.  If one is concerned about source / originator
identity and authentication, having to make a real-time direct
connection back to the sender's repository permits thinking
about much stronger methods than, e.g., header signatures.

On the other hand, if we were willing to say "if you can't get
email unless you have stable end-to-end connectivity to the
sender", one could presumably get equally strong identification
and authentication with an enhanced SASL method and no new SMTP
extension protocol bits.

     john

Date: Wed, 14 Nov 2007 09:27:47 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

[email-masked] writes:
> One of the unintended advantages of TBR over greylisting is that it 
> can be used for a message recall feature. :-)

So can greylisting if you can reach into the queue. Which noone can 
today, but if you're going to implement something new you might as well 
implement a reach-into-the-queue feature as TBR.

Arnt

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Tue, 13 Nov 2007 18:32:49 -0800
To: [email-masked]
Sender: [email-masked]

On Nov 13, 2007, at 10:03 AM, Dave Crocker wrote:
> I'm feeling kinda dumb for having missed the ultimate flaw in the  
> proposal.
>
> The goal of the proposal is to permit deferred filtering analysis,  
> or at least reputation analysis.

While being able to defer an assessment of reputation is a potential  
benefit, it is not the goal of this proposal.  Millions of IP sources  
are identified as being untrustworthy when their ISP classifies these  
addresses as belonging to a residential access point.  While these  
lists account for a sizeable amount of the spam being blocked (these  
systems are often compromised), an even greater amount of spam is  
blocked based upon dynamic reputations (often of their provider's  
servers).  The game has changed.  Handling a mixture of good and bad  
email must employ temporary 451 errors. : (

For ISP that host customers sending out a spam campaign, their  
servers may receive a temp error at that point in time.   
Unfortunately, this may cause some desired email to be delayed.  Once  
spamming from that server ceases, normal acceptance is restored.   
This dynamic reputation tactic blocks often more than twice the  
amount of spam stopped by static lists.  Graduated dynamic lists are  
available to suit the needs of different receivers.

Blocking an influx of spam using the temp error tactic frees up  
resources needed to process other (cleaner) sources of email.   
Simultaneously offering the TBR extension within the 451 4.7.8/9/10  
temp error would provide the transmitter a means to immediately  
complete their transaction by reverting to the TBR mode.  (Fewer  
transactions and less delay.)

The percentage of spam is increasing, and content filtering is being  
defeated on an ongoing basis.  Reacting to new methods of obfuscation  
gives bad actors a significant resource advantage.  The temp error  
tactic ensures this finite resource is not inundated to the point of  
collapse.  The TBR mode would permit receivers a means to better  
prioritize their backlog of messages waiting to be scanned.  The TBR  
mode would also provide transmitters a means to complete the SMTP  
portion of their exchange.  The TBR extension would be a win/win for  
both the transmitter and receiver.

> As someone else noted, most of this work requires the message  
> header and/or content.  That requires message transfer.

The TBR extension does not interfere with content checking.  The TBR  
extension allows receivers to decide when they are ready to check  
content, and what has priority.

> The information that is transferred during the retained SMTP  
> exchange is minimally helpful, except for previous-hop IP Address.   
> Everything else requires access to the actual message.  This means  
> reaching across the net to get the message for inspection.

Message content offers little assurance of its origination.  Where to  
reach for the message is more significant.  The TBR extension offers  
both an assured last hop IP address and a domain of origination.   
Often the origination of content is more important than the results  
of a scanning process.  Receivers would be foolish to trust the  
results of content scanning alone.

> And this is better than transferring the message during a regular  
> SMTP session how?

A message transferred in the normal manner fails to validate a point  
of origination.  DKIM might, but this signature scheme is vulnerable  
and requires additional receiver resources.  With many entities out- 
sourcing email services, messages will typically traverse different  
acceptance policies.  In the case of the TBR extension, content based  
policies can be consolidated at the last stages.  Without reducing  
email's delivery integrity, the TBR extension also curtails DSNs when  
the message is never fetched due to the origination being considered  
abusive or the recipient being invalid.

> We don't save cross-net transfers.  We add transaction overhead and  
> delay.

The cross-net transfers are not increased when applying point of  
origination reputation.  Due to the prevalence of spam, point of  
origination reputation (POOR) can significantly reduce the overall  
network traffic.  When blocked with 451 4.7.8/9/10, the TBR  
extensions provides a means to complete the transaction.  When the  
receiver decides to obtain content is when it becomes obligated to  
ensure the message is delivered or the MAIL FROM receives a DSN.  The  
TBR extension also ensures the MAIL FROM is valid without demanding  
any additional analysis by the receiver.

> The issue of hand-off responsibility is changed, but I have not  
> heard that asserted as a problem amidst anti-abuse efforts.

This is a problem when out-sourcing email services and often why  
recipient lists must be continuously updated.  This issue is also  
likely a reason for the tens of millions of servers now issuing DSNs  
containing the original spam and malware. : (

-Doug

Cc: [email-masked], [email-masked]
From: Douglas Otis <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Tue, 13 Nov 2007 15:44:13 -0800
To: [email-masked]
Sender: [email-masked]

On Nov 12, 2007, at 12:59 PM, [email-masked] wrote:

> On Mon, 12 Nov 2007 10:48:00 PST, Douglas Otis said:
>
>>> 2. At best, this reduces total bytes over the but the requirement  
>>> for a notification message does not reduce the number of network  
>>> 'transactions' -- in fact it increases them by 100% or more.
>>
>> This calculation is wrong.  You are assuming a substantial  
>> percentage of messages will be retrieved, yet the typical levels  
>> of spam would suggest otherwise.
>
> Note that a fairly large percentage of spam is flagged as such by  
> scanning the *body* of the message - in other words, you don't know  
> if it's spam until you fetch the data *anyhow*.

Content evaluation has required increasingly greater processing  
resources.  Analysing content now deals with structures displaying  
images demanding the resources needed to defeat Captchas.  Content  
screening needs to normalize nearly every popular attachment, such as  
PDF, DOC, WMV, etc.  Difficult cases will not reveal identifiers  
using fingerprinting algorithms either.

The TBR extension used in conjunction with coincident source  
reputation remains far more scalable than content analysis alone.   
The TBR extension offers a valid origination identifier without  
demanding added receiver resources.  Once a message has been fetched,  
the DSN path will have been assured valid as well, also without  
demanding additional receiver resources.  The TBR extension ensures  
no back-scatter while providing very high delivery integrity.   
Without TBR being used, silently dropping DSNs reduces email's  
delivery integrity.

The reputation of a message's true origination can reveal recent  
behaviour not seen only examining message content of often  
questionable validity.  Signature checking or path registration  
processing also represent DDoS vulnerabilities and demand handling  
exceptions.  The TBR extension safely provides a valid origination of  
a message, while also eliminating corner cases created by broken  
paths or signatures.

> What percent of spam are you blocking based *only* on the SMTP MAIL  
> FROM/RCPT TO data?

This question overlooks MAIL FROM is not assured as a basis for a  
decision.  Without the TBR extension, little within the message  
should be assumed valid without additional receiver analysis (which  
may offer dubious results).  In comparison, an invalid TBR is self  
correcting without demanding additional receiver resources.

While email can be rejected based upon RCPT TO being invalid, this  
also enables address harvesting.  The TBR extension can also thwart  
this tactic as well.  As you can see, this question overlooks the  
value of the TBR extension.

> That's the percentage you can avoid fetching the body - and if  
> you're smart you *currently* 4xx/5xx the transaction before you get  
> to DATA in that case, so you're not saving any network bandwidth by  
> not fetching it over a second connection.

Being able to offer temp or perm error status prior to the DATA phase  
is ideal.  The TBR extension avoids the DATA phase while also  
eliminating a need to formally assure delivery.  The TBR mode  
replaces the DATA phase with a solid identifier of a message's  
origination.  Questionable transmitters which do not offer the TBR  
extension will need to accept the resulting delay.  This delay would  
offer an incentive to implement TBR.  Better delivery integrity would  
be another incentive.

-Doug

To: SM <[email-masked]>
Cc: "David F. Skoll" <[email-masked]>, [email-masked]
Subject: Re: Fixing graylisting [was TBR]
From: [email-masked]
Date: Tue, 13 Nov 2007 17:30:53 -0500
Sender: [email-masked]

--==_Exmh_1194993053_3241P
Content-Type: text/plain; charset=us-ascii

On Tue, 13 Nov 2007 13:09:34 PST, SM said:

> One of the unintended advantages of TBR over greylisting is that it 
> can be used for a message recall feature. :-)

Not really, unless you *mandate* that a "cooling-off" period of (say) 5 minutes
be enforced before anybody can fetch the message.  Otherwise you encounter
a race condition between you finding 'recall message' in the menu, and another
site fetching the message.

(Yes, I see the smiley - I'm just filling in the blanks for less-clued people
who may someday be reading the list archives....)

--==_Exmh_1194993053_3241P
Content-Type: application/pgp-signature

--==_Exmh_1194993053_3241P--

Date: Tue, 13 Nov 2007 13:09:34 -0800
To: "David F. Skoll" <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

At 08:57 13-11-2007, David F. Skoll wrote:
>I don't see what's broken about a good greylisting implementation.

The assumptions about MAIL FROM and the IP address of the remote connection.

>The enormous advantage of greylisting over TBR is that it doesn't
>require an ESMTP extension, so it doesn't require a significant
>percentage of the Internet to adopt it for it to be useful.

One of the unintended advantages of TBR over greylisting is that it 
can be used for a message recall feature. :-)

Regards,
-sm 

From: Russ Allbery <[email-masked]>
To: John Leslie <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Fixing graylisting
Organization: The Eyrie
Date: Tue, 13 Nov 2007 13:15:08 -0800
Sender: [email-masked]

John Leslie <[email-masked]> writes:
> Dave Crocker <[email-masked]> wrote:

>> Its serious downside is that it also impacts first-time mail from good
>> actors.

>    First-time email from originators who haven't yet developed a
> reputation, alas, _deserves_ to be delayed.

Redefining the problem to fit your solution is certainly one way to fix
false positives.  It's not a bug, it's a feature!

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 13 Nov 2007 15:53:33 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: TBR again (was Re: Fixing graylisting [was TBR])
Sender: [email-masked]

John Leslie wrote:

>    I'd say it has a far better idea than current graylisting does.
> The originator has logs to see whether the URI has been accessed: I
> see no reason the originator would take down an unreferenced URI in
> less than a few days.

But again, you're not responding to one of my original criticisms:  One
of the banes of running a large-scale e-mail system is clogged queues.
This proposal essentially _mandates_ clogged queues because you _have_
to keep the message around for "a few days".

Furthermore, it's trivial for spammers to maintain a URL forever
because they can generate the message on the fly, whereas it's
extremely burdensome for high-volume legitimate senders to maintain
the URLs.

As far as I can see, this extension will be ignored by spammers and
will punish those legitimate senders who decide to use it, so I don't
really see the point.

Regards,

David.

Date: Tue, 13 Nov 2007 12:40:48 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Leslie <[email-masked]>
CC: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John Leslie wrote:
>> The information that is transferred during the retained SMTP exchange
>> is minimally helpful, except for previous-hop IP Address.
> 
>    Please don't forget the URI and RFC2821MailFrom. These tell you the

MailFrom does not always specify originator.  Let's not repeat the SPF 
conceptual error.

>> Everything else requires access to the actual message. This means
>> reaching across the net to get the message for inspection.
> 
>    Yes, there is a TCP setup time involved here. How responsive that
> server is is presumably under the control of the originator.

and extra cost on the receive side.

>> The issue of hand-off responsibility is changed, but I have not heard
>> that asserted as a problem amidst anti-abuse efforts.
> 
>    You must be following different mailing lists than I do. I have seen
> rather a bit of traffic about how to hold a SMTP connection open for
> a longer time.

Citations to this constituency, please?

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 13 Nov 2007 15:09:53 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Dave Crocker <[email-masked]> wrote:
> 
> The goal of the proposal is to permit deferred filtering analysis, or
> at least reputation analysis.

   That is one of the goals.

> As someone else noted, most of this work requires the message header
> and/or content. That requires message transfer.

   Correct. I recognize that many folks will want to filter based on
content, even for messages from trustworthy sites.

> The information that is transferred during the retained SMTP exchange
> is minimally helpful, except for previous-hop IP Address.

   Please don't forget the URI and RFC2821MailFrom. These tell you the
originator, and even before you do reputation checks on them, you know
the originator can advertise DNS records and maintains a web-like
service and receives email. This is quite a bit beyond what you know
from the previous-hop IP address.

> Everything else requires access to the actual message. This means
> reaching across the net to get the message for inspection.

   Yes, there is a TCP setup time involved here. How responsive that
server is is presumably under the control of the originator.

> The issue of hand-off responsibility is changed, but I have not heard
> that asserted as a problem amidst anti-abuse efforts.

   You must be following different mailing lists than I do. I have seen
rather a bit of traffic about how to hold a SMTP connection open for
a longer time.

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 12:01:23 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Leslie <[email-masked]>
CC: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John Leslie wrote:
> Dave Crocker <[email-masked]> wrote:
>> John Leslie wrote:
>>
>>> The "fix" Doug and I put into TBR is to extend the time to formal
>>> handoff, by any amount the receiving mail system may choose, which
>>> accomplishes much of what keeping the TCP connection open would -- at
>>> a far smaller cost (the queue of URIs could be written to disk, for
>>> one example).
>> Although only a near-term, tactical benefit, greylisting directly
>> impacts mail from bad actors.
> 
>    I wouldn't describe it as "only near-term". I don't notice folks
> abandoning graylisting in droves. When graylisting is used to buy time
> to gather enough reputation information, its long-term benefit can be
> substantial.

Try ignoring my reference to 'near-term' here.

My intended focus was on the direct impact on bad actors. That's what needs 
the attention.

>> Its serious downside is that it also impacts first-time mail from good
>> actors.
> 
>    First-time email from originators who haven't yet developed a
> reputation, alas, _deserves_ to be delayed.

Nicely cavalier.

>> In contrast, your scheme will only be used for mail from good actors.
> 
>    That's not as certain as you think. Many spammers seem to choose
> their customers for stupidity, and such customers might well believe

In the presence of meaningful reputation services, please provide some 
empirical support for the claim that bad actors use those services.

> that the spammer has delivered on their promise merely by delivering
> the TBR. In any case, we cannot _assume_ only good actors will use
> TBR.

Turns out we can, for the reasons others have cited.

>> This is exactly the mail that does *not* need to be held up.
...
>    The mail which "does not need to be held up" is mail from well-known
> and trusted senders. Anything else may well be abusive in someone's
> eyes. Hopefully, graylisting won't be applied to the first kind.

Did you by any chance notice that you did not refute the concern I raised?

>> So the mechanism increases delay
> 
>    Measurably, probably; perceptibly, probably not.

Again, absent empirical data, that's a dangerous assumption.

>> and at least doubles the transaction load for mail from good actors,
> 
>    I continue to ask you for the basis of this calculation. I'm _not_
> going to guess what you mean by this, Dave!

I explained it in a private note.  Now publicly:

SMTP is a single session that delivers a message.  That's a "transaction". 
Within SMTP are protocol exchanges.  They also constitute transactions, but at 
a different level.

The proposed scheme requires a minimum of two sessions, thereby at least 
doubling the higher-level transaction cost.  Since the proposal seems to 
retain the same number of within-SMTP protocol exchanges (swapping DATA for 
the URL citation), there's no saving at the micro-level, but there is the 
addition of whatever protocol exchanges are needed for the follow-on fetching. 
I gather that is, at the least, and HTTP connection and FETCH.

Protocol exchanges and sessions create their own costs, in packet switching 
worlds, separate from raw bandwidth requirements.  They draw from a different 
performance budget and often are the higher cost.  At a minimum, think of a 
satellite channel.  Might have very high bandwidth, but delay always sucks. 
Transaction overhead is like the cost of that satellite delay.  Separate from 
bandwidth.

ESMTP added options in a way that careful avoided adding even one protocol 
exchange round-trip.  That's how important this issue was considered amongst 
the email protocol community.

By contrast, the current proposal adds 100% or more round-trips with no 
certainty of massive benefit.  And I suggest that anything this expense pretty 
much needs a certified guarantee of massive benefit.

>    With the spam load generally acknowledged to be 90% or more, it
> would be a major victory to _have_ a mechanism used only by good
> actors.

If you are looking to created a trusted infrastructure, there are more 
efficient ways to do it.

For example, define SMTP over a new port, with required authentication by 
client SMTP sites, and whatever other trust-related capabilities are deemed 
appropriate.  Servers that support the new port publish an MX-like record.

That that approach that will require some sort of client certification is 
pretty much the same problem we are already tackling, albeit with a different 
twist.

>> Where is the benefit, here?
> 
>    There isn't only one benefit. Greylisting without encouraging a
> doubling of SMTP traffic would be a benefit.

Greylisting doubles SMTP traffic for first-timers.

Your scheme at least doubles total transaction traffic for all good-guys, forever.

> Smoothing network load
> by scheduling TBR retrievals would be another.

Traffic management by imposing arbitrary overhead is not generally held in 
high regard.

> There's the benefit
> of having a mechanism available to those who must work from Cable
> Internet addresses which are increasingly blacklisted. And please
> don't forget the benefit of transferring an immutable message text.

This suggests that you are trying to fix IP Address-based trust, just as the 
industry is attempting to move to Domain Name-based trust.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 13 Nov 2007 14:57:40 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: John Leslie <[email-masked]>, [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

[email-masked] <[email-masked]> wrote:
> On Tue, 13 Nov 2007 11:42:27 EST, John Leslie said:
> 
>> ... the -00 spec does allow for the case where you decide -- before
>> fetching any part of the message -- that the originator is not worth
>> trusting, so you simply discard the URI silently.
> 
> How is that any different than sending a 5xx in response to RCPT TO:,
> which is something we can (and should) do now?

   Mainly in that you can do it minutes after receiving the TBR command
without needing to keep the TCP connection open.

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 14:54:21 -0500
From: John Leslie <[email-masked]>
To: "David F. Skoll" <[email-masked]>
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

David F. Skoll <[email-masked]> wrote:
> John Leslie wrote:
> 
>> The "fix" Doug and I put into TBR is to extend the time to formal
>> handoff, by any amount the receiving mail system may choose, which
>> accomplishes much of what keeping the TCP connection open would -- at
>> a far smaller cost (the queue of URIs could be written to disk, for
>> one example).
> 
> Except the receiving mail system cannot really choose "any amount",
> because it has no idea for how long the URI will remain valid.

   I'd say it has a far better idea than current graylisting does.
The originator has logs to see whether the URI has been accessed: I
see no reason the originator would take down an unreferenced URI in
less than a few days.

> It can (I suppose) confidently assume it will remain valid for a few
> hours. Maybe even a day.

   I would restate that as "It can confidently assume anything taken
down before being accessed after less than one day is very likely to
have been abusive."

> Any longer is risky.

   I'm hard-pressed to think of a case where you'd need more than 24
hours get "good enough" reputation information.

> So unless you have a formal requirement for how long the URI must
> remain valid, you're still at the mercy of the sender's queueing
> policy (as with greylisting.)

   I suppose we could add a formal requirement: it doesn't seem
necessary to me. I expect originators to keep things available for
at least a week or two in the normal case -- taking down URIs
where the logs show successful access, and possibly expiring some
URIs according to a schedule the user requests.

>> What would others like to see in a fix for graylisting?
> 
> A good greylisting implementation will turn off greylisting once a
> host has been known to retry, and keep it off for that host for some
> fairly long time (like a month or so.)  This greatly reduces the
> annoying impact of greylisting on legitimate MTAs without much
> reducing its effectiveness.

   That's a good implementation practice, though IMHO the period to
keep it off needs to be configurable.

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 14:39:36 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

Dave Crocker <[email-masked]> wrote:
> John Leslie wrote:
> 
>> The "fix" Doug and I put into TBR is to extend the time to formal
>> handoff, by any amount the receiving mail system may choose, which
>> accomplishes much of what keeping the TCP connection open would -- at
>> a far smaller cost (the queue of URIs could be written to disk, for
>> one example).
> 
> Although only a near-term, tactical benefit, greylisting directly
> impacts mail from bad actors.

   I wouldn't describe it as "only near-term". I don't notice folks
abandoning graylisting in droves. When graylisting is used to buy time
to gather enough reputation information, its long-term benefit can be
substantial.

> Its serious downside is that it also impacts first-time mail from good
> actors.

   First-time email from originators who haven't yet developed a
reputation, alas, _deserves_ to be delayed.

> In contrast, your scheme will only be used for mail from good actors.

   That's not as certain as you think. Many spammers seem to choose
their customers for stupidity, and such customers might well believe
that the spammer has delivered on their promise merely by delivering
the TBR. In any case, we cannot _assume_ only good actors will use
TBR.

> This is exactly the mail that does *not* need to be held up.

   The world is not merely stranger than you imagine: it is stranger
than you _can_ imagine.

   The mail which "does not need to be held up" is mail from well-known
and trusted senders. Anything else may well be abusive in someone's
eyes. Hopefully, graylisting won't be applied to the first kind.

> So the mechanism increases delay

   Measurably, probably; perceptibly, probably not.

> and at least doubles the transaction load for mail from good actors,

   I continue to ask you for the basis of this calculation. I'm _not_
going to guess what you mean by this, Dave!

> while having no impact on mail from bad actors.

   With the spam load generally acknowledged to be 90% or more, it
would be a major victory to _have_ a mechanism used only by good
actors. We could double our resources devoted to the 10%, and the
overall cost would barely twitch.

   We don't _need_ any impact on mail from bad actors beyond setting
our graylisting parameters to graylist harder when our network load
is higher -- thus giving priority to what we classify as "good actors".

> Where is the benefit, here?

   There isn't only one benefit. Greylisting without encouraging a
doubling of SMTP traffic would be a benefit. Smoothing network load
by scheduling TBR retrievals would be another. There's the benefit
of having a mechanism available to those who must work from Cable
Internet addresses which are increasingly blacklisted. And please
don't forget the benefit of transferring an immutable message text.

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 18:03:44 +0000
From: Tony Finch <[email-masked]>
To: John Leslie <[email-masked]>
cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

On Tue, 13 Nov 2007, John Leslie wrote:
>
> Doug expects this to be frequent, if spammers actually use TBR.

I thought Doug had more sense than to expect spammers to adopt a protocol
that makes their job harder!

I echo Dave's comments.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
WEST FORTIES CROMARTY FORTH TYNE WEST DOGGER: NORTH OR NORTHWEST 6 OR 7
BECOMING VARIABLE THEN SOUTHWEST 3 OR 4. MODERATE OR ROUGH. SHOWERS. MODERATE
OR GOOD.

Date: Tue, 13 Nov 2007 10:03:00 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

[email-masked] wrote:
> On Tue, 13 Nov 2007 11:42:27 EST, John Leslie said:
> 
>>    You're essentially correct, but the -00 spec does allow for the case
>> where you decide -- before fetching any part of the message -- that the
>> originator is not worth trusting, so you simply discard the URI silently.
> 
> How is that any different than sending a 5xx in response to RCPT TO:,
> which is something we can (and should) do now?

I'm feeling kinda dumb for having missed the ultimate flaw in the proposal.

The goal of the proposal is to permit deferred filtering analysis, or at least 
reputation analysis.

As someone else noted, most of this work requires the message header and/or 
content.  That requires message transfer.

The information that is transferred during the retained SMTP exchange is 
minimally helpful, except for previous-hop IP Address.  Everything else 
requires access to the actual message.  This means reaching across the net to 
get the message for inspection.

And this is better than transferring the message during a regular SMTP session 
how?

We don't save cross-net transfers.  We add transaction overhead and delay.

The issue of hand-off responsibility is changed, but I have not heard that 
asserted as a problem amidst anti-abuse efforts.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

To: John Leslie <[email-masked]>
Cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
From: [email-masked]
Date: Tue, 13 Nov 2007 12:35:23 -0500
Sender: [email-masked]

--==_Exmh_1194975323_3241P
Content-Type: text/plain; charset=us-ascii

On Tue, 13 Nov 2007 11:42:27 EST, John Leslie said:

>    You're essentially correct, but the -00 spec does allow for the case
> where you decide -- before fetching any part of the message -- that the
> originator is not worth trusting, so you simply discard the URI silently.

How is that any different than sending a 5xx in response to RCPT TO:,
which is something we can (and should) do now?

--==_Exmh_1194975323_3241P
Content-Type: application/pgp-signature

--==_Exmh_1194975323_3241P--

Date: Tue, 13 Nov 2007 11:57:21 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John Leslie wrote:

>    I quite agree that graylisting needs fixing (and Doug and I did try
> to sneak some fixing into TBR).

I don't see what's broken about a good greylisting implementation.

[...]

>    I look upon graylisting as a temporary measure to gain time for
> better information about the sender -- not just whether they keep state
> and run the queues.

Yes, that's what it is.

>    The "fix" Doug and I put into TBR is to extend the time to formal
> handoff, by any amount the receiving mail system may choose, which
> accomplishes much of what keeping the TCP connection open would -- at
> a far smaller cost (the queue of URIs could be written to disk, for
> one example).

Except the receiving mail system cannot really choose "any amount", because
it has no idea for how long the URI will remain valid.  It can (I suppose)
confidently assume it will remain valid for a few hours.  Maybe even a day.
Any longer is risky.

So unless you have a formal requirement for how long the URI must remain
valid, you're still at the mercy of the sender's queueing policy (as
with greylisting.)

>    What would others like to see in a fix for graylisting?

A good greylisting implementation will turn off greylisting once a
host has been known to retry, and keep it off for that host for some
fairly long time (like a month or so.)  This greatly reduces the
annoying impact of greylisting on legitimate MTAs without much
reducing its effectiveness.

The enormous advantage of greylisting over TBR is that it doesn't
require an ESMTP extension, so it doesn't require a significant
percentage of the Internet to adopt it for it to be useful.

Regards,

David.

Date: Tue, 13 Nov 2007 11:42:27 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Alex van den Bogaerdt <[email-masked]> wrote:
> On Tue, Nov 13, 2007 at 09:53:24AM +0000, Tony Finch wrote:
> 
>> However, I believe that in doubtful cases it's better to apply AI to the
>> complete message data than to attempt to analyse some abbreviated
>> notification. In most cases MTAs have enough capacity to do this: at the
>> moment (according to my stats) doubtful cases are about 30% of the email
>> that gets past blacklists.
> 
> The big difference seems to be what happens after you decide a
> message to be spam.
> 
> Present day: you delete it without notification or you send it "back",
> generating backscatter.
> 
> With TBR: you send a notification to the sender's domain. That server
> should NOT forward the bounce to an innocent victim. Instead, the domain
> could even count such bounces and be on guard about this customer.
> 
> Am I seeing this wrong?

   You're essentially correct, but the -00 spec does allow for the case
where you decide -- before fetching any part of the message -- that the
originator is not worth trusting, so you simply discard the URI silently.
Doug expects this to be frequent, if spammers actually use TBR. YMMV.

   This is a good opportunity to point out this _is_ a -00 spec. My
standards for a -00 spec may be higher than some (as Dave Crocker will
attest), but I'm sure it has some down-and-out errors, as well as areas
which could be improved by a WG-like process. Doug and I are certainly
open to that.

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 08:31:12 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Leslie <[email-masked]>
CC: [email-masked]
Subject: Re: Fixing graylisting [was TBR]
Sender: [email-masked]

John Leslie wrote:
>    We could in principle accomplish that by keeping the SMTP connection
> open for however long that takes; but this feels wrong to me: it's
> probably cheaper for the spammer to keep a botnet connection open than
> it is for me to keep TCP state for a million spams.
> 
>    The "fix" Doug and I put into TBR is to extend the time to formal
> handoff, by any amount the receiving mail system may choose, which
> accomplishes much of what keeping the TCP connection open would -- at
> a far smaller cost (the queue of URIs could be written to disk, for
> one example).

John,

Although only a near-term, tactical benefit, greylisting directly impacts mail 
from bad actors.  It's serious downside is that it also impacts first-time 
mail from good actors.

In contrast, your scheme will only be used for mail from good actors.  This is 
exactly the mail that does *not* need to be held up. So the mechanism 
increases delay and at least doubles the transaction load for mail from good 
actors, while having no impact on mail from bad actors.

Where is the benefit, here?

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 13 Nov 2007 11:02:07 -0500
From: John Leslie <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: [email-masked]
Subject: Fixing graylisting [was TBR]
Sender: [email-masked]

Tony Finch <[email-masked]> wrote:
> 
> I agree that it is a worthwhile goal to reduce the interop problems of
> greylisting, and I agree that delaying email from unknown sources might
> give blacklists time to make a decision about the source.
> 
> I think it would be more useful for you to try to fix greylisting than to
> redesign IM2000 yet again.

   An interesting challenge!

   I quite agree that graylisting needs fixing (and Doug and I did try
to sneak some fixing into TBR).

   Graylisting, IMHO, started with a simple observation that many botnets
would only try once; thus giving 50% temporary errors (blindly) would
reduce botnet spam by 50%. Obviously, spam increased to fill the vacuum,
proving that a 50% solution is effectively no solution at all.

   Graylisting progressed to a realization that botnet spammers didn't
keep state, while genuine MTAs did. Thus graylisters now try to notice
which incoming emails are the result of running the queue of previous
tries which you gave temporory errors to. This is much better, but it
is difficult because the SMTP protocol does nothing to help.

   I look upon graylisting as a temporary measure to gain time for
better information about the sender -- not just whether they keep state
and run the queues.

   We could in principle accomplish that by keeping the SMTP connection
open for however long that takes; but this feels wrong to me: it's
probably cheaper for the spammer to keep a botnet connection open than
it is for me to keep TCP state for a million spams.

   The "fix" Doug and I put into TBR is to extend the time to formal
handoff, by any amount the receiving mail system may choose, which
accomplishes much of what keeping the TCP connection open would -- at
a far smaller cost (the queue of URIs could be written to disk, for
one example).

   A part of the problem which remains unfixed is how long the sender
should expect to wait. Doug and I despaired of fixing this.

   We did add an error message saying, in essence, "You are being
graylisted." We're of the opinion that any further information should
be out-of-band (or at least not part of protocol).

   What would others like to see in a fix for graylisting?

--
John Leslie <[email-masked]>

Date: Tue, 13 Nov 2007 10:24:15 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Alex van den Bogaerdt wrote:

[David Skoll]

>> Or you fail it with a 5xx SMTP failure code.

[Scenarios in which a 5xx can create backscatter]

Yes, I'm perfectly aware that 5xx's can cause backscatter.
But that's not my problem.  It's up to all MTAs to avoid backscatter
and those that cause it will (rightly) get blacklisted until they
mend their ways.

A 5xx is the best way to reject unwanted mail because it doesn't
silently discard (which decreases confidence in e-mail) and it doesn't
compose an NDN (which causes backscatter).  I realize that a 5xx is
not always possible and that you sometimes have to silently discard.

>> Rewriting the envelope sender to detect legitimate bounces vs. backscatter
>> can be done without an ESMTP extension.

> Yeah, except that you seem to be looking at this from D's perspective.

If "D" is me, then yes: I'm looking at it from my perspective because
my mail server is the only one I can control!

> I'm not. The key factor here is that the return path has to be at the
> originator's ISP server. Or so I think.

If this is an issue, it can be solved (or ameliorated) with SPF.

Regards,

David.

Date: Tue, 13 Nov 2007 15:38:51 +0100
From: Alex van den Bogaerdt <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Tue, Nov 13, 2007 at 07:12:38AM -0500, David F. Skoll wrote:
> 
> Alex van den Bogaerdt wrote:
> 
> > The big difference seems to be what happens after you decide a
> > message to be spam.
> 
> > Present day: you delete it without notification or you send it "back",
> > generating backscatter.
> 
> Or you fail it with a 5xx SMTP failure code.

A: the message submitter (a zombie)
B: A's ISP
C: you/your ISP
D: me

Scenario 1a:
A sends to B, B accepts.  B tries sending to C, C rejects.
B sends a NDN to B.

Scenario 1b:
A sends to B, B accepts.  B tries sending to C, C rejects.
B forwards the spam to /dev/null

Scenario 2a:
A sends to B, B accepts.  B tries sending to C, C accepts.
C sends a NDN to B.

Scenario 2b:
A sends to B, B accepts.  B tries sending to C, C accepts.
C forwards the spam to /dev/null

Things get more complicated when more relays are involved.

With TBR:

A sends to B, B accepts.  B informs C, C does nothing.
B kills the message eventually and does not send a NDN to D.

Things stay the same when more relays are involved.

> > With TBR: you send a notification to the sender's domain. That server
> > should NOT forward the bounce to an innocent victim. Instead, the domain
> > could even count such bounces and be on guard about this customer.
> 
> Rewriting the envelope sender to detect legitimate bounces vs. backscatter
> can be done without an ESMTP extension.

Yeah, except that you seem to be looking at this from D's perspective.
I'm not. The key factor here is that the return path has to be at the
originator's ISP server. Or so I think.

Date: Tue, 13 Nov 2007 07:12:38 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Alex van den Bogaerdt wrote:

> The big difference seems to be what happens after you decide a
> message to be spam.

> Present day: you delete it without notification or you send it "back",
> generating backscatter.

Or you fail it with a 5xx SMTP failure code.

> With TBR: you send a notification to the sender's domain. That server
> should NOT forward the bounce to an innocent victim. Instead, the domain
> could even count such bounces and be on guard about this customer.

Rewriting the envelope sender to detect legitimate bounces vs. backscatter
can be done without an ESMTP extension.

Regards,

David.

Date: Tue, 13 Nov 2007 12:31:22 +0100
From: Alex van den Bogaerdt <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Tue, Nov 13, 2007 at 09:53:24AM +0000, Tony Finch wrote:

> However, I believe that in doubtful cases it's better to apply AI to the
> complete message data than to attempt to analyse some abbreviated
> notification. In most cases MTAs have enough capacity to do this: at the
> moment (according to my stats) doubtful cases are about 30% of the email
> that gets past blacklists.

The big difference seems to be what happens after you decide a
message to be spam.

Present day: you delete it without notification or you send it "back",
generating backscatter.

With TBR: you send a notification to the sender's domain. That server
should NOT forward the bounce to an innocent victim. Instead, the domain
could even count such bounces and be on guard about this customer.

Am I seeing this wrong?

Date: Tue, 13 Nov 2007 09:53:24 +0000
From: Tony Finch <[email-masked]>
To: Douglas Otis <[email-masked]>
cc: John Leslie <[email-masked]>, [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

On Mon, 12 Nov 2007, Douglas Otis wrote:
>
> When the SMTP client's reputation is unknown, an SMTP server is unlikely
> to Perm Error at the EHLO.  Temp Errors at this point might be used to
> discern whether the SMTP client keeps state, as with greylisting.  Temp
> errors might also be used to delay acceptance when a spam campaign is
> detected as being active at an SMTP client handling messages from both
> good and bad actors.  TBR provides a safe alternative to greylisting and
> temporary holds.

I agree that it is a worthwhile goal to reduce the interop problems of
greylisting, and I agree that delaying email from unknown sources might
give blacklists time to make a decision about the source.

However, I believe that in doubtful cases it's better to apply AI to the
complete message data than to attempt to analyse some abbreviated
notification. In most cases MTAs have enough capacity to do this: at the
moment (according to my stats) doubtful cases are about 30% of the email
that gets past blacklists.

I think it would be more useful for you to try to fix greylisting than to
redesign IM2000 yet again.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOVER WIGHT: WEST 5 OR 6 VEERING NORTH OR NORTHWEST 5 TO 7, PERHAPS GALE 8
LATER. SLIGHT OR MODERATE, BECOMING ROUGH FOR A TIME. SQUALLY SHOWERS. GOOD.

To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject:  Re: SMTP Transferred-By-Reference
Date:  Tue, 13 Nov 2007 08:14:58 +0100
Lines: 13
Reply-To: "Frank Ellermann" <[email-masked]>
Sender: [email-masked]

John Leslie wrote:
=20
> Doug Otis and I have been working up a proposal for a SMTP extension
> to shift some of the burden of spam abatement away from the receiving
> SMTP servers towards the originator.

I've looked into it, it's a bit odd, a kind of "blog and IM over SMTP",
but in theory it might work.  The underlying problem is already solved
by SPF as far as I'm concerned, if folks want another solution, why not.

There might be some privacy concerns, I'm not sure about it. =20

 Frank

Date: Mon, 12 Nov 2007 13:37:10 -0800
To: John Leslie <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

At 07:09 12-11-2007, John Leslie wrote:
>    This quite specifically allows reputation-checking to proceed at
>leisure, not subject to time constraints (except for recipients who
>demand sub-millisecond delivery of every spam message ;^)

We won't have the entire message to make a proper 
assessment.  Reputation-checking only works with known 
senders.  Recipients expect zero spam and instant delivery of their emails. :-)

TBR assumes a direct path to retrieve the message.  That's not always 
the case for Internet mail.  There may also be HTTP proxies in 
between which may affect message retrieval.

The main problem these days is not storage.  Email may be seen as 
fire and forget.  I gather that your draft attempts to reverse 
that.  That may not have that much impact on "questionable" senders 
as they have unlimited resources and a propensity to overcome the hurdles.

Regards,
-sm 

Date: Mon, 12 Nov 2007 16:31:27 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Douglas Otis wrote:

> When the SMTP client's reputation is unknown, an SMTP server is unlikely
> to Perm Error at the EHLO.  Temp Errors at this point might be used to
> discern whether the SMTP client keeps state, as with greylisting.  Temp
> errors might also be used to delay acceptance when a spam campaign is
> detected as being active at an SMTP client handling messages from both
> good and bad actors.  TBR provides a safe alternative to greylisting and
> temporary holds.

How is it "safer" than greylisting?

[...]

> This critique does not apply for a few reasons.  The sender is unable to
> lie about the message source, and determining the source will not cost
> the recipient additional resources.

How does not being able to lie about the message source help reduce spam?
Remember, go back to the threat model:  Serious spammers have essentially
unlimited bandwidth and unlimited CPU power.  When you control an army
of tens of thousands of bots, it's cheap enough to set up a few hundred HTTP
servers to serve your TBR messages.  (That's assuming spammers even go to
the bother of using TBR.)

> Accepting and then processing a 512 byte message results in fewer
> packet exchanges than would refusing 2 or more multi-KB messages.
> (At today's level of spam, this could be estimated +20 at +8 KB.)

Is there evidence that the number of bytes or packages involved in e-mail
transfer is a problem?  I would guess that ISPs have far more of their
bandwidth chewed up by P2P and HTTP/HTTPS traffic than e-mail.

IOW, I don't think the problem that you are trying to solve is worth solving,
and I don't think the real spam problem is solved by TBR.

Regards,

David.

To: Douglas Otis <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: SMTP Transferred-By-Reference
From: [email-masked]
Date: Mon, 12 Nov 2007 15:59:01 -0500
Sender: [email-masked]

--==_Exmh_1194901141_3241P
Content-Type: text/plain; charset=us-ascii

On Mon, 12 Nov 2007 10:48:00 PST, Douglas Otis said:

> > 2. At best, this reduces total bytes over the but the requirement  
> > for a notification message does not reduce the number of network  
> > 'transactions' -- in fact it increases them by 100% or more.
> 
> This calculation is wrong.  You are assuming a substantial percentage  
> of messages will be retrieved, yet the typical levels of spam would  
> suggest otherwise.

Note that a fairly large percentage of spam is flagged as such by scanning
the *body* of the message - in other words, you don't know if it's spam until
you fetch the data *anyhow*.  What percent of spam are you blocking based
*only* on the SMTP MAIL FROM/RCPT TO data?  That's the percentage you can
avoid fetching the body - and if you're smart you *currently* 4xx/5xx the
transaction before you get to DATA in that case, so you're not saving any
network bandwidth by not fetching it over a second connection.

--==_Exmh_1194901141_3241P
Content-Type: application/pgp-signature

--==_Exmh_1194901141_3241P--

Cc: John Leslie <[email-masked]>, [email-masked]
From: Douglas Otis <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Mon, 12 Nov 2007 12:43:07 -0800
To: Tony Finch <[email-masked]>
Sender: [email-masked]

On Nov 12, 2007, at 8:34 AM, Tony Finch wrote:

Tony,

When the SMTP client's reputation is unknown, an SMTP server is  
unlikely to Perm Error at the EHLO.  Temp Errors at this point might  
be used to discern whether the SMTP client keeps state, as with  
greylisting.  Temp errors might also be used to delay acceptance when  
a spam campaign is detected as being active at an SMTP client  
handling messages from both good and bad actors.  TBR provides a safe  
alternative to greylisting and temporary holds.  TBR makes makes an  
additional demand on recipient resources only in cases where the  
message source is actually trusted.  However, for those sources,  
normal delivery can be used.  When a message does get fetched, this  
additional burden of TBR still represents less than that required for  
other source identification schemes.

> On Mon, 12 Nov 2007, John Leslie wrote:
>>
>> Doug Otis and I have been working up a proposal for a SMTP  
>> extension to shift some of the burden of spam abatement away from  
>> the receiving SMTP servers towards the originator.
>
> http://www1.ietf.org/mail-archive/web/asrg/current/msg12022.html

This critique does not apply for a few reasons.  The sender is unable  
to lie about the message source, and determining the source will not  
cost the recipient additional resources.  Accepting and then  
processing a 512 byte message results in fewer packet exchanges than  
would refusing 2 or more multi-KB messages.  (At today's level of  
spam, this could be estimated +20 at +8 KB.)  With TBR, the recipient  
is permitted to decide at anytime after receipt whether to perform a  
subsequent process (fetch or store and forward).  The typical  
percentage of undesired versus desired email results in the added  
extension being far less resource intensive.  TBR offers greater  
leeway in safely handling difficult cases.

-Doug

Date: Mon, 12 Nov 2007 14:34:38 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: response to David?
Sender: [email-masked]

John Leslie wrote:

[...]

>    Note the "temporary" errors listed in Section 6, e.g.

> " 451 4.7.8 TBR HTTP/HTTPS mode requested for immediate acceptance

> This can be an explicit graylisting message: though it doesn't say
> anything about how long you'll be graylisted for not using TBR, it
> is _some_ incentive to use it if you believe your message worthwhile.

I suppose.  That falls into my category of "punishing those who use
standard SMTP."

[...]

>    The devil is in the details. What is the right level of incentive?

You have to think about the right threat model.  Start from the
assumption that spammers have essentially unlimited bandwidth and CPU
power.  Next assume that they're not constrained by moral, ethical or
even legal considerations.  Once you have a threat model like that,
you'll see that TBR won't help against spam.

>    IMHO, deployment would start with organizations concerned enough
> about their reputation to desire the archiving features of TBR.

You don't need TBR to do archiving.

> Once it gets implemented in open-source MTA packages, there would
> be a dribble of receivers advertising it.

I'm not sure any open-source MTA packages will want to pick this up.  In
addition to implementing SMTP, they'll now have to worry about implementing
HTTP.  Sure, there are free libraries like libcurl that you can just throw
in --- assuming their licenses are compatible with yours.

>> Additionally, it's difficult for the server holding the message body
>> to know when it's safe to delete it.  Let's say a message is sent
>> to 100 recipients in 37 different domains.  Should the message body
>> be deleted only after 100 attempts to download it?  37 attempts?
>> Some other number?

>    In that particular case, any number picked out of thin air is about
> as good as any other.

So if we fail to retrieve the URL because it has been deleted... then what?
We generate a DSN?  We silently fail?  Either approach is unpalatable;
we've just introduced a *new* failure mechanism that doesn't exist in
standard SMTP.

> I wouldn't call it unreasonable to delete it
> after 37, given that the URI _should_ be hard to guess, and one would
> like to believe your server will be dependable. But it's more likely
> it would be kept on the server for some period of _time_.

So to be safe, the server will have to keep the message around for N days
even upon successfull delivery?  Considering that one of the most annoying
headaches of running a large-scale e-mail system is clogged queues, I really
can't see this prospect having much appeal.

>    We did not intend such a feature. Do understand that by the time
> the receiving Mail Delivery Agent fetches the URI, the DSN path is
> proven, so DSNs _could_ be sent safely for P, Q and R.

Yes, ok, but you don't need TBR to safely establish a DSN path.
If you use a MAIL FROM: rewriting scheme to detect and discard
backscatter, that would be fine -- but you can do that without any
SMTP extensions.

[...]

>    In any case, I find that in practice most users have completely
> lost the headers which would be helpful, and I have to reconstruct
> from logs. Doug argues that for scalability we need to strictly
> limit our storage responsibility for likely-unwanted email.

That makes no sense.  Existing e-mail systems store headers and entire
body and they seem to be scaling OK.  I don't think any proposal that
breaks SMTP traceability has a chance of passing.

[...]

>    OTOH, AOL can add a few terabytes of storage much more cheaply
> than the diffuse recipients can; and if AOL stores it, there's a
> far better assurance that it'll arrive as intended. I would
> expect TBR to be seen as a higher-value service -- increasing in
> value as deployment increases.

Mmmm, ok.  Anyone from AOL care to comment? :-)

TBR will have essentially no value until/unless a significant
percentage of the Internet deploys it (let's say 1% of all e-mail
recipients.)  So you have a chicken-and-egg problem trying to convince
people.  I don't mean to be cruel or flippant, but this looks like a
candidate for
http://www.rhyolite.com/anti-spam/you-might-be.html#senior-IETF-member-5

[...]

>    Hard to say... At first blussh, that looks more like a SMTP
> replacement which never reached critical mass. TBR is a SMTP
> extension, which is a much simpler implementation path.

Well, not really.  It's a rather large extension with a reasonably high
cost to implement and uncertain benefits.

Regards,

David.

Date: Mon, 12 Nov 2007 13:59:21 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Greylisting usefulness (was Re: SMTP Transferred-By-Reference)
Sender: [email-masked]

Dave Crocker wrote:

> Graylisting is generally acknowledged to have current tactical utility
> but something that spammers will adapt to whenever they feel the need.

Actually, I think greylisting will have longer-term uses than that.
Forcing spammers to be pinned to the same IP address for a while gives
techniques like DNS-based blocklists time to catch up.  So while it's
trivial to adapt to greylisting, it makes your zombie machine more
likely to be recognized as such during the retransmission.  I think
the OP had a good point about being able to make better decisions in
non-realtime, but I think his mechanism for deferring the decision is
seriously flawed.

Regards,

David.

Cc: [email-masked]
From: Douglas Otis <[email-masked]>
Subject: Re: SMTP Transferred-By-Reference
Date: Mon, 12 Nov 2007 10:48:00 -0800
To: [email-masked]
Sender: [email-masked]

On Nov 12, 2007, at 7:56 AM, Dave Crocker wrote:

>
> actual:
> <http://patft.uspto.gov/netacgi/nph-Parser? 
> Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch- 
> adv.htm&r=62&f=G&l=50&d=PTXT&s1=tumbleweed&p=2&OS=tumbleweed&RS=tumble 
> weed>
>
> They enforce this patents and its follow-ons very aggressively.

This is under review.

> 2. At best, this reduces total bytes over the but the requirement  
> for a notification message does not reduce the number of network  
> 'transactions' -- in fact it increases them by 100% or more.

This calculation is wrong.  You are assuming a substantial percentage  
of messages will be retrieved, yet the typical levels of spam would  
suggest otherwise.  In addition, this can be an optional mode used  
only  for unknown sources.  This calculation also fails to consider  
the transactions occurring when a DSN is generated.

> 3. This presumes that making a real-time decision is a current  
> problem, when it is not generally held to be a major factor among  
> the anti-abuse community.  Sure, it would be nice to be able to do  
> it, but it's a long way from the top of the list.

The number of back-scatter sources containing spam and malware  
represents more than the number of direct threat sources.   
Unfortunately, uncovering threats often necessitates more time than  
permitted for an Okay.  All modes of spam must be blocked, or  
exceptions soon become a predominate mode for bad-actors.  This  
battle simply does not scale as a real-time effort.  Some obfuscation  
strategies evade detection through rapid deployment of related  
infrastructure.  Postponing acceptance, even a few minutes, goes a  
long way toward thwarting this pernicious tactic.  Reputation is  
never instantaneous.

> 4. It presumes that users can make the right decision.  Experience  
> is pretty clear that that's too often not a correct presumption.   
> In addition, having users be required to make this decision burdens  
> them far more than is felt to be useful. (This is a derivation of  
> the transaction cost item, above, except that it moves the decision- 
> making from a receive-side front-end filter to the human user.  And  
> of course, it them requires them to wait for the message to show up.

You have misread the intent.  The mail delivery agent decides what  
gets delivered.

> 5. Doesn't work so well for disconnected users.

This will not impact disconnected users at all.

-Doug

Date: Mon, 12 Nov 2007 13:37:55 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: response to David?
Sender: [email-masked]

David F. Skoll <[email-masked]> wrote:
> John Leslie wrote:
> 
>> http://datatracker.ietf.org/drafts/draft-otis-smtp-tbr-ext
> 
> But what incentives do spammers have to use this extension?

   Good question! Quite possibly none...

> As long as normal SMTP exists, I doubt that this will do anything to
> reduce spam unless we somehow punish senders who do *not* use the
> extension.

   You may be right...

   Note the "temporary" errors listed in Section 6, e.g.
" 
" 451 4.7.8 TBR HTTP/HTTPS mode requested for immediate acceptance

This can be an explicit graylisting message: though it doesn't say
anything about how long you'll be graylisted for not using TBR, it
is _some_ incentive to use it if you believe your message worthwhile.

   (Actually, of course, being a temporary error, most users probably
wouldn't even see it: it would be seen by an MTA manager scanning the
logs. If users actually _did_ see it, it might lead to some irate
tech-support calls...)

> And that, IMO, is not a good idea.

   The devil is in the details. What is the right level of incentive?

   IMHO, deployment would start with organizations concerned enough
about their reputation to desire the archiving features of TBR.
Once it gets implemented in open-source MTA packages, there would
be a dribble of receivers advertising it.

> Additionally, it's difficult for the server holding the message body
> to know when it's safe to delete it.  Let's say a message is sent
> to 100 recipients in 37 different domains.  Should the message body
> be deleted only after 100 attempts to download it?  37 attempts?
> Some other number?

   In that particular case, any number picked out of thin air is about
as good as any other. I wouldn't call it unreasonable to delete it
after 37, given that the URI _should_ be hard to guess, and one would
like to believe your server will be dependable. But it's more likely
it would be kept on the server for some period of _time_.

> There is no way I can see to give feedback to the server that "Yes,
> I take responsibility for delivering this message to recipients
> X, Y and Z (but not P, Q and R.)"

   We did not intend such a feature. Do understand that by the time
the receiving Mail Delivery Agent fetches the URI, the DSN path is
proven, so DSNs _could_ be sent safely for P, Q and R.

> Making trace headers optional is asking for trouble.  It makes
> diagnosing mail problems nightmarish.

   I tend to agree with you; Doug does not...

   In any case, I find that in practice most users have completely
lost the headers which would be helpful, and I have to reconstruct
from logs. Doug argues that for scalability we need to strictly
limit our storage responsibility for likely-unwanted email.

   Personally, I'm happy enough to leave it optional, though I'd
be unhappy to see an implementation which didn't allow keeping the
headers.

> If a TBR message is not fetched, how long should the server hang
> on to it? Suppose you're AOL or Hotmail and in the business of
> sending several billion e-mails per day... what implications does
> implementing TBR have on your storage requirements?

   Considerable impact, IMHO.

   OTOH, AOL can add a few terabytes of storage much more cheaply
than the diffuse recipients can; and if AOL stores it, there's a
far better assurance that it'll arrive as intended. I would
expect TBR to be seen as a higher-value service -- increasing in
value as deployment increases.

   As for how long -- that's likely to track how long you keep
trying graylisted emails. I'd keep them at least long enough to
be able to notify users that we've stopped trying.

> This looks more-or-less like a reinvention of
> http://cr.yp.to/im2000.html
> and will probably meet with the same level of enthusiasm.

   Hard to say... At first blussh, that looks more like a SMTP
replacement which never reached critical mass. TBR is a SMTP
extension, which is a much simpler implementation path.

--
John Leslie <[email-masked]>

Date: Mon, 12 Nov 2007 10:24:36 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Leslie <[email-masked]>
CC: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John Leslie wrote:
>> 2. At best, this reduces total bytes over the [SMTP connection?]
> 
>    Actually, our concern was bytes an SMTP server is responsible for
> buffering: this may not be much of a problem for many folks (yet), 

In most Internet situations, transaction costs outweigh bandwidth costs.  Your 
scheme doubles the former, at a minimum, while introducing further delay and 
synchronization constraints.

To repeat:  while smtp-time evaluation does show up as a concern among some 
anti-abuse operations folks, it has never been seen as a serious priority by 
the general community.

With abuse email at 90+ percent of the total traffic, I think we'd have seen 
the 'scaling' problem by now, if we were going to.  Yes, it can (and will) get 
worse, but ultimately this needs a deeper solution, which I believe source 
authentication serves better.

>> but the requirement for a notification message does not reduce the
>> number of network 'transactions' -- in fact it increases them by
>> 100% or more.
> 
>    I'm not clear what "requirement" you're referring to.

>    There is no notification message required in this SMTP extension:
> in fact there is a prohibition of sending DSNs before the reverse
> path is verified. The notification of acceptance of responsibility
> for the TBR _is_ fetching the reference: none of the errors mentioned
> are required to be sent unless the originator is considered worthy.
> 
>> 3. This presumes that making a real-time decision is a current problem, 
>> when it is not generally held to be a major factor among the anti-abuse 
>> community.
> 
>    Your "community" may not be the same as mine. I take the very
> existence of graylisting as proof that real-time decisions are often
> impractical.

Graylisting is generally acknowledged to have current tactical utility but 
something that spammers will adapt to whenever they feel the need.  So it's 
not a long-term (strategic) tool. The cost of a standard requires that it 
provide stable, long-term benefit.

>> 4. It presumes that users can make the right decision.
> 
>    Actually, the TBR extension protocol leaves _no_ decisions to the
> user (unless you read the part about

Yeah, I missed that it's only goal was to defer the DATA portion for awhile. 
In effect, altering the timeout for the DATA portion.

So that just leaves the matter of realistic long-term efficacy.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 12 Nov 2007 13:05:50 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Dave Crocker <[email-masked]> wrote:
> 
> <http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=62&f=G&l=50&d=PTXT&s1=tumbleweed&p=2&OS=tumbleweed&RS=tumbleweed>
> 
> They enforce this patents and its follow-ons very aggressively.

   Thank you for the patent reference. I have only read the abstract
so far, but that's enough for me to say I need to have a lawyer give
his/her opinion before I can venture an opinion whether it applies.
I will discuss this with Doug.

> 2. At best, this reduces total bytes over the [SMTP connection?]

   Actually, our concern was bytes an SMTP server is responsible for
buffering: this may not be much of a problem for many folks (yet), but
we see it as a scaling issue which will definitely hit us if the spam
volume continues to grow. (Besides, SMTP server load is highly
variable, and there will be times when another megabit-per-second
really hurts.)

> but the requirement for a notification message does not reduce the
> number of network 'transactions' -- in fact it increases them by
> 100% or more.

   I'm not clear what "requirement" you're referring to.

   There is no notification message required in this SMTP extension:
in fact there is a prohibition of sending DSNs before the reverse
path is verified. The notification of acceptance of responsibility
for the TBR _is_ fetching the reference: none of the errors mentioned
are required to be sent unless the originator is considered worthy.

> 3. This presumes that making a real-time decision is a current problem, 
> when it is not generally held to be a major factor among the anti-abuse 
> community.

   Your "community" may not be the same as mine. I take the very
existence of graylisting as proof that real-time decisions are often
impractical.

> 4. It presumes that users can make the right decision.

   Actually, the TBR extension protocol leaves _no_ decisions to the
user (unless you read the part about
" 
" MAY prepend any additional trace headers to a notification sent to
" the recipient containing the eXAM-URI itself, along with any other
" appropriate information.

in Section 3.2 to be asking the user to decide something).

> Experience is pretty clear that that's too often not a correct
> presumption.

   Agreed: IMHO very few users can make the right decision about spam
more than 99% of the time. Many users miss that mark by more than one
order of magnitude. ;^)

> In addition, having users be required to make this decision burdens
> them far more than is felt to be useful.

   I agree. That's why we don't ask for it: the MAY cited above is a
sop to the users who worry themselves silly about missing _one_
legitimate email in the heap of spam.

> (This is a derivation of the transaction cost item, above, except
> that it moves the decision-making from a receive-side front-end
> filter to the human user.  And of course, it them requires them 
> to wait for the message to show up.

   (I don't think this applies, but I'm willing to be shown otherwise.)

> 5. Doesn't work so well for disconnected users.

   The extension envisions the fetching being done by a Mail Delivery
Agent (which presumably is well connected). But the fetching clearly
could be done earlier in the chain...

--
John Leslie <[email-masked]>

Date: Mon, 12 Nov 2007 16:34:03 +0000
From: Tony Finch <[email-masked]>
To: John Leslie <[email-masked]>
cc: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

On Mon, 12 Nov 2007, John Leslie wrote:
>
>    Doug Otis and I have been working up a proposal for a SMTP extension
> to shift some of the burden of spam abatement away from the receiving
> SMTP servers towards the originator.

Sigh. Not another one.

http://www1.ietf.org/mail-archive/web/asrg/current/msg12022.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOVER WIGHT PORTLAND PLYMOUTH: NORTHERLY BACKING NORTHWESTERLY 3 OR 4,
INCREASING 5 TO 7. MODERATE OR ROUGH. SHOWERS THEN RAIN. MODERATE OR GOOD.

Date: Mon, 12 Nov 2007 08:29:50 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
CC: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

Dave Crocker wrote:
> 
> 
> 
> John Leslie wrote:
>> Doug Otis and I have been working up a proposal for a SMTP extension 
>> to shift some of the burden of spam abatement away from the receiving 
>> SMTP servers towards the originator. It is now published at:
>>
>> http://datatracker.ietf.org/drafts/draft-otis-smtp-tbr-ext
>>
>> In essence it holds the originator responsible for maintaining the 
>> copy of the message until a receiving Mail User Agent determines that 
>> it should be delivered to the recipient.
> 
> 1. <http://tinyurl.com/2lpvkm>

For some reason, that tinyurl gives the wrong page.

This one seems to work correctly:

   <http://tinyurl.com/2jhsqt>

d/

ps. David Skoll's point, about adoption incentives, nicely opens the door to 
another set of reasons this type of mechanism does not really solve the stated 
problem.
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 12 Nov 2007 11:06:59 -0500
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John Leslie wrote:

>    Doug Otis and I have been working up a proposal for a SMTP extension
> to shift some of the burden of spam abatement away from the receiving
> SMTP servers towards the originator. It is now published at:

> http://datatracker.ietf.org/drafts/draft-otis-smtp-tbr-ext

>    In essence it holds the originator responsible for maintaining the
> copy of the message until a receiving Mail User Agent determines that
> it should be delivered to the recipient.

But what incentives do spammers have to use this extension?  As long
as normal SMTP exists, I doubt that this will do anything to reduce
spam unless we somehow punish senders who do *not* use the extension.
And that, IMO, is not a good idea.

Additionally, it's difficult for the server holding the message body
to know when it's safe to delete it.  Let's say a message is sent
to 100 recipients in 37 different domains.  Should the message body
be deleted only after 100 attempts to download it?  37 attempts?  Some
other number?  There is no way I can see to give feedback to the server
that "Yes, I take responsibility for delivering this message to recipients
X, Y and Z (but not P, Q and R.)"

Making trace headers optional is asking for trouble.  It makes
diagnosing mail problems nightmarish.

If a TBR message is not fetched, how long should the server hang on to it?
Suppose you're AOL or Hotmail and in the business of sending several billion
e-mails per day... what implications does implementing TBR have on your
storage requirements?

This looks more-or-less like a reinvention of http://cr.yp.to/im2000.html
and will probably meet with the same level of enthusiasm.

Regards,

David.

Date: Mon, 12 Nov 2007 07:56:38 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: SMTP Transferred-By-Reference
Sender: [email-masked]

John Leslie wrote:
> Doug Otis and I have been working up a proposal for a SMTP extension to 
> shift some of the burden of spam abatement away from the receiving SMTP 
> servers towards the originator. It is now published at:
> 
> http://datatracker.ietf.org/drafts/draft-otis-smtp-tbr-ext
> 
> In essence it holds the originator responsible for maintaining the copy of 
> the message until a receiving Mail User Agent determines that it should be 
> delivered to the recipient.

1. <http://tinyurl.com/2lpvkm>

actual:
<http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=62&f=G&l=50&d=PTXT&s1=tumbleweed&p=2&OS=tumbleweed&RS=tumbleweed>

They enforce this patents and its follow-ons very aggressively.

2. At best, this reduces total bytes over the but the requirement for a 
notification message does not reduce the number of network 'transactions' -- 
in fact it increases them by 100% or more.

3. This presumes that making a real-time decision is a current problem, when 
it is not generally held to be a major factor among the anti-abuse community. 
  Sure, it would be nice to be able to do it, but it's a long way from the top 
of the list.

4. It presumes that users can make the right decision.  Experience is pretty 
clear that that's too often not a correct presumption.  In addition, having 
users be required to make this decision burdens them far more than is felt to 
be useful. (This is a derivation of the transaction cost item, above, except 
that it moves the decision-making from a receive-side front-end filter to the 
human user.  And of course, it them requires them to wait for the message to 
show up.

5. Doesn't work so well for disconnected users.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 12 Nov 2007 10:09:54 -0500
From: John Leslie <[email-masked]>
To: [email-masked]
Subject: SMTP Transferred-By-Reference
Sender: [email-masked]

   Doug Otis and I have been working up a proposal for a SMTP extension
to shift some of the burden of spam abatement away from the receiving
SMTP servers towards the originator. It is now published at:

http://datatracker.ietf.org/drafts/draft-otis-smtp-tbr-ext

   In essence it holds the originator responsible for maintaining the
copy of the message until a receiving Mail User Agent determines that
it should be delivered to the recipient.

   This quite specifically allows reputation-checking to proceed at
leisure, not subject to time constraints (except for recipients who
demand sub-millisecond delivery of every spam message ;^)

   It also gives us a "for-free" version of graylisting, in that no
further network traffic is needed to enforce a time delay before a
message is accepted from an unknown originator.

   As part of the proposal, we add conditions that ensure a workable
return path for Delivery Status Notifications, requiring that the
same domain which manages the server from which the message will be
fetched also publishes a MX record to receive the DSNs.

   Obviously, comments are welcome...

--
John Leslie <[email-masked]>

Date: Thu, 01 Nov 2007 21:48:54 -0400
From: Tony Hansen <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Mailing List Last Call now open for draft-klensin-rfc2821bis-05.txt
Sender: [email-masked]

<pseudo-chair hat on>

Ok, the Mailing List Last Call for draft-klensin-rfc2821bis-05.txt is
now over. The comments have dwindled off and I think we're just about
ready for an IETF last call.

John will probably be issuing one more issue before doing the IETF last
call, to incorporate some of the comments that have been expressed in
the last two weeks. Stay tuned.

</pseudo-chair hat off>

	Tony Hansen
	[email-masked]

Tony Hansen wrote:
> This is a "formal" Mailing List Last Call on
> draft-klensin-rfc2821bis-05.txt. The last call will last for two weeks
> time, ending on October 30, 2007.