Subject: "addressing" comments on draft-crocker-email-arch-04.txt
From: Bruce Lilly <[email-masked]>
To: Dave Crocker <[email-masked]>
Cc: ietf-smtp <[email-masked]>
Date: Thu, 31 Mar 2005 20:41:52 -0500
List-Id: <ietf-smtp.imc.org>

The email architecture draft uses several terms (and neglects at least
one term) in ways that may cause or perpetuate confusion and lead to
loss of interoperability.  The specific terms used are addr-spec,
address, and mailbox.  Path and angle-addr are related terms used in the
core email RFCs, but are not mentioned in the draft.  The terms are
related, mostly by a hierarchy, and have specific uses in the message
format and mail architecture.

The differences are perhaps best illustrated by concrete examples as
well as descriptions and references:

addr-spec   [email-masked]

angle-addr  <@mx.mail.rcn.net:[email-masked]>

mailbox     Bruce Lilly <@mx.mail.rcn.net:[email-masked]>

address     Bruce Lilly: <[email-masked]>, [email-masked];

An addr-spec (RFCs 822, 2822) is the basic addressing specification
consisting of a local-part and domain, delimited by the special
character '@'.  Every addr-spec is (syntactically) a mailbox and an
address.  No addr-spec is an angle-addr.  No angle-addrs are addr-specs.
Not all mailboxes are addr-specs.  Not all addresses are addr-specs.

An angle-addr (RFCs 822, 2822, called path in RFCs 821, 2821) is an
angle-bracketed construct which consists of an optional comma-separated
list of domains terminated by a colon (a route) followed by an addr-
spec.  N.B. although officially deprecated, a non-null route is still
sometimes required to work around temporary DNS problems.  An angle-addr
is used in SMTP MAIL FROM and RCPT TO commands.  During SMTP transfer
using an angle-addr with a non-null route forward path, the route may be
modified by having a host strip its domain from the front of the route.
At "final" delivery, SMTP copies the (possibly modified) path used for
delivery notifications and places the resulting angle-addr in a
Return-Path field.  All angle-addrs are (syntactically) mailboxes and
addresses.  No angle-addrs are addr-specs and vice versa.  Not all
mailboxes are angle-addrs.  Not all addresses are angle-addrs.
Angle-addrs with non-null routes are deprecated in header fields, but
parsers are required to be able to parse paths with routes.

The term "mailbox" has two uses (RFCs 822, 2822):

   o As an abstract term for an entity that receives email.

   o As a syntactic unit corresponding to that abstract term.

As a syntactic unit, a mailbox may consist of a naked addr-spec (all
addr-specs are syntactically mailboxes), or as an angle-addr with an
optional display name (all angle-addrs are syntactically mailboxes).
Mailboxes which are not also raw angle-addrs cannot be used in SMTP
commands (naked addr-specs and display names are prohibited; therefore
the valid mailbox example above, which is also a valid address, cannot
be used in SMTP command arguments).  A mailbox is a single entity; each
message author might (or might not) have a mailbox; authors' mailboxes
are specified in (comma-separated) list form in the From message header
field.  A message may have multiple authors, a sole author, or one or
more authors with no mailbox.  A message sender (the individual entity
which causes message transport to be initiated) is expected to
unconditionally have a (syntactically valid) mailbox, which is specified
in the message Sender header field.  The From field has a list of
mailboxes of any valid syntactic form.  The Sender field has only a
single mailbox, which may have any valid syntactic form.

An address is the richest form of specification.  An address may be a
mailbox (all mailboxes are addresses) or a display name followed by a
colon, a possibly empty comma-separated list of mailboxes, followed by a
semicolon (a named group).  While named groups are addresses, addresses
which are named groups are not themselves mailboxes.  Message header To,
Cc, and Reply-To fields may contain non-empty comma-separated lists of
addresses.  Message Bcc header fields may be empty or may contain
comma-separated lists of addresses.

The restrictions are important to bear in mind:

   o Message From and Sender fields cannot contain named groups
     (addresses which are not mailboxes), only mailboxes (and only a
     single mailbox in Sender).

   o SMTP command arguments cannot use named groups (addresses which are
     not mailboxes) or mailboxes with display names, or naked
     addr-specs; only angle-addrs.  The message header Return-Path field
     is derived from an SMTP command argument and is subject to the same
     restrictions.

Bandying these terms about carelessly, e.g. as if they were
interchangeable, leads to confusion.  If the restrictions are violated
as a result, loss of interoperability can occur.

Unfortunately, the draft does not maintain the distinctions between
these terms.

Specific suggestions for improvement:

Section 2.1.1: Maintain the distinction between Author and Sender.  By
   coincidence, the Sender may be an author, or by additional
   coincidence, the sole author.  In general, Author(s) and Sender are
   distinct.

Section 2.1.3: Ditto. The transport initiator (Sender) is not
   necessarily the (an) author.

Section 2.2.3, last paragraph, two places: "Bounce" (implies non-
   delivery, though delivery notifications may be positive as well as
   negative) messages are sent to a path (angle-addr), not an address.

Section 3: conflates three of the terms, and inconsistently mixes in
   angle brackets as well!  It ignores angle-addrs and the routes which
   they may contain.

Section 3.1: conflates two terms in the heading and three in the first
   paragraph after the RFC 2822 quote.  That paragraph fails to note
   that a mailbox may be an angle-addr (not even mentioned) with an
   optional display name.  It also states that the RHS (a domain) is
   equivalent to an Administrative Unit, which is incorrect (a single
   Administrative Unit might manage several domain names, or a single
   domain name might be shared among several Administrative Units).  The
   following paragraph uses "address" where "mailbox" or "addr-spec"
   would be appropriate, and mixes in angle brackets.

Section 3.1.1, second paragraph: "addresses" should be "mailboxes"

Section 3.1.2, heading and first paragraph: "address" (and addresses)
   should be "mailbox" (mailboxes)

Section 3.2: "mailbox address" should be "addr-spec or angle-addr"

Section 3.3, first sentence: "mailbox addresses" should be addr-specs
   and angle-addrs"

Section 4.2: References to From and Sender fields should refer to
   "mailboxes" and "mailbox" respectively.  The description of the
   Reply-To field is an oversimplification; as noted in RFCs 822 and
   2822, the Reply-To field has multiple uses, including specification
   of alternate mailboxes (as addresses) for the author(s), for
   delegation of responses to others, and for directing responses (e.g.)
   to a mailing list mailbox (as an address).  As noted by several
   persons in a number of messages on the ietf-822 mailing list, the
   mailbox(es) for responses are not a simple mechanical matter; they
   depend as much on the intent of the respondent as on the suggestion
   of the originator(s).  Several MUAs have separate response functions
   to direct responses to the author(s), regardless of the Reply-To
   field.  The section states that the Sender field is set by "Source"
   (as distinct? from Originator), whereas RFC 2822 section 3.6.2
   clearly states that Sender is an Originator field.  Discussion
   paragraph should replace "address" (several places) with "mailbox";
   but there are additional problems.  It claims that the "bounce"
   "address" (see above) is set by the Sender field, which is incorrect
   -- that might happen in a particular implementation, but the Sender
   mailbox and the SMTP return path (for delivery notifications) may in
   general be unrelated.

To, Cc, Bcc: use of "address" is correct!

Section 4.3: Discussion of SMTP MAIL FROM and RCPT TO (several places)
   should use "angle-addr" or "path" in lieu of "address", and the
   "mailing list address" should be "mailing list mailbox".

Section 4.4: Ditto; also note that routes in paths might be modified.

Section 4.5: first paragraph should use "mailbox" instead of "address".
   Reference to SMTP MAIL FROM and the Return-Path field should use
   "path" or "angle-addr" insted of "address".

Section 5: Same comments as 4.3/4.4; also "mailbox address" -> "path" or
   "angle-addr".

Section 5.1: references to SMTP "envelope", MAIL FROM, RCPT TO should
   use "path" or "angle-addr".  Reference to delivery (under Received)
   should use "mailbox" instead of "address".

Section 5.2: The From, Resent-From, and Resent-Sender fields contain
   mailboxes, not addresses.  Regarding the display name, it is
   structured (an RFC [2]822 phrase), not "free-form".  And as for
   modification: GACK! PTUI!  Delivery notifications are specified with
   a path (angle-addr) and are independent of the Resent-Sender mailbox.
   SMTP MAIL FROM and RCPT TO are paths (angle-addrs).

Section 5.3: Mailing lists, authors, and the sender have mailboxes, not
   addresses.  The Reply-To field (RFC 2822 section 3.6.2) is an
   Originator field, full stop.  It MUST NOT be modified by any entity.
   "Reply to author", on MUAs that have a specific function for that
   (Kmail, e.g.)  use the mailbox(es) in the From field, since that
   field holds the mailbox(es) for the author(s).  As noted above,
   Sender is an Originator field.  SMTP MAIL FROM and RCPT TO use paths
   (angle-addrs), not addresses.  List expanders SHOULD set the reverse
   path (MAIL FROM) to point to the list administrator, and SHOULD NOT
   leave MAIL FROM unaltered.

Section 5.4: Sending is based on paths (angle-addrs) in the SMTP
   envelope, not addresses.  From and Sender fields use mailboxes;
   Sender is an Originator field.  MAIL FROM path (angle-addr) might
   have to be transformed into something comparable on the "foreign"
   side of the gateway, but SHOULD NOT be changed to point to a gateway
   operator mailbox except as a last resort (otherwise delivery
   notifications will never reach the correct mailbox).

Whew. There might well be other sections that need to be edited to
maintain the distinctions.  Obviously there are many changes; I would
recommend another draft revision review here before wider IETF review.

From: "Hector Santos" <[email-masked]>
To: "Mark E. Mallett" <[email-masked]>, <[email-masked]>
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 18:26:21 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

And just for final completness and the interested "Can't we all just get
along" Windows crowd..

/*  f.c -- fetch via HTTP and dump the entire session to stdout
    very stupidly.

    - ported to Windows to illustrate need to change the stdout device
      default _O_TEXT cooked mode setting to _O_BINARY raw mode.

*/

#ifdef _WIN32
#include <windows.h>
#include <stdio.h>
#include <string.h>
#include <winsock.h>
#include <fcntl.h>
#include <io.h>
#pragma comment(lib,"wsock32.lib")
#define close(a)      closesocket(a)
#define read(a,b,c)   recv(a,b,c,0)
#define write(a,b,c)  send(a,b,c,0)
#else
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <signal.h>
#endif

main(argc, argv)
 int  argc;
 char  **argv;
{
   int  pfd;  /* fd from socket */
   int  len;
   char  *hostP, *fileP;
   char  buf[1024];
   struct hostent *hP;  /* for host */
   struct sockaddr_in sin;
#ifdef _WIN32
   WSADATA wd;
#endif

    if ( argc != 3 ) {
       fprintf( stderr, "Usage: %s host file\n", argv[0] );
       exit( 1 );
    }

#ifdef _WIN32
    if (WSAStartup(MAKEWORD(1, 1), &wd) != 0) {
        exit(1);
    }
    _setmode( _fileno( stdout ), _O_BINARY );
#endif

    hostP = argv[1];
    fileP = argv[2];

    hP = gethostbyname( hostP );
    if ( hP == NULL ) {
       fprintf( stderr, "Unknown host \"%s\"\n", hostP );
       exit( 1 );
    }

    pfd = socket( AF_INET, SOCK_STREAM, 0 );
    if ( pfd < 0 ) {
        perror( "socket" );
        exit( 1 );
    }

    sin.sin_family = hP->h_addrtype;
    memcpy( (char *)&sin.sin_addr, hP->h_addr, hP->h_length );
    sin.sin_port = htons( 80 );
    if ( connect( pfd, (struct sockaddr *)&sin, sizeof(sin) ) < 0 ) {
       perror( "connect" );
       close( pfd );
       exit( 1 );
    }

    sprintf( buf, "GET %s HTTP/1.0\r\n\r\n", fileP );
    write( pfd, buf, strlen(buf));

    while ( ( len = read( pfd, buf, sizeof(buf)) ) > 0)
       fwrite( buf, 1, len, stdout );

    close( pfd );
    fflush( stdout );
    exit( 0 );
}

----
Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

----- Original Message -----
From: "Mark E. Mallett" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, March 31, 2005 4:32 PM
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt

>
> On Thu, Mar 31, 2005 at 09:46:19AM -0800, Dave Crocker wrote:
> > >  I'm going to conclude that the problem is on
> > >  your end (do you perhaps have CRs embedded in the file on that
"Fedora"
> > >  Linux system instead of native (newline) line endings?) and move on.
> >
> >
> > As far as I can tell:
> >
> > 1. All of the components involved on my "side" of this are pretty
popular, so
> > serious problems would be expected to have emerged before this.
> >
> > 2. You are the only one having a problem. No one else has reported a
problem
> > with this document, nor with any other document I have generated and
made
> > available in a similar fashion.  I've been using these pieces for
something
> > like a year, so I would have expected problems to have been reported by
now.
>
> Well, I see the raw CR characters too, but I don't see it as a problem.
> I just edit them out.  I (too) suspect that Dave is storing the text
> file on bbiw.net in "wire format" and not in its native OS format.
>
> It's indeed up to the presentation client to work all this out and show
> you the correct text, but I guess some of us aren't fetching the document
> using a real viewer.  Both 'fetch' and 'wget' retain all of the original
> data including CRs -- I guess I wouldn't expect them to translate these
> things any more than I would expect them to format an HTML document.
>
>     % fetch http://bbiw.net/specifications/draft-crocker-email-arch-04.txt
>     % wget http://bbiw.net/specifications/draft-crocker-email-arch-04.txt
>
> don't do any presentation formatting, they just fetch the data.  However
> a real HTTP client has a better chance, so something like:
>
>     % lynx -dump
http://bbiw.net/specifications/draft-crocker-email-arch-04.txt > local-file
>
> will fetch and format for local presentation (and remove the extra CRs).
>
> And if you really don't trust anything and want to know for sure what's
> coming in on the wire, you can snoop with ethereal or tcpdump.  Or write
> a dumb little program such as the one below, and:
>
>     % cd /var/tmp
>     % cc -o f f.c
>     % ./f bbiw.net /specifications/draft-crocker-email-arch-04.txt > file
>
> and look at 'file' any way you want to.  The CRLFs are there on the wire
> and the receiving client may or may not be translating them, depending
> on whether it is supposed to.
>
> I hope this has beaten the topic to death :-)
>
> mm
>
> /*  f.c -- fetch via HTTP and dump the entire session to stdout
>     very stupidly.
> */
> #include <stdio.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netinet/in.h>
> #include <netdb.h>
> #include <signal.h>
>
> main( argc, argv )
>  int  argc;
>  char  **argv;
> {
>  int  pfd;  /* fd from socket */
>  int  len;
>  char  *hostP, *fileP;
>  char  buf[1024];
>  struct hostent *hP;  /* for host */
>  struct sockaddr_in sin;
>
>     if ( argc != 3 ) {
>  fprintf( stderr, "Usage: %s host file\n", argv[0] );
>  exit( 1 );
>     }
>     hostP = argv[1];
>     fileP = argv[2];
>
>     hP = gethostbyname( hostP );
>     if ( hP == NULL ) {
>  fprintf( stderr, "Unknown host \"%s\"\n", hostP );
>  exit( 1 );
>     }
>
>     pfd = socket( AF_INET, SOCK_STREAM, 0 );
>     if ( pfd < 0 ) {
>         perror( "socket" );
>  exit( 1 );
>     }
>
>     sin.sin_family = hP->h_addrtype;
>     memcpy( (char *)&sin.sin_addr, hP->h_addr, hP->h_length );
>     sin.sin_port = htons( 80 );
>     if ( connect( pfd, (struct sockaddr *)&sin, sizeof(sin) ) < 0 ) {
>  perror( "connect" );
>  close( pfd );
>  exit( 1 );
>     }
>
>     sprintf( buf, "GET %s HTTP/1.0\r\n\r\n", fileP );
>     write( pfd, buf, strlen(buf) );
>
>     while ( ( len = read( pfd, buf, sizeof(buf) ) ) > 0 )
>  fwrite( buf, 1, len, stdout );
>
>     close( pfd );
>     fflush( stdout );
>
>     exit( 0 );
> }
>
>

Cc: ietf-smtp <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 17:39:06 -0500
To: Dave Crocker <[email-masked]>
Sender: [email-masked]

> CRLF is the correct Internet method of indicating newline.  As I 
> understand
> HTTP, that convention applies to it, as well as the other 
> 'telnet-based'
> Internet protocols.
>
> So what is the problem you believe exists?

your server is doing the right thing according to the specifications.  
unfortunately, most HTTP clients don't properly handle the conversion 
from canonical format to native format when storing downloaded 
text/plain files.   they tend to display them reasonably, but not to 
store them properly.

many HTTP clients also botch when trying to print text/plain files, for 
instance by deleting or ignoring formfeeds.

Keith

From: [email-masked]
Cc: [email-masked]
Date: Thu, 31 Mar 2005 13:48:18 -0800 (PST)
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
To: Mark E Mallett <[email-masked]>
Sender: [email-masked]

> On Thu, Mar 31, 2005 at 09:46:19AM -0800, Dave Crocker wrote:
> > >  I'm going to conclude that the problem is on
> > >  your end (do you perhaps have CRs embedded in the file on that "Fedora"
> > >  Linux system instead of native (newline) line endings?) and move on.
> >
> >
> > As far as I can tell:
> >
> > 1. All of the components involved on my "side" of this are pretty popular, so
> > serious problems would be expected to have emerged before this.
> >
> > 2. You are the only one having a problem. No one else has reported a problem
> > with this document, nor with any other document I have generated and made
> > available in a similar fashion.  I've been using these pieces for something
> > like a year, so I would have expected problems to have been reported by now.

> Well, I see the raw CR characters too, but I don't see it as a problem.
> I just edit them out.  I (too) suspect that Dave is storing the text
> file on bbiw.net in "wire format" and not in its native OS format.

Or better still, use an editor like BBEdit that is line-terminator-smart and
just does the right thing with no muss and no fuss. BBEdit handles CRLF, LF,
and CR terminators with equal facility, and you can tell it to preserve line
terminators or force them to a particular sort on saves. I tried it on Dave's
draft and it handles the CRLFs fine, even though they aren't the Mac-native
CR-only format.

My one gripe with BBEdit is that it uses binary mode to store files via FTP.
This works fine when the line terminators choice you've made is workable on the
destination system, but fails badly when there's a mismatch. Mode A would fix
this in a lot of cases, but it isn't available. Grrr.

Another alternative is to use an OS that has filesystem level support for
different sorts of terminators. (OpenVMS supports 8-9 different terminator
styles, including but not limited to CR, LF, and CRLF.) Actually, the Mac is
approaching this given its mixed LF-only and CR-only ancestry.

				Ned

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 31 Mar 2005 16:32:25 -0500
To: [email-masked]
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Sender: [email-masked]

On Thu, Mar 31, 2005 at 09:46:19AM -0800, Dave Crocker wrote:
> >  I'm going to conclude that the problem is on
> >  your end (do you perhaps have CRs embedded in the file on that "Fedora"
> >  Linux system instead of native (newline) line endings?) and move on.
> 
> 
> As far as I can tell:
> 
> 1. All of the components involved on my "side" of this are pretty popular, so 
> serious problems would be expected to have emerged before this.
> 
> 2. You are the only one having a problem. No one else has reported a problem 
> with this document, nor with any other document I have generated and made 
> available in a similar fashion.  I've been using these pieces for something 
> like a year, so I would have expected problems to have been reported by now.

Well, I see the raw CR characters too, but I don't see it as a problem.
I just edit them out.  I (too) suspect that Dave is storing the text
file on bbiw.net in "wire format" and not in its native OS format.

It's indeed up to the presentation client to work all this out and show
you the correct text, but I guess some of us aren't fetching the document
using a real viewer.  Both 'fetch' and 'wget' retain all of the original
data including CRs -- I guess I wouldn't expect them to translate these
things any more than I would expect them to format an HTML document.

    % fetch http://bbiw.net/specifications/draft-crocker-email-arch-04.txt
    % wget http://bbiw.net/specifications/draft-crocker-email-arch-04.txt

don't do any presentation formatting, they just fetch the data.  However
a real HTTP client has a better chance, so something like:

    % lynx -dump http://bbiw.net/specifications/draft-crocker-email-arch-04.txt > local-file

will fetch and format for local presentation (and remove the extra CRs).

And if you really don't trust anything and want to know for sure what's
coming in on the wire, you can snoop with ethereal or tcpdump.  Or write
a dumb little program such as the one below, and:

    % cd /var/tmp
    % cc -o f f.c
    % ./f bbiw.net /specifications/draft-crocker-email-arch-04.txt > file

and look at 'file' any way you want to.  The CRLFs are there on the wire
and the receiving client may or may not be translating them, depending
on whether it is supposed to.

I hope this has beaten the topic to death :-)

mm

/*  f.c -- fetch via HTTP and dump the entire session to stdout
    very stupidly.
*/
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <signal.h>

main( argc, argv )
	int		argc;
	char		**argv;
{
	int		pfd;		/* fd from socket */
	int		len;
	char		*hostP, *fileP;
	char		buf[1024];
	struct hostent	*hP;		/* for host */
	struct sockaddr_in sin;

    if ( argc != 3 ) {
	fprintf( stderr, "Usage: %s host file\n", argv[0] );
	exit( 1 );
    }
    hostP = argv[1];
    fileP = argv[2];

    hP = gethostbyname( hostP );
    if ( hP == NULL ) {
	fprintf( stderr, "Unknown host \"%s\"\n", hostP );
	exit( 1 );
    }

    pfd = socket( AF_INET, SOCK_STREAM, 0 );
    if ( pfd < 0 ) {
        perror( "socket" );
	exit( 1 );
    }

    sin.sin_family = hP->h_addrtype;
    memcpy( (char *)&sin.sin_addr, hP->h_addr, hP->h_length );
    sin.sin_port = htons( 80 );
    if ( connect( pfd, (struct sockaddr *)&sin, sizeof(sin) ) < 0 ) {
	perror( "connect" );
	close( pfd );
	exit( 1 );
    }

    sprintf( buf, "GET %s HTTP/1.0\r\n\r\n", fileP );
    write( pfd, buf, strlen(buf) );

    while ( ( len = read( pfd, buf, sizeof(buf) ) ) > 0 )
	fwrite( buf, 1, len, stdout );

    close( pfd );
    fflush( stdout );

    exit( 0 );
}

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked], Dave Crocker <[email-masked]>
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 15:37:42 -0500
Sender: [email-masked]

On Thu March 31 2005 14:30, Dave Crocker wrote:
> 
> >  If you have a document with CRLF line endings on a UNIX or Linux system,
> >  it is likely to be transmitted over the wire with CRCRLF and stored
> >  (inappropriately) on other UNIX/Linux systems with CRLF endings.
> 
> If.
> 
> Yes, that is one of a number of plausible scenarios.

Supported by the difference in Content-Length reported by your server,
as mentioned this morning.

> However the fact that my windows system sees bare line-feeds from documents 
> sent by the ietf server suggest that, instead, bare linefeeds are being sent, 
> contrary to internet standards.

That would depend on whether you're using HTTP or FTP, and if FTP
whether transfer mode is set to type I or type A (RFC 959).

HTTP (as previously mentioned) lacks such fine control; RFC 1945
section 3.6.1 does not require CRLF on-the-wire (so one cannot
justifiably claim "contrary to internet standards" if HTTP is
used).

For all of the above reasons, FTP is preferable for transfer of
plain text (as type A transfers) where practical.

If you retrieve (for example)
ftp://ftp.isi.edu/in-notes/internet-drafts/draft-crocker-email-arch-04.txt
in ASCII mode on your "windows system" (in between BSODs), you should
have whatever local line endings you expect on that system (assuming
you're using an ftp client designed for that system).  If you transfer
in IMAGE mode, you should get precisely what is stored on the ISI.EDU
server (newline as line endings).

Use of FTP in IMAGE mode intentionally transfers content as stored on
the server, so again one cannot justifiably claim "contrary to internet
standards" in that case.

Now, if you're claiming that the ISI.EDU FTP server is not correctly
converting to NVT-ASCII (CRLF line endings) on-the-wire for type A
transfers, that's another matter (and I would dispute that claim, as I
just completed a test capturing packets with Ethereal and verified that
with type ascii ftp.isi.edu sends with CRLF line endings).

Bottom line: with all due respect, the "contrary to internet standards"
claim doesn't seem to hold water.

From: "Hector Santos" <[email-masked]>
To: "Dave Crocker" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 15:30:25 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: "Dave Crocker" <[email-masked]>
To: "Bruce Lilly" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, March 31, 2005 2:30 PM
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt

>
> >  If you have a document with CRLF line endings on a UNIX or Linux
system,
> >  it is likely to be transmitted over the wire with CRCRLF and stored
> >  (inappropriately) on other UNIX/Linux systems with CRLF endings.
>
> If.
>
> Yes, that is one of a number of plausible scenarios.
>
> However the fact that my windows system sees bare line-feeds from
documents
> sent by the ietf server suggest that, instead, bare linefeeds are being
sent,
> contrary to internet standards.
>
> So, at least one other plausible scenario is that some of us have software
> that treats crlf and lf equally and some of us do not.
>

No.  There is nothing wrong with your setup (as far as I can see).  It is
all about the interface.

Much depends on your UI (user interface).  <lf> can be translated to
<cr><lf> and vice versa.

When it comes to text file storage,  <lf> is traditionally a "unix platform"
concept.  <cr><lf> a DOS/Windows concept.

Often programmers need to deal with ideas such as "raw vs cooked" mode when
dealing with text based file I/O.  Raw means basically binary I/O.  No
translation.  Cooked means the EOL (end of lines) definition are translated.
The idea is to "cook it" for proper "printer based" output where a <CR>
literally means to move the printer head to the beginning of the line and
<LF> means move the head down one line (next line).  Of course, the console
evolved from a printer concept so the control characters concepts all apply.
The 80 (or 132) characters wide screen evolved from the 80/132 character
punch card, etc.

For Telnet, by default, it depends on how you have it setup.  Typically,
inbound no translation is done. On outbound, it is translated.  For example,
you hit the <ENTER> key, which is a <cr> (move printer head to begining of
line) character,  is sent along with a <lf> (move printer head down one
line).  But that depends on the negotiation.

For FTP,  the clients generallly needs to define the type of data transfer,
binary or text.

With that said, I see nothing wrong with your HTTP response. This is a GET
request and the WEB Server is responsible to provide the proper response.
HTTP requires CRLF delimiters.  It properly used CRLF delimited and in
addition, it properly performed MIME type association of the file extension.
In your case, TXT extension tells the web server (with a Mime Extension
Lookup) to respond with a Content-type: Text/Plain.

So the question might be ask. How is original file stored? and whether the
WEB server is reading the file in raw or cooked mode.  In addition, from the
client standpoint, is it going to do a translation.  A web client, for
example, will typically not do a translation.

In practice, atleast for a Windows based Web Server (like ours),  you read
the file "as is" - in raw mode and dump it out as so.  The reason is
straight forward.  In the HTTP response header, you need to add the
Content-Size: header. It would be extremely rare to have to pre-read the
entire file to count the number of delimiters in order to do an adjustment
to get the final output size.  That doesn't design well in a streaming
environment.

In any case, nothing new here.  The people who have a problem with the file
is most likely accessing it in cooked mode whether there might be a
translation.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

From: Dave Crocker <[email-masked]>
To: Bruce Lilly <[email-masked]>
CC: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 31 Mar 2005 11:30:47 -0800
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  If you have a document with CRLF line endings on a UNIX or Linux system,
>  it is likely to be transmitted over the wire with CRCRLF and stored
>  (inappropriately) on other UNIX/Linux systems with CRLF endings.

If.

Yes, that is one of a number of plausible scenarios.

However the fact that my windows system sees bare line-feeds from documents 
sent by the ietf server suggest that, instead, bare linefeeds are being sent, 
contrary to internet standards.

So, at least one other plausible scenario is that some of us have software 
that treats crlf and lf equally and some of us do not.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dave Crocker <[email-masked]>
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 14:25:19 -0500
Cc: [email-masked], Claus Assmann <[email-masked]>
Sender: [email-masked]

On Thu March 31 2005 13:09, Dave Crocker wrote:
> 
> Thanks, but I'm still not understanding.
> 
> CRLF is the correct Internet method of indicating newline.  As I understand 
> HTTP, that convention applies to it, as well as the other 'telnet-based' 
> Internet protocols.
> 
> So what is the problem you believe exists?

Perhaps ftp://ftp.isi.edu/in-notes/rfc-editor/instructions2authors.txt
section 3.1, subsection labeled "(2)", 2 paragraphs under heading
"Note:" (top of page 17) may help.

If you have a document with CRLF line endings on a UNIX or Linux
system, it is likely to be transmitted over the wire with CRCRLF and
stored (inappropriately) on other UNIX/Linux systems with CRLF endings.

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 31 Mar 2005 10:58:49 -0800
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  \r doesn't show up in other documents when they are downloaded, here are
>  two random examples:

Indeed, that IS interesting.

The problem is that bare linefeeds is not the internet standard for newline.

I mean, it's nice and all that, that your system (and mine and, apparently 
other folks') can handle bare linefeeds and interpret them as newline, but 
that is not the Internet Standard.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

Date: Thu, 31 Mar 2005 10:27:34 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Mail-Followup-To: Claus Assmann <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu, Mar 31, 2005, Dave Crocker wrote:

> CRLF is the correct Internet method of indicating newline.  As I understand
> HTTP, that convention applies to it, as well as the other 'telnet-based'
> Internet protocols.
> 
> So what is the problem you believe exists?

\r doesn't show up in other documents when they are downloaded,
here are two random examples:

$ wget http://www.ietf.org/internet-drafts/draft-housley-mass-sec-review-00.txt
$ od -c draft-housley-mass-sec-review-00.txt | head -1
0000000   \n  \n   N   e   t   w   o   r   k       W   o   r   k   i   n

$ wget http://www.ietf.org/rfc/rfc3597.txt
$ od -c rfc3597.txt | head -1
0000000   \n  \n  \n  \n  \n  \n   N   e   t   w   o   r   k       W   o

and here's one from your own server:

$ wget http://bbiw.net/specifications/draft-crocker-abnf-rfc2234bis-00.txt
$ od -c draft-crocker-abnf-rfc2234bis-00.txt | grep '\\r'
$ od -c draft-crocker-abnf-rfc2234bis-00.txt | head -1
0000000   \n  \n  \n  \n   N   e   t   w   o   r   k       W   o   r   k

PS: please don't append a full copy of my mail to your reply.
PPS: there's no need to Cc: me.

From: Dave Crocker <[email-masked]>
To: Claus Assmann <[email-masked]>, <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 31 Mar 2005 10:09:50 -0800
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Sender: [email-masked]

Thanks, but I'm still not understanding.

CRLF is the correct Internet method of indicating newline.  As I understand 
HTTP, that convention applies to it, as well as the other 'telnet-based' 
Internet protocols.

So what is the problem you believe exists?

d/

>  On Thu, Mar 31, 2005, Dave Crocker wrote:
>
> >  2. You are the only one having a problem. No one else has reported a
> >  problem with this document, nor with any other document I have
> >  generated and made
> >
>
>  Ok, here's another report:
>  $ ftp http://bbiw.net/specifications/draft-crocker-email-arch-04.txt $ od -
>  c draft-crocker-email-arch-04.txt | head -1 0000000   \r  \n  \r  \n  \r
>  \n   S   M   T   P
>  $ wget http://bbiw.net/specifications/draft-crocker-email-arch-04.txt $ od
>  -c draft-crocker-email-arch-04.txt | head -1 0000000   \r  \n  \r  \n  \r
>  \n   S   M   T   P
>
>  PS: that's on OpenBSD where "ftp" can deal with http (similar to "fetch"
>  on FreeBSD).

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

Date: Thu, 31 Mar 2005 10:02:28 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Mail-Followup-To: Claus Assmann <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu, Mar 31, 2005, Dave Crocker wrote:

> 2. You are the only one having a problem. No one else has reported a problem
> with this document, nor with any other document I have generated and made

Ok, here's another report:
$ ftp http://bbiw.net/specifications/draft-crocker-email-arch-04.txt
$ od -c draft-crocker-email-arch-04.txt | head -1
0000000   \r  \n  \r  \n  \r  \n   S   M   T   P
$ wget http://bbiw.net/specifications/draft-crocker-email-arch-04.txt
$ od -c draft-crocker-email-arch-04.txt | head -1
0000000   \r  \n  \r  \n  \r  \n   S   M   T   P

PS: that's on OpenBSD where "ftp" can deal with http (similar to
"fetch" on FreeBSD).

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 31 Mar 2005 09:53:08 -0800
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

> >  | In particular, can a Recipient's MUA automatically formulate | a
> >  valid Reply?
>
> >  ... that will reach the original Sender.
> >
>  Behind a gateway ?  Tricky.  Auto-Replies go to the address in the Return-
>  Path.  

Hmmm.  This is an interesting misunderstanding of what I am trying to have the 
document say, so the question is how to find better words.

This is intended to be about an MUA doing the "reply" command. The "reply" 
command automatically formulates a Reply.  It is supposed to use From and 
Reply-to.

I am not used to hearing anyone say that any sort of "reply" goes to the 
return-path.  Delivery and Failure notices, sure, but not "replies".

In any event, the idea that a delivered address would not work through a 
gateway sort of highlights why I characterize it as a test.

In other words, one of the goals of a gateway needs to be that it keeps 
addresses usable.

At least, that was the point I was/am trying to have the document make.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Dave Crocker <[email-masked]>
To: Bruce Lilly <[email-masked]>, <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 31 Mar 2005 09:46:19 -0800
Subject: Re: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  I'm going to conclude that the problem is on
>  your end (do you perhaps have CRs embedded in the file on that "Fedora"
>  Linux system instead of native (newline) line endings?) and move on.

As far as I can tell:

1. All of the components involved on my "side" of this are pretty popular, so 
serious problems would be expected to have emerged before this.

2. You are the only one having a problem. No one else has reported a problem 
with this document, nor with any other document I have generated and made 
available in a similar fashion.  I've been using these pieces for something 
like a year, so I would have expected problems to have been reported by now.

so I'm going to conclude that the problem is on YOUR end, absent more 
diagnostic information.

  d/

ps. this is all reminiscent of an aprocryphal -- but true -- anecdote i tell 
about the handling of interoperability problems with an Internet product I 
managed, 18 years ago.  I'll spare folks from the full recitation.  The simple 
summary is that I was on the receiving end of a customer assessment that 
looked a lot like my above analysis, except that there was absolutely no 
question that my product was conforming to the specs perfectly.
 
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Extraneous CRs in transfer of draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 08:13:42 -0500
Cc: Dave Crocker <[email-masked]>
Sender: [email-masked]

On Thu March 31 2005 00:14, Bruce Lilly wrote:

> I suspect the problem is that
> 1. HTTP (RFC 1945) doesn't have provision for conversion, unlike
>    FTP (STD 9), and you provide only an HTTP link.
> 2. (I forgot to mention) the document has non-ASCII cruft (RFC 2223)
>    in one of the references, causing the transfer to be treated as
>    binary content

Well, it's probably not those after all, as the official I-D is out and
retrieval from the http link in the ID-announce notice yielded a file
of 83751 octets, no CRs, but with different non-ASCII cruft in the
"Tussle" reference. [your Apache 2.0.51 server says
Content-Length: 85933]  I'm going to conclude that the problem is on
your end (do you perhaps have CRs embedded in the file on that "Fedora"
Linux system instead of native (newline) line endings?) and move on.

Date: Thu, 31 Mar 2005 13:11:52 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

Frank Ellermann writes:
> It's by far the most dangerous text I've seen for months, it tries to 
> standardize the abuse of MAIL FROM as arbitrary "bounces-to" (instead 
> of an entity responsible for problems with the SMTP transmission of 
> the message).

Could you elaborate on this? I agree that the two entities are different, but:

1) I don't see a big difference between them, and certainly not a very 
dangerous one

2) isn't "bounces-to" a fairly good description of how most running code 
treats the MAIL FROM address?

Arnt

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date:  Thu, 31 Mar 2005 11:55:42 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 34
Sender: [email-masked]

Bruce Lilly wrote:

> it is highly desirable to eliminate half-baked
> cockamamie schemes which are harmful to the Internet mail 
> architecture.

That would be the draft under discussion.  It's by far the most
dangerous text I've seen for months, it tries to standardize
the abuse of MAIL FROM as arbitrary "bounces-to" (instead of
an entity responsible for problems with the SMTP transmission
of the message).

> SPF is not an RFC. It is therefore not part of the standard
> Internet mail architecture per se.

It is certainly NOT a part of the crocker-mail-arch-04, and it
will never be.  In fact I hope that the real mail architecture
will turn away from this fatal concept a.s.a.p. before bounces
are disabled worldwide (bogus "bounces-to" are already a reason
for an entry in SpamCop's blocklist, now here's "harmful" if
you really want it, but not in SA or SPF).

> It is fatally flawed w.r.t. that architecture, as previously
> discussed here.

We totally disagree.  The crocker-mail-arch-04.txt is fatally
flawed.  With a "bounces-to" the _easy_ exit for all "innocent
bystanders" is to submit the bogus bounces to Spamcop, SC will
list the bouncing system, therefore the bouncing system will
disable its bounces, and SMTP minus bounces can't work, it's
too fragile without proper error messages.

                           Bye, Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date:  Thu, 31 Mar 2005 09:51:12 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 18
Sender: [email-masked]

Mark E. Mallett wrote:

>| In particular, can a Recipient's MUA automatically formulate
>| a valid Reply?

> ... that will reach the original Sender.

Behind a gateway ?  Tricky.  Auto-Replies go to the address in
the Return-Path.  In my parallel universe it's the responsible
sender, i.e. some local part @ gateway.

In Dave's parallel universe it could be an address _set_ by the
original sender, it's not necessarily related to the original
sender.  In fact most of the time it's an "innocent bystander",
the default case for mail is spam today.

                             Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Date: Thu, 31 Mar 2005 00:14:39 -0500
Cc: Dave Crocker <[email-masked]>
Sender: [email-masked]

On Wed March 30 2005 22:53, Dave Crocker wrote:

> CRLF is internet standard text line ending.  Sounds like your own software 
> isn't doing the conversion to local convention.

I suspect the problem is that
1. HTTP (RFC 1945) doesn't have provision for conversion, unlike
   FTP (STD 9), and you provide only an HTTP link.
2. (I forgot to mention) the document has non-ASCII cruft (RFC 2223)
   in one of the references, causing the transfer to be treated as
   binary content

> Do you have any idea how long both Strunk and While have been dead?

Yes. Do you think that's relevant?  If so, do you have any idea how
long Thackeray has been dead?
 
> And the re-casting suggestion is only made if you can't live with using 
> 'they'.  I can.

So if "Source" is plural, that should be "Source are...".

> > >  not sure what you think an attr/val pair is, if not that.
> > >
> >  For a start, a "pair" is two things...
> 
> 1. attribute
> 
> 2. value
> 
> 
> looks like a pair to me.

Except there *isn't* a "value" in the case of several fields, leaving
only an "attribute", which is no more a pair than a single penny is a
pair (of coins).

From: Dave Crocker <[email-masked]>
To: "Mark E. Mallett" <[email-masked]>
CC: ietf-smtp <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 30 Mar 2005 20:50:18 -0800
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  2.2.4  Gateway
>
> >  Differences between mail services can be as small as minor syntax
> >  variations, but usually encompass significant, semantic distinctions.
> >  For example, the concept of an email address might be as different as a
> >  hierarchical, machine-specific address versus a flat, global name space.
> >
>
>  I think you mean "concepts of an email address" not "the concept of"

well, this is an interesting one.  i'm inclined to keep it singular, because i 
see it as two variations on the same concept.

an alternative phrasing might be "their concepts of an email address", I 
suppose.

anyhow, i played with the text of the paragraph a bit.

>  4.3  Mail Submission Agent (MSA)
>
> >  RFC2821.HELO or RFC2821.EHLO
> >
> >  Set by: Source
> >
> >  The MSA may specify its hosting domain identity for the SMTP HELO or
> >  EHLO command operation.
> >
>
>  The HELO name is offered to the MSA, not specified by the MSA.  Maybe I
>  don't get what you mean here.

mumble.  two of you have commented on this.  i take the point, but REALLY 
don't want to have additional lists, one for client and the other for server.

>  5.  Mediators
>
> >  RFC2821.HELO or RFC2821.EHLO
> >
> >  Set by: Source or Relay
> >
> >  The MSA may specify its hosting domain identity for the SMTP HELO or
> >  EHLO command operation.
> >
>
>  Same comment as above, but I'd also add that this doesn't seem relevant
>  here.

well, there is so much community confusion about the different identifier 
fields, that i think it important to list the major ones for every entity 
(role) in the document.

>  5.1  Aliasing
>
> >  What is most distinctive about this forwarding mechanism is how closely
> >  it compares to normal MTA store-and-forward Relaying.
> >
>
>  This is the first mention that this might also be known as a "forwarding
>  mechanism" -- actually it's confusing because earlier in the draft there
>  has been other text describing various forwarding actions that don't
>  resemble this kind of aliasing.

well, this is one TYPE of forwarding...

>  Also, you might want to mention that this use of "aliasing" is different
>  from a fairly common use of that word to describe admin-level list
>  expansion (e.g. the /etc/aliases sendmail model). 

actually, I think it is not all that different.

 Unfortunately there's a
>  lot of overlap between words commonly in use, such as "alias" and
>  "forward" and "bounce" and "resend" and "redirect."  Me, I wouldn't use
>  "alias" here since it already has a lot of other meaning and I think this
>  use contradicts what a lot of people already think "alias" means.

the problem is the lack of crisp words that will cause no one any problems.

for example...

>  "Forwarding" is closer since there's the ".forward" model out there that

forwarding is highly ambiguous.  as in the mua forward command.  as in 
store-and-forward.  and so on.

again, thanks for the detailed review.  very helpful.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Dave Crocker <[email-masked]>
To: Bruce Lilly <[email-masked]>, <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 30 Mar 2005 19:53:24 -0800
Subject: Re: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  On Wed March 30 2005 20:19, Dave Crocker wrote:
>
> >  2. OD is carriage-return.  The newline sequence is OD-OA (CR-LF).  What
> >  do you believe is the problem with having carriage-returns in a
> >  document?
> >
>
>  Well, I haven't used an ASR-33 since the mid 1970's; my laptop doesn't
>  have a carriage and therefore doesn't need a "carriage return".

but internet character protocols often do.

>  Seriously, transfer protocols dealing with text are supposed to
>  appropriately convert line endings. I've never had a problem with the ftp
>  URIs in the ID-announce messages.

CRLF is internet standard text line ending.  Sounds like your own software 
isn't doing the conversion to local convention.

> > >  Section 2.2.1 also says: "Hence, Source is best held accountable for
> > >  the message content, even when they did not create any or most of
> > >  it". That appears to have disagreement in number between "source"
> > >  and "they" (see Strunk & White re. usage of "their").
> > >
> >
> >  See the "Gender Neutral" section, at <http://dcrocker.net>
> >
>  "Source" seems in this case to refer to a piece of software, so "it" would
>  seem appropriate.  Failing that, your first reference suggests recasting
>  the sentence, and the second reference agrees with Messrs. Strunk & White
>  that "he" is gender-neutral in such a context.

Do you have any idea how long both Strunk and While have been dead?

And the re-casting suggestion is only made if you can't live with using 
'they'.  I can.

> > >  Section 4.1.2 describes header fields as "attribute/value pairs",
> > >  which is technically incorrect (they are named fields).
> > >
> >
> >  not sure what you think an attr/val pair is, if not that.
> >
>  For a start, a "pair" is two things...

1. attribute

2. value

looks like a pair to me.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 22:34:03 -0500
Cc: Dave Crocker <[email-masked]>
Sender: [email-masked]

On Wed March 30 2005 20:19, Dave Crocker wrote:

> 2. OD is carriage-return.  The newline sequence is OD-OA (CR-LF).  What do you 
> believe is the problem with having carriage-returns in a document?

Well, I haven't used an ASR-33 since the mid 1970's; my laptop doesn't
have a carriage and therefore doesn't need a "carriage return".

Seriously, transfer protocols dealing with text are supposed to
appropriately convert line endings. I've never had a problem with
the ftp URIs in the ID-announce messages.

> >  Section 2.2.1 also says: "Hence, Source is best held accountable for the
> >  message content, even when they did not create any or most of it". That
> >  appears to have disagreement in number between "source" and "they" (see
> >  Strunk & White re. usage of "their").
> 
> See the "Gender Neutral" section, at <http://dcrocker.net>

"Source" seems in this case to refer to a piece of software, so "it"
would seem appropriate.  Failing that, your first reference suggests
recasting the sentence, and the second reference agrees with Messrs.
Strunk & White that "he" is gender-neutral in such a context.

> >  Section 4 purports to describe "functional components" under the heading
> >  "Services".  The first bullet item is "Message", which is neither a
> >  service nor a *functional* component, i.e. a message doesn't "do", it just
> >  "is".
> 
> You think that a discussion of functional components may not, also, discuss 
> the objects used by and for those components?

Discuss them by all means, but don't call them services or functional
components when they're not.

> >  Section 4.1.2 describes header fields as "attribute/value pairs", which is
> >  technically incorrect (they are named fields).
> 
> not sure what you think an attr/val pair is, if not that.

For a start, a "pair" is two things...  I suppose you might say that a
field is an "attribute" that might or might not have an associated
"value".  But one problem with "attribute" is that it means a
characteristic of something; whereas a header field is a component of
message content rather than a characteristic.  One problem with "value"
is that it implies a numerical quantity, which isn't applicable to most
header field bodies.  See RFC 2822 section 3.6 for possible alternative
text.

> yeah.  use of the term 'header' is actually variable among the email 
> standards, but i did, in fact, mean to use header field.

OK, but note that the registry registers just field *names*, not the
entire field.

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 30 Mar 2005 22:15:14 -0500
To: Dave Crocker <[email-masked]>
Cc: ietf-smtp <[email-masked]>
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

Sorry if there's some duplicate stuff in here.  I did read the entire
thread so far, but didn't memorize it.  Without trying to nitpick
too much, here are some comments.

2.2.4  Gateway

   > Differences between mail services can be as small as minor syntax
   > variations, but usually encompass significant, semantic distinctions.
   > For example, the concept of an email address might be as different as
   > a hierarchical, machine-specific address versus a flat, global name
   > space.

I think you mean "concepts of an email address" not "the concept of"

But then:

   > Or between text-only content and multi-media.  

Here I think you must be referring to "differences between mail
services" and not to "concepts of email addresses" -- because if it's
the latter it really doesn't make sense.  If it does make sense, maybe
it needs to be said differently.

   > In particular, can a Recipient's
   > MUA automatically formulate a valid Reply?

... that will reach the original Sender.   (the reply can certainly
be formulated; the word "valid" doesn't really imply that it will go
to the sender..)

3.1  Mailbox Addresses

   > A mailbox is specified as an Internet Mail address <addr-spec>.  It
   > has two distinct parts, divided by an at-sign ("@").  The right-hand
   > side contains a globally interpreted name for an Administrative Unit.
   > Domain Names are discussed in Section 3.2.

This is better than before, but I still think it's funky.  The right
side of an Internet Mail address is an Internet Domain Name.  That
domain name does directly lead to the "Administrative Unit" but it
really isn't the name for the AU.  There can be a many-to-one mapping
of domain names to AU, but this does not make one a synonym for
the other.  JMHO.

This is compatible with the mention in 3.3 that the right-hand side of a
message-id specifies the AU assigning the id.  (Even that's a a bit
inexact: however it's better, as it merely says that this "specifies"
the AU and isn't the "name for" the AU -- again the difference is
between a mapping-to and a name-of).

4.1  Message

   > Internet Mail has distinguished some special versions of messages,
   > for exchanging control information:

    ... 

   > Message Filtering (SIEVE):

Other controversies about Sieve aside, this is not a special version of
a message, and doesn't belong in this list.  I suppose it could be a
Relay or Mediator function.

4.3  Mail Submission Agent (MSA)

   > RFC2821.HELO or RFC2821.EHLO
   > 
   >    Set by: Source
   > 
   >    The MSA may specify its hosting domain identity for the SMTP HELO
   >    or EHLO command operation.

The HELO name is offered to the MSA, not specified by the MSA.  Maybe
I don't get what you mean here.

(repeat this for 4.4 also)

5.  Mediators

   > RFC2821.HELO or RFC2821.EHLO
   > 
   >    Set by: Source or Relay
   > 
   > The MSA may specify its hosting domain identity for the SMTP HELO
   > or EHLO command operation.

Same comment as above, but I'd also add that this doesn't seem
relevant here.

5.1  Aliasing

   > What is most distinctive about this forwarding mechanism is how
   > closely it compares to normal MTA store-and-forward Relaying.

This is the first mention that this might also be known as a "forwarding
mechanism" -- actually it's confusing because earlier in the draft there
has been other text describing various forwarding actions that don't
resemble this kind of aliasing.  

Also, you might want to mention that this use of "aliasing" is different
from a fairly common use of that word to describe admin-level list
expansion (e.g. the /etc/aliases sendmail model).  Unfortunately there's
a lot of overlap between words commonly in use, such as "alias" and
"forward" and "bounce" and "resend" and "redirect."  Me, I wouldn't use
"alias" here since it already has a lot of other meaning and I think
this use contradicts what a lot of people already think "alias" means.
"Forwarding" is closer since there's the ".forward" model out there that
you seem to be describing here, except that the word "forwarding"
implies different things in different contexts.  Actually your
"re-address" term is a good way to think of it, too.

5.2  ReSending

   > Also called ReDirecting, ReSending differs from Forwarding by virtue
   > of having the Mediator "splice" a message's addressing information,

Mention of "Forwarding" without prior establishment of that term.

mm

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 30 Mar 2005 17:19:15 -0800
Subject: Re: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Sender: [email-masked]

Bruce,

Thanks for the details. Just the sort of careful proof-reading the document 
needs.

Comments, below, when there is an issue:

> >  <http://bbiw.net/current.html#email>
> >
>  The server either does not recognize the "text" version as text or fails
>  to honor text transfer protocols; I retrieved a document which, unlike I-
>  Ds retrieved from the official process, contained thousands of spurious
>  control characters (0x0D).

1. It works fine with Internet Explorer and with Firefox.  

2. OD is carriage-return.  The newline sequence is OD-OA (CR-LF).  What do you 
believe is the problem with having carriage-returns in a document?

Feel free to provide more detail, especially if anyone else is experiencing 
problems.

>  "SMTP" in the first page heading should be "Network Working Group" (see
>  RFC 2223).

The document is produced by xml2rfc and I try to use it carefully. It is 
popular enough and behaves well enough that I am not likely to try to change 
such things, particularly when I think they are helpful.

>  Section 2.2.1 also says: "Hence, Source is best held accountable for the
>  message content, even when they did not create any or most of it". That
>  appears to have disagreement in number between "source" and "they" (see
>  Strunk & White re. usage of "their").

See the "Gender Neutral" section, at <http://dcrocker.net>

>  Given the fact that section 2.2.3 mentions store-and-forward, changing
>  "(re)-transmits" to "forwards" would clarify the operation.

except that forward, by itself, is at best ambiguous, given its uses for 
discussion of mediators.  

>  In section 3.2 and many subsequent sections, "Address" in "IP Address" is
>  unnecessarily capitalized.

It's quite intentional.  The multiaddressing work has made the term "address" 
generally problematic to use, particularly on its own.  The preferred terms 
are locator and identifier.  Hence, using the word "address" makes sense only 
when it is part of a formal term.  That's why I capitalized it.

>  Section 4 purports to describe "functional components" under the heading
>  "Services".  The first bullet item is "Message", which is neither a
>  service nor a *functional* component, i.e. a message doesn't "do", it just
>  "is".

You think that a discussion of functional components may not, also, discuss 
the objects used by and for those components?

>  Section 4.1.2 describes header fields as "attribute/value pairs", which is
>  technically incorrect (they are named fields).

not sure what you think an attr/val pair is, if not that.

 For example, the Bcc field
>  may have no field body (a.k.a. "value"); 

null value.

>  The same section also destroys consistency of technical term usage by
>  using "registering headers" where it should say "registering header field
>  names".

yeah.  use of the term 'header' is actually variable among the email 
standards, but i did, in fact, mean to use header field.

>  The section 5.3 discussion of "RFC2919.List-id" [capitalization sic]
>  states "Although [RFC2919] is a standards-track specification, it has not
>  gained significant adoption." I note that the (ahem) ietf-smtp list (to
>  name but one example) uses the List-ID field. I doubt the wisdom of making
>  such value judgments in a document intended to be long-lived, and I
>  suggest eliding that sentence.

The IETF does not provide a very representative sampling basis for 
generalizing global Internet practise.  That said, however, I agree that a 
document like this should be very wary (or vary wery) of such text.

>  "Acknowledgements" (Appendix A) is usually spelled "Acknowledgments" in
>  RFCs (and by many spelling checkers).

I believe this issue received plenty of it's own discussion, today.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked], Dave Crocker <[email-masked]>
Subject: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 17:04:10 -0500
Cc: ietf-smtp <[email-masked]>
Sender: [email-masked]

On Tue March 29 2005 11:23, Dave Crocker wrote:

> Until the draft is issued through the Internet Drafts process, it can be
> accessed at:
> 
>   <http://bbiw.net/current.html#email>

The server either does not recognize the "text" version as text or
fails to honor text transfer protocols; I retrieved a document which,
unlike I-Ds retrieved from the official process, contained thousands
of spurious control characters (0x0D).

Anyway, here are some comments that do not delve into complex
substantive issues.  I expect to comment on such issues in a series
of separate messages.

"SMTP" in the first page heading should be "Network Working Group"
(see RFC 2223).

The document uses out-of-date boilerplate; refer to RFCs 3978/3979.

The Abstract (and Section 1 from which it was apparently extracted)
contains "marked by many independent operators, many different
components for providing users service and many others for performing
message transfer" which is somewhat awkward.  I suggest "... providing
service to users and many other components for ...".

Section 1.3 says "Disp" which isn't a word and which appears nowhere
in the text. It appears as a label in a box in one figure. Surely there
is room for a more descriptive label.

Section 1.3 also states that "Administrative Domain" was changed to
"Administrative Unit", however section 2.2.1 appears not to have been
updated.

Section 2.2.1 also says: "Hence, Source is best held accountable for the
message content, even when they did not create any or most of it". That
appears to have disagreement in number between "source" and "they" (see
Strunk & White re. usage of "their").

Given the fact that section 2.2.3 mentions store-and-forward, changing
"(re)-transmits" to "forwards" would clarify the operation.

Section 3.1 refers to and speaks of "conventions", but the section
heading instead says "Standards". I suggest changing "Standards" in the
heading to "Conventions".

In section 3.2 and many subsequent sections, "Address" in "IP Address"
is unnecessarily capitalized.

Section 4 purports to describe "functional components" under the
heading "Services".  The first bullet item is "Message", which is
neither a service nor a *functional* component, i.e. a message doesn't
"do", it just "is".

Figure 5 contains the line
         .           |      +---- -------------------+         |
which appears to have a typographical error (gap in horizontal line).

Description of MDN states "indicating that the message has been read",
which is not quite correct.  Changing "read" to "presented" would fix
the problem.

Following that MDN description under the heading "Messages" and
following the introductory line "Internet Mail has distinguished some
special versions of messages, for exchanging control information:" is
a subsection headed "Message Filtering (SIEVE)". Sieve is not a
message at all, much less a special version of a message.

Section 4.1.2 describes header fields as "attribute/value pairs",
which is technically incorrect (they are named fields). For example,
the Bcc field may have no field body (a.k.a. "value"); likewise for
all unstructured fields (Subject, Comments, Content-Disposition, etc.).
The header fields Incomplete-Copy, Generate-Delivery-Report, and
Prevent-Nondelivery-Report are required to have no field body (RFC
2156).

The same section also destroys consistency of technical term usage by
using "registering headers" where it should say "registering header
field names".

Section 4.1.3 refers to RFC 2048, which is the MIME media type
registration procedure, which is unrelated to the Internet mail
architecture (it's a registration procedure!).  RFC 2048 is about to
be superseded (draft-freed-media-type-reg-02.txt is in Last Call).
RFC 2048 can probably be dropped from the architecture draft text and
references.

The word "or" in the last line of section 4.2 should be "of".

The third paragraph of section 4.3 says "it operates on a separate TCP
port, and will typically impose". That should be "it may operate on a
separate TCP port and may impose".  And I would add that a client oMUA
cannot reliably distinguish an MSA server from an MTA server.

The same section states:
   RFC2821.HELO or RFC2821.EHLO

      Set by: Source

      The MSA may specify its hosting domain identity for the SMTP HELO
      or EHLO command operation.
1. several subsequent sections mention only "RFC2821.HELO" but not EHLO.
2. while an MSA does indeed set HELO/EHLO domain when operating as an
   SMTP client, the oMUA sets the HELO/EHLO domain when communicating to
   the MSA when the latter is acting as a server.

Section 4.4, third paragraph, second line has a superfluous comma after
"host". Same section, fourth paragraph, first line has superfluous
commas before and after "over the open Internet" and after
"prearrangement" and "participants".

The comma after "mechanism" in the second paragraph of section 4.5
should be replaced by a colon.

Commas in the first two lines of the last paragraph beginning at the
bottom of page 27 are superfluous.

The apostrophe in "it's" (meaning a contraction for "it is") is
incorrect near the top of page 28 (use "its").

The section 5.3 discussion of "RFC2919.List-id" [capitalization sic]
states "Although [RFC2919] is a standards-track specification, it has
not gained significant adoption." I note that the (ahem) ietf-smtp
list (to name but one example) uses the List-ID field. I doubt the
wisdom of making such value judgments in a document intended to be
long-lived, and I suggest eliding that sentence.

The last paragraph of section 5.3 contains a conflict in number between
"fields" and "it" (change "it" to "them").

RFC 2298 has been obsoleted by RFC 3798 (see the "rfc-ref.txt" file
made available by the RFC Editor at
ftp://ftp.rfc-editor.org/in-notes/rfc-ref.txt
).

"Acknowledgements" (Appendix A) is usually spelled "Acknowledgments" in
RFCs (and by many spelling checkers).

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked], Dave Crocker <[email-masked]>
Subject: Editorial/typographical/grammatical/punctuation/usage/formatting/charset/simple comments on draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 17:04:10 -0500
Cc: ietf-smtp <[email-masked]>
Sender: [email-masked]

On Tue March 29 2005 11:23, Dave Crocker wrote:

> Until the draft is issued through the Internet Drafts process, it can be
> accessed at:
> 
>   <http://bbiw.net/current.html#email>

The server either does not recognize the "text" version as text or
fails to honor text transfer protocols; I retrieved a document which,
unlike I-Ds retrieved from the official process, contained thousands
of spurious control characters (0x0D).

Anyway, here are some comments that do not delve into complex
substantive issues.  I expect to comment on such issues in a series
of separate messages.

"SMTP" in the first page heading should be "Network Working Group"
(see RFC 2223).

The document uses out-of-date boilerplate; refer to RFCs 3978/3979.

The Abstract (and Section 1 from which it was apparently extracted)
contains "marked by many independent operators, many different
components for providing users service and many others for performing
message transfer" which is somewhat awkward.  I suggest "... providing
service to users and many other components for ...".

Section 1.3 says "Disp" which isn't a word and which appears nowhere
in the text. It appears as a label in a box in one figure. Surely there
is room for a more descriptive label.

Section 1.3 also states that "Administrative Domain" was changed to
"Administrative Unit", however section 2.2.1 appears not to have been
updated.

Section 2.2.1 also says: "Hence, Source is best held accountable for the
message content, even when they did not create any or most of it". That
appears to have disagreement in number between "source" and "they" (see
Strunk & White re. usage of "their").

Given the fact that section 2.2.3 mentions store-and-forward, changing
"(re)-transmits" to "forwards" would clarify the operation.

Section 3.1 refers to and speaks of "conventions", but the section
heading instead says "Standards". I suggest changing "Standards" in the
heading to "Conventions".

In section 3.2 and many subsequent sections, "Address" in "IP Address"
is unnecessarily capitalized.

Section 4 purports to describe "functional components" under the
heading "Services".  The first bullet item is "Message", which is
neither a service nor a *functional* component, i.e. a message doesn't
"do", it just "is".

Figure 5 contains the line
         .           |      +---- -------------------+         |
which appears to have a typographical error (gap in horizontal line).

Description of MDN states "indicating that the message has been read",
which is not quite correct.  Changing "read" to "presented" would fix
the problem.

Following that MDN description under the heading "Messages" and
following the introductory line "Internet Mail has distinguished some
special versions of messages, for exchanging control information:" is
a subsection headed "Message Filtering (SIEVE)". Sieve is not a
message at all, much less a special version of a message.

Section 4.1.2 describes header fields as "attribute/value pairs",
which is technically incorrect (they are named fields). For example,
the Bcc field may have no field body (a.k.a. "value"); likewise for
all unstructured fields (Subject, Comments, Content-Disposition, etc.).
The header fields Incomplete-Copy, Generate-Delivery-Report, and
Prevent-Nondelivery-Report are required to have no field body (RFC
2156).

The same section also destroys consistency of technical term usage by
using "registering headers" where it should say "registering header
field names".

Section 4.1.3 refers to RFC 2048, which is the MIME media type
registration procedure, which is unrelated to the Internet mail
architecture (it's a registration procedure!).  RFC 2048 is about to
be superseded (draft-freed-media-type-reg-02.txt is in Last Call).
RFC 2048 can probably be dropped from the architecture draft text and
references.

The word "or" in the last line of section 4.2 should be "of".

The third paragraph of section 4.3 says "it operates on a separate TCP
port, and will typically impose". That should be "it may operate on a
separate TCP port and may impose".  And I would add that a client oMUA
cannot reliably distinguish an MSA server from an MTA server.

The same section states:
   RFC2821.HELO or RFC2821.EHLO

      Set by: Source

      The MSA may specify its hosting domain identity for the SMTP HELO
      or EHLO command operation.
1. several subsequent sections mention only "RFC2821.HELO" but not EHLO.
2. while an MSA does indeed set HELO/EHLO domain when operating as an
   SMTP client, the oMUA sets the HELO/EHLO domain when communicating to
   the MSA when the latter is acting as a server.

Section 4.4, third paragraph, second line has a superfluous comma after
"host". Same section, fourth paragraph, first line has superfluous
commas before and after "over the open Internet" and after
"prearrangement" and "participants".

The comma after "mechanism" in the second paragraph of section 4.5
should be replaced by a colon.

Commas in the first two lines of the last paragraph beginning at the
bottom of page 27 are superfluous.

The apostrophe in "it's" (meaning a contraction for "it is") is
incorrect near the top of page 28 (use "its").

The section 5.3 discussion of "RFC2919.List-id" [capitalization sic]
states "Although [RFC2919] is a standards-track specification, it has
not gained significant adoption." I note that the (ahem) ietf-smtp
list (to name but one example) uses the List-ID field. I doubt the
wisdom of making such value judgments in a document intended to be
long-lived, and I suggest eliding that sentence.

The last paragraph of section 5.3 contains a conflict in number between
"fields" and "it" (change "it" to "them").

RFC 2298 has been obsoleted by RFC 3798 (see the "rfc-ref.txt" file
made available by the RFC Editor at
ftp://ftp.rfc-editor.org/in-notes/rfc-ref.txt
).

"Acknowledgements" (Appendix A) is usually spelled "Acknowledgments" in
RFCs (and by many spelling checkers).

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 15:54:15 -0500
Sender: [email-masked]

On Wed March 30 2005 13:21, Hector Santos wrote:
> 
> ----- Original Message ----- 
> From: "Bruce Lilly" <[email-masked]>
> To: <[email-masked]>
> Sent: Wednesday, March 30, 2005 11:13 AM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
> 
> 
> 
> >> Suggestion:
> >> 
> >> - Change the Sieve box to MFA  (Mail Filtering Agent).
> 
> > While that might be a generalization worthy of due consideration,
> 
> I'm glad you agree.

Please consult a dictionary for the meanings of the words "might",
"due", and "consideration".  Agreement was neither stated nor implied.
There was also a further qualification, viz. that such a change is
contingent upon a list of "other" Standards Track "mail filtering"
documents, a condition that has not been (and evidently will not be)
fulfilled.

From: Dave Crocker <[email-masked]>
To: Bruce Lilly <[email-masked]>, <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 30 Mar 2005 10:53:11 -0800
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

> > >  - Change the Sieve box to MFA  (Mail Filtering Agent).
> > >
> >  While that might be a generalization worthy of due consideration,

For consistency, that means changing the left-hand-side Message reference to 
say something generic, like "Message" and the MDN box to something like 
Recipient Notice, and the DSN box to something like Handling Notice.

Oh, wait.  The text says "This section describes the specific functional 
components for Internet Mail, and the standard protocols associated with 
performing them."  

So, references to standards is explicitly the purpose of the section.

And, by the way, the Sieve box is an object, not an agent.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 13:21:07 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message ----- 
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, March 30, 2005 11:13 AM
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt

>> Suggestion:
>> 
>> - Change the Sieve box to MFA  (Mail Filtering Agent).

> While that might be a generalization worthy of due consideration,

I'm glad you agree.

Everything else is well...nonsense.

Sincerely, 

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 13:18:00 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Wednesday, March 30, 2005 10:34 AM
Subject: Re: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt

> > Suggestion:
> >
> > - Change the Sieve box to MFA (Mail Filtering Agent).
> > - Reference Sieve parentheically in the document, such like others are
(i,e.
> > POP, IMAP, etc).

> If you have an other name for the Sieve Message (Filter Script message?
FSM) I
> think we are in agreement on that as well. (we do not talk about POP3,
ODMR
> and IMAP emails either)

Whatever.

I believe the document benefits by generalizing it.  Sieve may be a RFC
standard, but certainly should not be viewed or construed as the required
component for implementing a mail filtering system in a GIEF.

Thats it!  No mas!

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 30 Mar 2005 09:22:56 -0800
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  Indeed, although "must" appears
>  many times in the draft, it is unclear whether the draft (when published
>  as an RFC) would be Informational or Standards Track; if the latter, RFC

I intend to seek Proposed Standard status.  I did a review of other RFCs that 
cover architecture and it is quite common for them to be on standards track.

Hence, yes, we (I) need to be very careful about the document's use of the 
magic normative vocabulary.  Thanks for bringing this up.

I've just gone through the draft and checked all occurrences of the magic 
words.  Most seem ok, to me, though I capitalized them to emphasize their 
normative nature.  A few others have been changed to use non-magic wording, 
especially when the normative specification for the item being discussed is 
actually contained in an existing standards specification.  

(An example is whether a Received header must/should/may be recorded; this is 
specificed in RFC2821 and I would rather not have the email-arch draft 
replicate such statements, since it invites loss of synchrony among the 
documents.)

>  2119 should probably be specified as a normative reference with
>  appropriate use of MUST, SHOULD, etc. to clarify requirements,

ack.

>  SpamAssassin is not an RFC.  It is therefore not part of the standard
>  Internet mail architecture per se.

Even better:  It also is an implementation, rather than an archtecture.  The 
email-arch draft reminds folks to be careful about that distinction.

> >  Can it
> >  implement SPF for example?
> >
>  SPF is not an RFC. It is therefore not part of the standard Internet mail
>  architecture per se.  It is fatally flawed w.r.t. that architecture, as
>  previously discussed here.

If it WERE documented (and standardized), should it be in the email-arch 
document?  My feeling is no, in that there are many things left out of the 
document and these sorts of evaluation mechanisms are, so far, a long way from 
being core to the service.  Reference to filtering is the one concession to 
the emerging area of standardized, remote specification of specialized 
handling.  (along with the conneg stuff.) 

The document tries to describe the core, modern Internet Mail service.  At 
that, it describes a remarkably complicated service.  It would be better to 
find ways to simplify it, to improve its use as a framework for technical 
discussion.  (That is what I view an architecture document as seeking to 
accomplish, more than anything.)

So my own view is that the goal is to leave OUT as much detail as we can 
tolerate.  

>  Both DSN and MDN are based on the RFC 3462 (nee 1892) MIME
>  multipart/report media type.  MIME itself is not a necessary part of the
>  Internet mail architecture or Internet Message Format; as noted in the
>  MIME RFCs, MIME is "largely orthogonal to" the Internet Message Format.

Here is where the subjective assessment comes in, in terms of what to include 
and what to exclude from the architecture document:  Yes, individual messages 
and, perhaps, even individual implementations, can operate without supporting 
MIME, even today.  However the global service certainly relies on its 
presence.  So I believe that MIME is not orthogonal or optional in an Internet 
Mail architecture discussion.

>  One might quibble about some aspects of its inclusion in draft figure 5
>  (the gap in one of the lines, the fact that Sieve can be implemented
>  entirely in the rMUA or entirely in the MDA, etc.), but it is a widely
>  used Standards Track part of the architecture and should not be removed
>  from the draft.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 11:13:17 -0500
Sender: [email-masked]

On Wed March 30 2005 09:32, Hector Santos wrote:

> Look, when not just throw in IMAP in the picture.  Its a RFC too.

It IS (appropriately) in draft figure 5. 

> I am not asking for the removal of Sieve but to change the wording around to
> point out very clearly it is not a requirement.  As it is worded, it makes
> it seem to current or future newcomers that Sieve is a natural part of the
> process, which it is clearly not.

I see nothing that states a requirement. Indeed, although "must" appears
many times in the draft, it is unclear whether the draft (when published
as an RFC) would be Informational or Standards Track; if the latter, RFC
2119 should probably be specified as a normative reference with
appropriate use of MUST, SHOULD, etc. to clarify requirements,
recommendations, etc.  If the draft is to become merely an Informational
RFC, it is meaningless to speak of requirements (though the hypothetical
Informational RFC might appropriately DOCUMENT requirements imposed by
Standards Track RFCs).

> First,  it is clearly not a generalized technology, but a specific
> technology that is clearly not required.

>From the (Standards Track) RFC 3028 Abstract:

   It is meant to be extensible, simple,
   and independent of access protocol, mail architecture, and operating
   system. 

Sounds "generalized" to me.  And it is part of the Standards Track
Internet mail architecture (which, as noted in the quote above, does
not preclude its use in non-Internet mail architectures).

> Second, it is a specific language that upon itself imposes specific
> installation of specialized software.

It is, as RFC 3028's title states, a Mail Filtering Language. ANY
mail filtering language is necessarily a language and specific to
mail filtering.  You have stated a tautology; so what?
 
> Third, there are a vast degree of different integrated frameworks that use
> different "message filter" hooks.  Spam Assassin is probably the most
> popular,  and there are many more mail filtering "hooks."

SpamAssassin is not an RFC.  It is therefore not part of the standard
Internet mail architecture per se.
 
> Forth, you certainly do not wish to minimize new ideas because of the
> erroneous notion that it must be based on Sieve.

On the contrary, it is highly desirable to eliminate half-baked
cockamamie schemes which are harmful to the Internet mail architecture.
Any "erroneous notion" that a specific filtering method "must be based
on Sieve" is an error on the part of the parson making such an
assumption.  On the other hand, given the fact of the "extensible,
simple, and independent of access protocol, mail architecture, and
operating system" standardized mail filtering language, any new method
that ignores Sieve is likely to be a niche method at best.

> In fact, I don't believe 
> Sieve has yet to incorporate DNS based security functionality.

The beliefs of somebody who has apparently not read the first two
sentences of RFC 3028 section 2.7.4 are irrelevant.

> Can it 
> implement SPF for example?

SPF is not an RFC. It is therefore not part of the standard Internet
mail architecture per se.  It is fatally flawed w.r.t. that
architecture, as previously discussed here.

[...]
> But we know for sure there are other "mail filtering" systems that do 
> offer DNS based lookup methods.

As specified in which Standards Track RFCs, specifically?
 
> Finally, Dave brought up the question, what about DSN and MDN.
> 
> Same basic issues. It should be pointed out they are not requirements to be
> part of the IEF.  But DSN and MDN are more well defined, very easy,
> generalized structural issues that fit cleanly into the automated and
> generalize framework.

Both DSN and MDN are based on the RFC 3462 (nee 1892) MIME
multipart/report media type.  MIME itself is not a necessary part of
the Internet mail architecture or Internet Message Format; as noted in
the MIME RFCs, MIME is "largely orthogonal to" the Internet Message
Format.  Moreover, DSNs depend on ESMTP (won't work with vanilla SMTP)
and a specific ESMTP extension.  A case could certainly be made that
therefore neither DSN nor MDN are integral parts of the Internet mail
architecture because of their reliance on the separate MIME
specifications.  In practice, however, I see nothing wrong with their
inclusion, as both are based on Standards Track mail-related RFCs.

> Sieve, on the other hand, is a special and specific computer language that
> requires a specific set of compilers and/or p-code interpreters, etc.

Nonsense.  No compiler or "p-code" is required for Sieve.  One
particular implementation may compile Sieve filters, but that is an
implementation issue, not a characteristic nor requirement of the Mail
Filtering Language.

> It 
> requires a set of tertiary tools, such as MYSQL or some other requirement.

More nonsense.

> If you don't have all the requirements, you have to write it yourself.  In
> fact, I believe there might be just one Windows implementation of it,
> http://www.cyrusoft.com/sieve/.

So what?  There is no requirement that any specific part of the
Internet mail architecture be implemented on a particular "operating"
"system" (or non-operating collection of cruft, as the case may be).
 
> DSN and MDN do not require special "compilers" or "p-code!"

Neither does Sieve. DSN and MDN require MIME, though, and Sieve does not
(scripts that test MIME parts do, but that is not a requirement imposed
by Sieve per se).  DNSs require a specific ESMTP extension, Sieve does
not.
 
[...]
> DSN is based on a generalized bounce requirement that is a natural part of
> the IEF.  DSN is an optional and specific standard implementation of the
> requirement.
> 
> MDN is based on a RFC 2822 header existence. Also a natural part of a IEF.

You are mistaken. Both DSN and MDN are based on MIME multipart/report.
DSNs are based on a specific ESMTP extension.
 
> On the other hand:
> 
> Sieve is based on a Mail Filtering concept that is _not_ a requirement to be
> part of the IEF process.

Filtering is not a requirement.

> You can pull the "SIEVE" box, the message 
> filtering concept, out from figure 5 and you still have a generalized IEF
> process that is widely in place.  You can not do the same with Bounce (DSN)
> or return receipt (MDN) concepts without breaking the framework.

Sure one can. Indeed, unless MIME (and in the case of DSN, DSN-specific
ESMTP extensions) is fully supported end-to-end, neither DSNs nor MDNs
are usable.  Anything removed, however, results in failure to document
parts of the de jure Standards Track mail architecture.  Just because
some part(s) of that architecture are unimplemented on some specific
platforms or on some parts of networks that are connected to the Internet
is not a justification for removal from the draft documenting the mail
architecture which is comprised of many Standards Track RFCs. 
 
> Suggestion:
> 
> - Change the Sieve box to MFA  (Mail Filtering Agent).

While that might be a generalization worthy of due consideration,
barring specific references to existing Standards Track RFCs or drafts
which are likely to become Standards Track RFCs for other Internet
mail filtering methods, I see no reason to do so -- Sieve is THE
mail filtering language for the INTERNET mail architecture; it is
widely implemented, including by cross-platform MUAs (e.g. Mulberry).

One might quibble about some aspects of its inclusion in draft figure 5
(the gap in one of the lines, the fact that Sieve can be implemented
entirely in the rMUA or entirely in the MDA, etc.), but it is a widely
used Standards Track part of the architecture and should not be removed
from the draft.

To: Hector Santos <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
From: [email-masked]
Subject: Re: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 16:34:07 +0100
Sender: [email-masked]

O I think our messages crossed eachother.
I was writing my reply while yours was comming in sorry.

Suggestion:
> 
> - Change the Sieve box to MFA  (Mail Filtering Agent).
> - Reference Sieve parentheically in the document, such like others are (i,e.
> POP, IMAP, etc).

MFA box  I am not sure it does not have a specific place 
But i fully agree with changing the name for the service I mentioned to MFA (Mail Filtering Agent).

If you have an other name for the Sieve Message (Filter Script message? FSM) i think we are in agreement on that as well. (we do not talk about POP3, ODMR  and IMAP emails either)

> Clarification:
> 
> ----- Original Message -----
> From: "Hector Santos" <[email-masked]>
> To: <[email-masked]>
> Cc: "IETF-SMTP" <[email-masked]>
> Sent: Wednesday, March 30, 2005 8:32 AM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
> 
> 
> >
> > Finally, Dave brought up the question, what about DSN and MDN.
> >
> > Same basic issues. It should be pointed out they are not requirements to
> be
> > part of the IEF. But DSN and MDN are more well defined, very easy,
> > generalized structural issues that fit cleanly into the automated and
> > generalize framework.
> 
> More to my points:
> 
> DSN is based on a generalized bounce requirement that is a natural part of
> the IEF. DSN is an optional and specific standard implementation of the
> requirement.
> 
> MDN is based on a RFC 2822 header existence. Also a natural part of a IEF.
> 
> On the other hand:
> 
> Sieve is based on a Mail Filtering concept that is _not_ a requirement to be
> part of the IEF process.  You can pull the "SIEVE" box, the message
> filtering concept, out from figure 5 and you still have a generalized IEF
> process that is widely in place. You can not do the same with Bounce (DSN)
> or return receipt (MDN) concepts without breaking the framework.
> 
> Suggestion:
> 
> - Change the Sieve box to MFA (Mail Filtering Agent).
> - Reference Sieve parentheically in the document, such like others are (i,e.
> POP, IMAP, etc).
> 
> Sincerely,
> 
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
> http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
> http://www.winserver.com/spamstats (WcSAP Anti-Spam Stats)
> 
> 
> 
> 
> 
> 
> 

To: Hector Santos <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
From: [email-masked]
Subject: Re: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 16:26:54 +0100
Sender: [email-masked]

> Clarification:
> 
> ----- Original Message -----
> From: "Hector Santos" <[email-masked]>
> To: <[email-masked]>
> Cc: "IETF-SMTP" <[email-masked]>
> Sent: Wednesday, March 30, 2005 8:32 AM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
> 
> 
> >
> > Finally, Dave brought up the question, what about DSN and MDN.
> >
> > Same basic issues. It should be pointed out they are not requirements to
> be
> > part of the IEF. But DSN and MDN are more well defined, very easy,
> > generalized structural issues that fit cleanly into the automated and
> > generalize framework.
> 
> More to my points:
> 
> DSN is based on a generalized bounce requirement that is a natural part of
> the IEF. DSN is an optional and specific standard implementation of the
> requirement.
> 
> MDN is based on a RFC 2822 header existence. Also a natural part of a IEF.
> 
> On the other hand:
> 
> Sieve is based on a Mail Filtering concept that is _not_ a requirement to be
> part of the IEF process.  You can pull the "SIEVE" box, the message
> filtering concept, out from figure 5 and you still have a generalized IEF
> process that is widely in place. You can not do the same with Bounce (DSN)
> or return receipt (MDN) concepts without breaking the framework.
> 
> Suggestion:
> 
> - Change the Sieve box to MFA (Mail Filtering Agent).
> - Reference Sieve parentheically in the document, such like others are (i,e.
> POP, IMAP, etc).
> 
> Sincerely,
> 
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
> http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
> http://www.winserver.com/spamstats (WcSAP Anti-Spam Stats)
> 
> 
> 
> 
> 
> 
> 

To: Hector Santos <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
From: [email-masked]
Subject: Re: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 16:26:54 +0100
Sender: [email-masked]

> Clarification:
> 
> ----- Original Message -----
> From: "Hector Santos" <[email-masked]>
> To: <[email-masked]>
> Cc: "IETF-SMTP" <[email-masked]>
> Sent: Wednesday, March 30, 2005 8:32 AM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
> 
> 
> >
> > Finally, Dave brought up the question, what about DSN and MDN.
> >
> > Same basic issues. It should be pointed out they are not requirements to
> be
> > part of the IEF. But DSN and MDN are more well defined, very easy,
> > generalized structural issues that fit cleanly into the automated and
> > generalize framework.
> 
> More to my points:
> 
> DSN is based on a generalized bounce requirement that is a natural part of
> the IEF. DSN is an optional and specific standard implementation of the
> requirement.
> 
> MDN is based on a RFC 2822 header existence. Also a natural part of a IEF.
> 
> On the other hand:
> 
> Sieve is based on a Mail Filtering concept that is _not_ a requirement to be
> part of the IEF process.  You can pull the "SIEVE" box, the message
> filtering concept, out from figure 5 and you still have a generalized IEF
> process that is widely in place. You can not do the same with Bounce (DSN)
> or return receipt (MDN) concepts without breaking the framework.
> 
> Suggestion:
> 
> - Change the Sieve box to MFA (Mail Filtering Agent).
> - Reference Sieve parentheically in the document, such like others are (i,e.
> POP, IMAP, etc).
> 
> Sincerely,
> 
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
> http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
> http://www.winserver.com/spamstats (WcSAP Anti-Spam Stats)
> 
> 
> 
> 
> 
> 
> 

To: Hector Santos <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
From: [email-masked]
Subject: Re: -draft-crocker-email-arch-04.txt SIEVE and Message filtering
Date: Wed, 30 Mar 2005 16:25:31 +0100
Sender: [email-masked]

Hello Hector

Thanks for your reply it made me think again about message filtering in general and sieve in particular.

I hope that I am not seen as a promotor of Sieve and that in my responce I made clear that I was not talking about Sieve as mailfiltering service but as a special kind of e-mail message. (an email message that goes from the MUA >>TO<< the MDA instead of the other normal way)
I am aware that the distinction is easy made especially because in the current draft messages and services are put in one section. That is one of the reasons why I wanted to split the services section in two seperate sections one for  messages and one for  services setting. The other was to create sub sections for message technicalities and the different messages. 

A general remark in the services section about mailfiltering services like the one you suggested is fine with me, but I would suggest:

---

section ? Services

 ?.1  Message Filters:

Spam and virusses are becomming a bigger and bigger tread to the safety and usability of the internet mail system.

Message Filters provides a means of specifying conditions for differential handling of mail,
Most common is to do that in the MDA or MUA, but filtering can be done at many different points along the transit path.

Sieve [RFC3028] is one such implementation but there are many other Message Filtering dialect directives, especially within a single AU.

---

BTW I always thought of Sieve more as a scripting-language that would use the spam and virusscores of REAL mail filtering programs.

>From the sieve homepage http://www.cyrusoft.com/sieve/

Sieve is intended as a first-stage building block for mail filters, although one that in and of itself provides significant functionality for a large number of possible uses.

Sieve scripts are simply data for Sieve engines to execute.

Sieve is not a once-and-for-all solution for pressing problems addressed by filtering, such as anti-Spam efforts, although it certainly is intended to facilitate construction of such solutions.

I hope You and Dave (and everybody else) can live with this sugestion, but please respond if you do not agree with this 

Last remarks:
IMAP is in the draft
SFP is not (it is no RFC yet that is maybe why on http://www.imc.org/ietf-mta-filters/mail-archive/maillist.html there are no post on SFP , or maybe I just did not searh long enough)

> 
> 
> ----- Original Message -----
> From: <[email-masked]>
> To: "ietf-smtp" <[email-masked]>
> Sent: Wednesday, March 30, 2005 4:55 AM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
> 
> 
> >
> > No sorry I disagree i think this draft is not ready yet
> >
> >
> >
> > A point about SIEVE
> >
> > I think that sieve should stay in the draft becaurse of the Sieve
> > MIME extension (see RFC3028 section 7) Sieve itself is on of the
> > many, many ways to filter spam, what makes it special is that it
> > is RFC and it has its own MIME extension. But I am open for
> > discussion on this point. (this is also why i mentioned the
> > sieve message and not the sieve messagefiltering itself)
> 
> Look, when not just throw in IMAP in the picture. Its a RFC too.
> 
> I am not asking for the removal of Sieve but to change the wording around to
> point out very clearly it is not a requirement. As it is worded, it makes
> it seem to current or future newcomers that Sieve is a natural part of the
> process, which it is clearly not.
> 
> First, it is clearly not a generalized technology, but a specific
> technology that is clearly not required.
> 
> Second, it is a specific language that upon itself imposes specific
> installation of specialized software.
> 
> Third, there are a vast degree of different integrated frameworks that use
> different "message filter" hooks. Spam Assassin is probably the most
> popular, and there are many more mail filtering "hooks."
> 
> Forth, you certainly do not wish to minimize new ideas because of the
> erroneous notion that it must be based on Sieve. In fact, I don't believe
> Sieve has yet to incorporate DNS based security functionality. Can it
> implement SPF for example? Don't know myself, but I don't see it in RFC
> 3038. But we know for sure there are other "mail filtering" systems that do
> offer DNS based lookup methods.
> 
> Finally, Dave brought up the question, what about DSN and MDN.
> 
> Same basic issues. It should be pointed out they are not requirements to be
> part of the IEF. But DSN and MDN are more well defined, very easy,
> generalized structural issues that fit cleanly into the automated and
> generalize framework.
> 
> Sieve, on the other hand, is a special and specific computer language that
> requires a specific set of compilers and/or p-code interpreters, etc. It
> requires a set of tertiary tools, such as MYSQL or some other requirement.
> If you don't have all the requirements, you have to write it yourself. In
> fact, I believe there might be just one Windows implementation of it,
> http://www.cyrusoft.com/sieve/.
> 
> DSN and MDN do not require special "compilers" or "p-code!"
> 
> In my view, Sieve does not fit well in a "Generalized Internet Email
> Framework (GIEF)." Sure, mention it, no doubt, but don't make it sound like
> its a "requirement" or a natural part of the GIEF. It is not.
> 
> This document becomes nearly useless with such dependencies.
> 
> But then again, maybe that is what Dave wants the world to believe, that
> Sieve is indeed a natural part of the Internet Email Framework - when in
> fact, it is not.
> 
> Sincerely,
> 
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
> http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
> http://www.winserver.com/spamstats (WcSAP Anti-Spam Stats)
> 
> 

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 09:32:16 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

Clarification:

----- Original Message -----
From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Wednesday, March 30, 2005 8:32 AM
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt

>
> Finally, Dave brought up the question, what about DSN and MDN.
>
> Same basic issues. It should be pointed out they are not requirements to
be
> part of the IEF.  But DSN and MDN are more well defined, very easy,
> generalized structural issues that fit cleanly into the automated and
> generalize framework.

More to my points:

DSN is based on a generalized bounce requirement that is a natural part of
the IEF.  DSN is an optional and specific standard implementation of the
requirement.

MDN is based on a RFC 2822 header existence. Also a natural part of a IEF.

On the other hand:

Sieve is based on a Mail Filtering concept that is _not_ a requirement to be
part of the IEF process.   You can pull the "SIEVE" box, the message
filtering concept, out from figure 5 and you still have a generalized IEF
process that is widely in place.  You can not do the same with Bounce (DSN)
or return receipt (MDN) concepts without breaking the framework.

Suggestion:

- Change the Sieve box to MFA  (Mail Filtering Agent).
- Reference Sieve parentheically in the document, such like others are (i,e.
POP, IMAP, etc).

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 08:32:30 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: <[email-masked]>
To: "ietf-smtp" <[email-masked]>
Sent: Wednesday, March 30, 2005 4:55 AM
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt

>
> No sorry I disagree i think this draft is not ready yet
>
>
>
> A point about SIEVE
>
> I think that sieve should stay in the draft becaurse of the Sieve
> MIME extension (see RFC3028 section 7)  Sieve itself is on of the
> many, many ways to filter spam, what makes it special is that it
> is RFC and it has its own MIME extension. But I am open for
> discussion on this point.  (this is also why i mentioned the
> sieve message and not the sieve messagefiltering itself)

Look, when not just throw in IMAP in the picture.  Its a RFC too.

I am not asking for the removal of Sieve but to change the wording around to
point out very clearly it is not a requirement.  As it is worded, it makes
it seem to current or future newcomers that Sieve is a natural part of the
process, which it is clearly not.

First,  it is clearly not a generalized technology, but a specific
technology that is clearly not required.

Second, it is a specific language that upon itself imposes specific
installation of specialized software.

Third, there are a vast degree of different integrated frameworks that use
different "message filter" hooks.  Spam Assassin is probably the most
popular,  and there are many more mail filtering "hooks."

Forth, you certainly do not wish to minimize new ideas because of the
erroneous notion that it must be based on Sieve.  In fact, I don't believe
Sieve has yet to incorporate DNS based security functionality.  Can it
implement SPF for example?  Don't know myself, but I don't see it in RFC
3038. But we know for sure there are other "mail filtering" systems that do
offer DNS based lookup methods.

Finally, Dave brought up the question, what about DSN and MDN.

Same basic issues. It should be pointed out they are not requirements to be
part of the IEF.  But DSN and MDN are more well defined, very easy,
generalized structural issues that fit cleanly into the automated and
generalize framework.

Sieve, on the other hand, is a special and specific computer language that
requires a specific set of compilers and/or p-code interpreters, etc. It
requires a set of tertiary tools, such as MYSQL or some other requirement.
If you don't have all the requirements, you have to write it yourself.  In
fact, I believe there might be just one Windows implementation of it,
http://www.cyrusoft.com/sieve/.

DSN and MDN do not require special "compilers" or "p-code!"

In my view, Sieve does not fit well in a "Generalized Internet Email
Framework (GIEF)."  Sure, mention it, no doubt, but don't make it sound like
its a "requirement" or a natural part of the GIEF.  It is not.

This document becomes nearly useless with such dependencies.

But then again, maybe that is what Dave wants the world to believe, that
Sieve is indeed a natural part of the Internet Email Framework - when in
fact, it is not.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

To: ietf-smtp <[email-masked]>
From: [email-masked]
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Wed, 30 Mar 2005 10:55:01 +0100
Sender: [email-masked]

No sorry I disagree i think this draft is not ready yet 

I think the following discussion points are still open:

Most importand open discussion points:
1 You havn't included nor refuted the  MRA (message retrieval agent) solution to the Ms-Ms dichotomy (but I agree my text needs polishing and extension for example ODMR and SMTP/etrn ect. are not mentioned)
I will post a new text later this week 

2 The discussion if MUA's are part of the MailHandling system is still open. Do we need the seperate identities "internet mailsystem" and "(internet) mailhandling system",if the only difference seems to be that ONLY MUA's are part of the "internet mail system" but not of the "(internet) mailhandling system".   

3 The administrative units is should still be in development.
  (Glad to know that you had other ideas than i mentioned in my last email but i think this has nothing to do with the architecture of the mailservice, it is mora about the internet layer, and to create tussle space as far as i can see.)
Maybe some sentences that AU should not block any email at their border is enough about this.

4 The problem how real life and entities in other RFC's relate to entities in this draft. This is now even more the case now this draft will enter the standard track. Otherwise this draft will generate more confusion than it can ever eradicate. 

Smaller points:
5 SMTP bounces (the SMTP error codes and otherways to come to a DNS) 
  (the draft does not say anything about it)
  And I think there a section about it is very helpful

6 Messages as seperate from services
  (better in seperate sections)

7 DNS MX description 

8 Messgage tracking (is now RFC)

Further see my emails:
http://www.imc.org/ietf-smtp/mail-archive/msg01618.html
http://www.imc.org/ietf-smtp/mail-archive/msg01464.html
http://www.imc.org/ietf-smtp/mail-archive/msg01475.html
http://www.imc.org/ietf-smtp/mail-archive/msg01482.html

Bruce Lilly s emails:
http://www.imc.org/ietf-smtp/mail-archive/msg01472.html
http://www.imc.org/ietf-smtp/mail-archive/msg01471.html

And other emails on the mailinglist

A small point not mentioned earlier:

The reference for Dns messages is wrong RFC3461 is about the SMTP DNS extension, the DNS message MIME is described in RFC3464 (with in his turn builds on RFC3642)

A point about SIEVE

I think that sieve should stay in the draft becaurse of the Sieve MIME extension (see RFC3028 section 7)  Sieve itself is on of the many, many ways to filter spam, what makes it special is that it is RFC and it has its own MIME extension. But I am open for discussion on this point.  (this is also why i mentioned the sieve message and not the sieve messagefiltering itself)

PS If you really want to use version 04 of the draft as RFC then please remove my name from the list of reviewers. 
(I can not agree with the present version)

PS 2
Dave 
There seems to be something wrong with your mailhandling system, emails send to you get bounced back. (To your bbiw.net account)

Greetings willemien

> 
> 
> 
> Folks,
> 
> I've submitted draft-crocker-email-arch-04.txt as the latest revision to the
> Internet Mail Architecture document. 
> 
> Since the changes are relatively small, I'm going to ask folks to treat this
> as a Last Call on the document. 
> 
> In a couple of weeks, I'll ask the IESG to consider the document for Proposed 
> Standard.
> 
> Please consider doing a review on the document, with that goal in mind.
> 
> Until the draft is issued through the Internet Drafts process, it can be
> accessed at:
> 
>  <http://bbiw.net/current.html#email>
> 
> Thanks.
> 
>  d/
>  ---
>  Dave Crocker
>  Brandenburg InternetWorking
>  +1.408.246.8253
>  dcrocker a t ...
>  WE'VE MOVED to: www.bbiw.net
> 
> 
> 

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Fw: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Tue, 29 Mar 2005 12:56:21 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: "Hector Santos" <[email-masked]>
To: "Dave Crocker" <[email-masked]>
Sent: Tuesday, March 29, 2005 12:53 PM
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt

> That's too bad. I don't see the logic.  SIEVE is a specific complex
language
> is not mandatory for internet email operations.  Systems already have
other
> options in place, proprietary, open or otherwise.   How about Spam
Assassin?
> A pretty popular package.  How about wsSPAMGuard, a 3rd party package for
> our system.   How about McAfee or Norton?   All "Mail Filtering"
"blackbox"
> components that can be hooked into a framework.  The future of the email
> framework is most definitely headed to integrated frameworks.  SIEVE is
just
> one option and I strongly believe it takes away from your document to make
> it read like it is a requirement, which it is not.
>
> I posted my original reply to you and the list.  But you use the wrong
> IETF-SMTP address in your original message.  So anyone doing a REPLY to
your
> post to the list will fail  I just got the bounce.  I will forward my
> response to the list.   Of course, I don't expect much feedback, but I do
so
> just to get it on public record.
>
> Sincerely,
>
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
> http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
> http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)
>
>
>
> ----- Original Message -----
> From: "Dave Crocker" <[email-masked]>
> To: "Hector Santos" <[email-masked]>; "ietf-smtp"
> <[email-masked]>
> Sent: Tuesday, March 29, 2005 12:44 PM
> Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
>
>
> >  My only problem with the draft in your attempt to make it a standard is
> >  the how you insist in imposing "Sieve"  a specific technology and
clearly
> >  a specific language framework in what is otherwise a generalize
Internet
> >  Email Framework.
>
> Sieve is listed in the context of Internet standards.  Are there other
> internet standards for filtering that have gained any significant
adoption?
>
> Note that that same list shows MDN and DSN.
>
>
>
> >  We don't use SIEVE and we don't intend nor plan to use SIEVE anytime in
>
> That is nice, but not overly relevant.  The document does not claim that
> everyone uses any of the cited standards nor does it claim that anyone
must
> use them.
>
> Absent a groundswell of community support for the change you are
requesting,
> the text should stay as it is.
>
>
>
>   d/
>   ---
>   Dave Crocker
>   Brandenburg InternetWorking
>   +1.408.246.8253
>   dcrocker  a t ...
>   WE'VE MOVED to:  www.bbiw.net
>
>
>

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Date: Tue, 29 Mar 2005 12:54:50 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

Dave,

My only problem with the draft in your attempt to make it a standard is the
how you insist in imposing "Sieve"  a specific technology and clearly a
specific language framework in what is otherwise a generalize Internet Email
Framework.

I brought this up atleast twice already over the year and I don't see why
you refuse to make the "Mail Filtering" element of your framework more
generalized.  Everything else in the document is generalized.   You didn't
mention specific MSA, MTA software for obvious reasons.   SIEVE is a
specific MAIL FILTERING language.  Sure, reference to it.  But please don't
impose it as the required MAIL FILTERING element in your framework.

Why is it an issue with me?

We don't use SIEVE and we don't intend nor plan to use SIEVE anytime in the
future.  We have our own "Mail Filtering" component, just like we have our
own MSA, MTA, etc.   We spent an awful lot of time and energy making sure
our software is compliant with the standard.  We provide confident to our
customers based on these standard operational assurances. The last thing we
want is to be getting into is silly debate games such as "Based on the
standard Internet Email Framework (IEF), since you don't use SIEVE, you are
not compliant with the IEF standard."

Suggested small changes:

| 4.1 Message
|
| Message Filtering:
|
| Message Filtering provides a means of specifying
| implemention specific conditions for differential handling of mail,
| at the time of delivery. One such implementation is SIEVE [RFC3028].
Figure 5
| shows a Message Filter operation going from the rMUA to the MDA. However
| filtering can be done at many different points along the transit path and
| any one or more of them might be subject to Message Filtering dialect
| directives, especially within a single AU. Hence, the Figure shows
| only one relationship, for simplicity.

The small changes wouldn't be much to ask right?  These small wording
changes will remove all PR pressures on vendors who don't use sieve or don't
wish to use it.

This small consideration will be appreciated.  As far as I'm concern, the
lack of the consideration completely undermines (make it useless) in what is
otherwise a fine document.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
http://www.winserver.com/wcsap (Wildcat! Sender Authentication Protocol)
http://www.winserver.com/spamstats  (WcSAP Anti-Spam Stats)

----- Original Message -----
From: "Dave Crocker" <[email-masked]>
To: "ietf-smtp" <[email-masked]>
Sent: Tuesday, March 29, 2005 11:23 AM
Subject: pseudo LAST CALL - draft-crocker-email-arch-04.txt

>
>
> Folks,
>
> I've submitted draft-crocker-email-arch-04.txt as the latest revision to
the
> Internet Mail Architecture document.
>
> Since the changes are relatively small, I'm going to ask folks to treat
this
> as a Last Call on the document.
>
> In a couple of weeks, I'll ask the IESG to consider the document for
Proposed
> Standard.
>
> Please consider doing a review on the document, with that goal in mind.
>
> Until the draft is issued through the Internet Drafts process, it can be
> accessed at:
>
>   <http://bbiw.net/current.html#email>
>
> Thanks.
>
>   d/
>   ---
>   Dave Crocker
>   Brandenburg InternetWorking
>   +1.408.246.8253
>   dcrocker  a t ...
>   WE'VE MOVED to:  www.bbiw.net
>
>
>
>

From: Dave Crocker <[email-masked]>
To: Hector Santos <[email-masked]>, ietf-smtp <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Tue, 29 Mar 2005 09:44:49 -0800
Subject: Re: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

>  My only problem with the draft in your attempt to make it a standard is
>  the how you insist in imposing "Sieve"  a specific technology and clearly
>  a specific language framework in what is otherwise a generalize Internet
>  Email Framework.

Sieve is listed in the context of Internet standards.  Are there other 
internet standards for filtering that have gained any significant adoption?

Note that that same list shows MDN and DSN.

>  We don't use SIEVE and we don't intend nor plan to use SIEVE anytime in

That is nice, but not overly relevant.  The document does not claim that 
everyone uses any of the cited standards nor does it claim that anyone must 
use them.

Absent a groundswell of community support for the change you are requesting, 
the text should stay as it is.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: Dave Crocker <[email-masked]>
To: ietf-smtp <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Tue, 29 Mar 2005 08:23:10 -0800
Subject: pseudo LAST CALL - draft-crocker-email-arch-04.txt
Sender: [email-masked]

Folks,

I've submitted draft-crocker-email-arch-04.txt as the latest revision to the
Internet Mail Architecture document.  

Since the changes are relatively small, I'm going to ask folks to treat this
as a Last Call on the document.  

In a couple of weeks, I'll ask the IESG to consider the document for Proposed 
Standard.

Please consider doing a review on the document, with that goal in mind.

Until the draft is issued through the Internet Drafts process, it can be
accessed at:

  <http://bbiw.net/current.html#email>

Thanks.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

From: "Willemien" <[email-masked]>
To: "Dave Crocker" <[email-masked]>, <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 more comments: internet mail service & Administrative domains
Date: Wed, 23 Mar 2005 14:42:09 -0000
Sender: [email-masked]

Hello Dave
I hope you are not abandoning this draft.
I find the (some parts) of the discussion very enlightening. But can imagine
that you get bogged down by it.
(it just is NOT an easy subject)

And I  have  two more points

Maybe a textual point:
Is there a difference between the
"mail Handling service" (section 1 and 2.2) and the
"Internet Mail Service" (section 2.3 and section 6)
And if there is a difference  can this be elaborated?

Administrative domains
I did some more thinking about this but the more t think about it the  more
I get confused about this idea.
Is the idea behind this section:

1: To indicate the (static) recipient domain, All the MTA 's where the MX
records point to, and every agent from there to the MS or even the rMUA (if
it is local)

2: something like the mailflow schedule
<http://www.ripe.net/ripe/meetings/ripe-47/mailflows.pdf>
   (the sender/originator part is dynamic, change according to the place of
the originator)
   ( PS for this point there are 4 slightly different mailflow diagrams in
circulation)

3: the prior agreement links in mail handling. (Within administrative
domains everything is based on a prior agreement)

4: namedomain (example.com etc) description (very static, is not very useful
I think )

5: Something completely different, or an confusing mix from the idea's above
and I have not grasped it yet....

----- Original Message ----- 
From: "Dave Crocker" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, February 17, 2005 3:50 PM
Subject: draft-crocker-email-arch-03

>
> Folks,
>
> I think (hope) the email architecture document has reached a stable point,
in its attempt to describe the current service.
>
> The latest version has substantial changes, but I think they are
refinements from the previous version and reflect feedback I've received. Of
course, any additional comments are more than welcome, and I have no doubt
there can be debate about lots of the details.
>
> But the goal is to get this published sooner, rather than later, so I am
hoping this is the last major revision.
>
> I've submitted the doc to the i-d folks. In the interim, you can access it
at:
>
> ; <http://bbiw.net/specifications/draft-crocker-email-arch-03.html>
>
> ; <http://bbiw.net/specifications/draft-crocker-email-arch-03.txt>
>
>
> d/
> --
> Dave Crocker
> Brandenburg InternetWorking
> +1.408.246.8253
> dcrocker a t ...
> WE'VE MOVED to: www.bbiw.net
>

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 15 Mar 2005 18:08:10 -0500
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Tue, Mar 15, 2005 at 02:03:04PM -0800, Russ Allbery wrote:
> Hector Santos <[email-masked]> writes:
> 
> > 220 mail6.velocitywest.com ESMTP
> > helo hdev1
> > 250 mail6.velocitywest.com
> > mail from: <>
> > 250 ok
> > rcpt to: <[email-masked]>
> > 250 ok
> > data
> > 354 go ahead
> > hi there david! No RFC Header!
> > .
> > 250 ok 1110911858 qp 17196
> 
> > PS: I didn't show it above, but at a minimum, your server should check
> > for MAIL FROM correct syntax. It accepted a typo address.  That alone
> > will add some security.
> 
> qmail accepts a space before the envelope sender specifically because
> sendmail does, and therefore many e-mail clients send it that way.  This
> gets back to that previous discussion about false positives on enforcement
> of strict SMTP checking.

I had the impression that he was talking about something not shown, other
than the dialog above.

While we're off here on a tangent, qmail also takes things like:

   mail blah:<[email-masked]>
   rcpt to-diddly-do:<[email-masked]>

> (Although for this issue in particular, there are several different
> SMTP front ends that can be used instead of qmail-smtpd that do stricter
> checking.)

Indeed there are...

mm

To: [email-masked]
Subject: Re: (lack of) message header field ordering
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 15 Mar 2005 14:03:04 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:

> 220 mail6.velocitywest.com ESMTP
> helo hdev1
> 250 mail6.velocitywest.com
> mail from: <>
> 250 ok
> rcpt to: <[email-masked]>
> 250 ok
> data
> 354 go ahead
> hi there david! No RFC Header!
> .
> 250 ok 1110911858 qp 17196

> PS: I didn't show it above, but at a minimum, your server should check
> for MAIL FROM correct syntax. It accepted a typo address.  That alone
> will add some security.

qmail accepts a space before the envelope sender specifically because
sendmail does, and therefore many e-mail clients send it that way.  This
gets back to that previous discussion about false positives on enforcement
of strict SMTP checking.

qmail also does no syntax checking of the mail message (the DATA portion)
at all in the SMTP listener specifically to limit possible remote attacks
by doing the minimal possible work in the publically accessible portion of
the system.  It's a questionable decision (and believe me, lots of people
have questioned it), but it was an intentional design choice and one that
arguing about on IETF mailing lists is guaranteed not to change anything
(most of us here have been there and done that).

If one doesn't like these design choices, one shouldn't use qmail.  qmail
is pretty much an all or nothing deal; either you can live with djb's
quirks as well as his good design points, or you need to use a different
MTA.  (Although for this issue in particular, there are several different
SMTP front ends that can be used instead of qmail-smtpd that do stricter
checking.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, "David MacQuigg" <[email-masked]>
Subject: Re: (lack of) message header field ordering
Date: Tue, 15 Mar 2005 14:11:46 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: "David MacQuigg" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 15, 2005 11:04 AM
Subject: Re: (lack of) message header field ordering

>
> This has been a very interesting discussion.  I didn't follow all the
> details, but if I understand the bottom line, some MTAs do not follow the
> common practice of only prepending new headers.  If these MTAs are rare on
> the public part of the Internet, and are non-compliant with existing
> standards in other ways, I would say ignoring them is OK.

David,  check this out:

220 mail6.velocitywest.com ESMTP
helo hdev1
250 mail6.velocitywest.com
mail from: <>
250 ok
rcpt to: <[email-masked]>
250 ok
data
354 go ahead
hi there david! No RFC Header!
.
250 ok 1110911858 qp 17196

It accepted the data block with no RFC header?  How did your system handle
this? What is the standard to check for the header? and under what criteria?
Should we ignore your system based on your statement above?

These are rthetorical questions. No need to answer them. The fact is, most,
if not every, mail server behaves the same way.

Go from there.

PS: I didn't show it above, but at a minimum, your server should check for
MAIL FROM correct syntax. It accepted a typo address.  That alone will add
some security.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Tue, 15 Mar 2005 09:04:15 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

This has been a very interesting discussion.  I didn't follow all the 
details, but if I understand the bottom line, some MTAs do not follow the 
common practice of only prepending new headers.  If these MTAs are rare on 
the public part of the Internet, and are non-compliant with existing 
standards in other ways, I would say ignoring them is OK.

If they are not so rare, and do in fact perform some useful function, we 
can deal with them as we do any untrusted source.  We already block open 
relays, even though they may be SMTP compliant, and even if someone, 
somewhere, has an essential need for them.  My goal is not to put spammers 
out of business, but just to keep them out of my inbox.

Immediate blocking is not the only option in handling messages from 
insecure MTAs.  A properly designed authentication protocol will not 
*force* a policy on what domains should be blocked, but simply *facilitate* 
enforcement of a policy decision by the Recipient.  Any complaints about 
"impairment of email" will then go to the intended Recipient.

Here is a typical scenario:

    Spammer -->  Forwarder -->  Receiver --> Recipient

I trust my own Forwarder, or I get another one.  I am counting on the 
headers above and including the Forwarder's Received header to be unaltered 
by the Spammer.  That will allow me to use the Spammer's domain name 
(authenticated by the Forwarder) as an input to my spam filter.

A slightly different scenario is that the Receiver does the filtering on 
behalf of all its Recipients.  Then it could have a list of trusted 
forwarders, and step down through the pile of headers until it gets to the 
first domain not on the list.  That domain will be used for filtering, just 
as if it had connected directly to the Receiver.

In either scenario it is the domain just ahead of the Recipient's Forwarder 
that gets rated.  Normally, this is the Originator's domain, but it could 
be another Forwarder authorized by the Originator.  The sticky situation is 
where we have somewhere in the middle a "text-to-binary translator", a 
"header randomizer", or some other weird thing not authorized by either the 
Originator or the Recipient.

Here is how I would write the requirements for a domain that wishes to 
participate in authenticated transfers:

 From http://www.ece.arizona.edu/~edatools/etc/draft-authent-interop-00.htm:

5) If the message is forwarded, the results of authentication must be made
available to all subsequent receiving MTAs.  The format should be simple and
standardized to facilitate later blocking and filtering.

6) A Forwarder must not alter either the incoming headers nor the body of a
message.  Prepending of headers is acceptable.

6a) A Forwarder that needs to alter a message must act as a new Originator,
using a domain name for which it is authorized.

There are also some example headers further down in the draft.

Will these requirements conflict with any existing or proposed 
protocol?  I'm looking for the minimum requirements that will still meet 
our goal of having simple and reliable inter-operation for all MTAs 
involved in an authenticated transfer.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Mon, 14 Mar 2005 11:48:38 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Mon March 14 2005 10:32, Tony Finch wrote:

> Anyone who uses such software will find it difficult to diagnose problems
> and will suffer from less effective spam filtering until the software is
> fixed. I don't think improvements to the specification should be held back
> by idiot implementations.

The specification clearly states that order is not significant and
notes that fields may be reordered in transit.  Always has said that.
The problem isn't the specification; it's the unwarranted assumption
that reordering never happens.

> Did you find out how widely deployed these manglers were and whether they
> were usually installed as end-points or as relays?

Most are deployed at one site fewer than if they didn't rearrange
fields, failed to comply with [E]SMTP requirements, were reasonably
configurable, etc.:-)   Many were fairly widely-used MTAs; others
were POP->SMTP converters for retrieving (or forwarding) POP-based
message stores (ISP-hosted) to another message store.

In a similar discussion several months ago on the ietf-822 mailing
list, Laird Breyer stated that there are a number of database-
based UA implementations (Outlook and Lotus Notes were specifically
mentioned).

FWIW, I ended up with a home-brew POP->SMTP converter, sendmail,
and Cyrus IMAP.

Date: Mon, 14 Mar 2005 15:32:36 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Mon, 14 Mar 2005, Bruce Lilly wrote:
>
> Four and a half years ago I looked in detail at 34 email transfer
> products; several of them reordered or otherwise mangled message
> header fields.  A certain breed of developer hears "name-value
> pairs" and thinks "database"; unless pains are taken to preserve
> ordering, a database approach results in (pseudo-)random
> rearrangement of fields.

Anyone who uses such software will find it difficult to diagnose problems
and will suffer from less effective spam filtering until the software is
fixed. I don't think improvements to the specification should be held back
by idiot implementations.

Did you find out how widely deployed these manglers were and whether they
were usually installed as end-points or as relays?

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAIR ISLE: NORTHWEST 4 OR 5 VEERING EAST THEN SOUTHEAST 5 OR 6. FAIR. GOOD.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Mon, 14 Mar 2005 10:23:41 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Mon March 14 2005 09:00, Tony Finch wrote:

> It does depend on a reasonable expectation of minimal
> re-ordering, but that is true almost all the time.

Four and a half years ago I looked in detail at 34 email transfer
products; several of them reordered or otherwise mangled message
header fields.  A certain breed of developer hears "name-value
pairs" and thinks "database"; unless pains are taken to preserve
ordering, a database approach results in (pseudo-)random
rearrangement of fields.  Unless direct sending from originator
MUA to recipient MX MTA is used (and even sometimes then),
mangling by intermediate MTAs is not under control of either
sender or recipient; it just happens.

Date: Mon, 14 Mar 2005 14:00:01 +0000
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

> > I think a simpler approach to this problem would be to
> > encourage sites to append locally defined trace information
> > (e.g. anti-spam scores) to the start of the header instead of
> > the end, so that it appears interleaved with the Received:
> > fields. This makes it much clearer when a field was added. A
> > fair amount of software already does this, e.g. Delivered-To:
> > and X-Sieve: fields. It also happens to agree with the logic
> > of DomainKeys signatures :-)
>
> As pointed out earlier, doing this risks getting a much higher
> spam probability score from software that doesn't know about
> your particular new headers and hence treats them as evidence of
> header-tampering.

As I said, they have to deal with this already.

> The advantages of separating MTA-provided in-transit information from
> message body ("header") information are pretty obvious. To move that
> way, we would then need to address the question of balancing ease of
> converting to and from the new mechanism and format against additional
> power and flexibility from new formats.

I think that it will be easiest to enhance the semantics of the existing
header format than to add a new mechanism for separate trace information.
In practice you can draw an imaginary line through the header immediately
after the textually-last Received: field; everything before the line will
relate closely to the path that the message took (trace fields, resent-*
fields, locally-defined fields); what comes after the line will be mostly
user-related, but it will be jumbled with trace-like stuff such as List-*
fields and spam score fields. This is a practical distinction based on
observation rather than a formal one, so it's not going to be perfect. If
you can encourage implementations to add their trace-like stuff to the
start of the header then this observed distinction will become gradually
more and more useful. No need for significant new functionality, no loss
of functionality caused by MTAs in a message's path that don't understand
the distinction. It does depend on a reasonable expectation of minimal
re-ordering, but that is true almost all the time.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORWEGIAN BASIN: NORTH OR NORTHEAST 6 TO GALE 8 DECREASING 5, BECOMING
CYCLONIC LATER IN EAST. OCCASIONAL SNOW. GOOD OCCASIONALLY MODERATE OR POOR.
LIGHT ICING IN NORTHWEST, MINIMUM TEMPERATURE MS03 TO MS05.

From: Bruce Lilly <[email-masked]>
Reply-To: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Mon, 14 Mar 2005 08:39:37 -0500
Sender: [email-masked]

On Mon March 14 2005 05:11, Arnt Gulbrandsen wrote:
> Bruce Lilly writes:
> > It might make sense to put transport-related additions (trace fields, 
> > List- fields, etc.) in a separate place ("envelope") distinct from 
> > the end-to-end, user-to-user message header (date, to/from/cc, 
> > subject, etc.).
> >
> > Either way, it's something different from what we have now.
> 
> It does sound a bit like an S/MIME message, doesn't it?

Yes, a "triple-wrapped" one; and sadly many "MIME" implementations
can't handle fairly simple MIME nesting (e.g.
https://bugzilla.mozilla.org/show_bug.cgi?id=184869
).  So, almost certainly, many recipients would be unable to view
such messages.  I think that's the right general direction, though,
and perhaps it would encourage developers to fix buggy "MIME"
implementations.

Date: Mon, 14 Mar 2005 07:50:19 -0500
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>, John C Klensin <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

--On Monday, 14 March, 2005 12:29 +0000 Tony Finch
<[email-masked]> wrote:

> On Mon, 14 Mar 2005, John C Klensin wrote:
>> 
>> And like X.400.  And, to a limited extent (motivated by trying
>> to maintain compatibility), like
>> draft-klensin-email-envelope-00.
> 
> It appears to continue the mistake of not allowing extensions
> to the trace field syntax and semantics - though the draft
> doesn't clearly define the contents of the ENVL .

Again, it was a deliberately half-baked draft that was posted in
a specific context.  If there were any interest in going in that
direction --and my belief at this point is that there is not--
then all sorts of refinements are possible and should be
discussed.  One of them is clearly a review of what should be in
the trace fields.
 
>> Unless there is evidence to refute that conclusion, or at
>> least a more persuasive draft, the discussion about getting
>> headers into the envelope is probably a waste of time.  In
>> other words, from my point of view at least, this would be a
>> good idea, and we could do it --although certainly not
>> overnight.  We just do not, in practice, care.
> 
> I think a simpler approach to this problem would be to
> encourage sites to append locally defined trace information
> (e.g. anti-spam scores) to the start of the header instead of
> the end, so that it appears interleaved with the Received:
> fields. This makes it much clearer when a field was added. A
> fair amount of software already does this, e.g. Delivered-To:
> and X-Sieve: fields. It also happens to agree with the logic
> of DomainKeys signatures :-)

As pointed out earlier, doing this risks getting a much higher
spam probability score from software that doesn't know about
your particular new headers and hence treats them as evidence of
header-tampering.   You will have to determine whether it is
worth it on balance, but, for me, keeping Internet mail globally
interoperable is even more important than techniques that will
provide a delta in the amount of spam.

The advantages of separating MTA-provided in-transit information
from message body ("header") information are pretty obvious. To
move that way, we would then need to address the question of
balancing ease of converting to and from the new mechanism and
format against additional power and flexibility from new
formats.  The draft is too preliminary to seriously address
those questions; if someone else feels that the special envelope
plan is worth pursuing, feel free.

   john

Date: Mon, 14 Mar 2005 12:29:50 +0000
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Mon, 14 Mar 2005, John C Klensin wrote:
>
> And like X.400.  And, to a limited extent (motivated by trying
> to maintain compatibility), like
> draft-klensin-email-envelope-00.

It appears to continue the mistake of not allowing extensions to the trace
field syntax and semantics - though the draft doesn't clearly define the
contents of the ENVL .

> Unless there is evidence to refute that conclusion, or at least a more
> persuasive draft, the discussion about getting headers into the envelope
> is probably a waste of time.  In other words, from my point of view at
> least, this would be a good idea, and we could do it --although
> certainly not overnight.  We just do not, in practice, care.

I think a simpler approach to this problem would be to encourage sites to
append locally defined trace information (e.g. anti-spam scores) to the
start of the header instead of the end, so that it appears interleaved
with the Received: fields. This makes it much clearer when a field was
added. A fair amount of software already does this, e.g. Delivered-To: and
X-Sieve: fields. It also happens to agree with the logic of DomainKeys
signatures :-)

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MULL OF KINTYRE TO ARDNAMURCHAN POINT: SOUTHWEST BACKING SOUTH OR SOUTHEAST 4,
INCREASING 5 OR 6, VEERING SOUTHWEST 4 LATER. MAINLY FAIR, RAIN LATER. GOOD
BECOMING MODERATE. SLIGHT OR MODERATE, OCCASIONALLY ROUGH IN THE NORTH.

Date: Mon, 14 Mar 2005 07:15:54 -0500
From: John C Klensin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
cc: Tony Finch <[email-masked]>, Bruce Lilly <[email-masked]>
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

--On Monday, 14 March, 2005 11:11 +0100 Arnt Gulbrandsen
<[email-masked]> wrote:

> 
> Bruce Lilly writes:
>> It might make sense to put transport-related additions (trace
>> fields,  List- fields, etc.) in a separate place ("envelope")
>> distinct from  the end-to-end, user-to-user message header
>> (date, to/from/cc,  subject, etc.).
>> 
>> Either way, it's something different from what we have now.
> 
> It does sound a bit like an S/MIME message, doesn't it?

And like X.400.  And, to a limited extent (motivated by trying
to maintain compatibility), like
draft-klensin-email-envelope-00.  The abstract from that draft
read:

	During the last few years, a number of proposals for
	extensions or improvements to email have run into trouble
	with a side-effect of mail relaying.  In the current
	Internet Mail model, every SMTP server is required to
	break strict layering by inserting one or more additional
	"trace" headers into the message headers which are
	actually part of the SMTP payload.  Since the headers are
	altered in transit, header-signing is not an available
	option, various anti-spam and internationalization
	strategies are infeasible or much more complex, and so on.
	This document proposes to change the Internet mail model
	to place the in-transit trace information in the envelope,
	removing the requirement that relaying systems modify the
	message payload.

The draft was posted a year ago last January and long expired:
find your friendly archive or convince me that this discussion
is likely enough to go somewhere that I should repost it.  I've
got a never-posted revision that, among other things, clarifies
the downgrading mechanism (but it is pretty obvious).   So far,
the amount of traction the idea has gotten is consistent with a
fairly clear answer to the sanity check remark at the end of the
introduction.  Unless there is evidence to refute that
conclusion, or at least a more persuasive draft, the discussion
about getting headers into the envelope is probably a waste of
time.  In other words, from my point of view at least, this
would be a good idea, and we could do it --although certainly
not overnight.  We just do not, in practice, care.

    john

Date: Mon, 14 Mar 2005 11:11:48 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Cc: Tony Finch <[email-masked]>, Bruce Lilly <[email-masked]>
Sender: [email-masked]

Bruce Lilly writes:
> It might make sense to put transport-related additions (trace fields, 
> List- fields, etc.) in a separate place ("envelope") distinct from 
> the end-to-end, user-to-user message header (date, to/from/cc, 
> subject, etc.).
>
> Either way, it's something different from what we have now.

It does sound a bit like an S/MIME message, doesn't it?

Arnt

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: (lack of) message header field ordering
Date:  Mon, 14 Mar 2005 00:17:06 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 28
Sender: [email-masked]

John C Klensin wrote:

>> Is this on topic, important? a non-issue?
> Mostly a non-issue, I think.

The problem of "empty mails" as the result of DATA<CRLF>.<CRLF>
is unfortunately a permanent issue in "mail abuse" news groups.
Some "missing" header fields like Date / From / To created by
the MDA are not what most users want.

> The delivery MTA may look (although it doesn't have to), but
> there is a strong case to be made that, if it doesn't like
> what it receives, it should bounce the message rather than
> trying to fix it up

Yes, that could be better, at least if it's also the MX, or if
the envelope passed some tests before.  Otherwise sending
bounces to (most probably) innocent bystanders would be a waste
of bandwidth, then it's cheaper to dump this garbage in the
mailbox of a local user.  Maybe adding "Subject: sorry", if all
header fields are fictitious then a bogus subject isn't worse.

It can't be a SEND FROM if it's a MAIL FROM.  If an instant
messaging protocol uses SEND FROM to transport empty messages,
then that special case should be obvious.

                              Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Sun, 13 Mar 2005 18:15:38 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Sun March 13 2005 17:49, Tony Finch wrote:

> It might make sense to define a subset of the header field name space
> which can be used by these fields, by analogy with Resent-*.

It might make sense to put transport-related additions
(trace fields, List- fields, etc.) in a separate place ("envelope")
distinct from the end-to-end, user-to-user message header
(date, to/from/cc, subject, etc.).

Either way, it's something different from what we have now.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Sun, 13 Mar 2005 18:08:07 -0500
Cc: John C Klensin <[email-masked]>
Sender: [email-masked]

On Sun March 13 2005 16:41, John C Klensin wrote:

> MTAs that are not the
> submission server should not even be looking at the headers to
> determine that the information is missing.   The delivery MTA
> may look (although it doesn't have to), but there is a strong
> case to be made that, if it doesn't like what it receives, it
> should bounce the message rather than trying to fix it up.  The
> standard doesn't require that-- it encourages delivery on the
> basis of the envelope only, i.e., not looking at the headers to
> determine what might be missing.
> 
> > Which begs the question, what is the minimum requirement for
> > an valid RFC header required by a x821 process?
> 
> There is an old controversy about this.  One interpretation is
> that, unlike 2821 with MIME, 821 does not require any particular
> message format at all and, in particular, does not require any
> 822-style headers.   Some people have taken the position for
> many years that a header-free message was essentially required
> for an implementation of SEND to make any sense at all.  
> 
> If one takes a strong version of the "MTAs don't look at the
> headers" principle, and states your question as above, the
> answer is "none".   Of course, one might then claim that
> delivery from the delivery MTA into the mail store (or to the
> receiving MUA) is not part of the "x821 process".
[...]
> > So what logic should a x821 receiver have to determine whether
> > there is a valid RFC header?
> 
> If it is not the final delivery MTA, it doesn't need any
> "logic", because it should not be asking.

Note that both 821 and 2821 require prepending a time-stamp
line (a.k.a. Received field) to the message.  Clearly, if
there is already a message header, that means prepending
a field, and if there is no message header (body only) it
means prepending a field *and* the empty line that separates
header from body.

So, to handle the hypothetical case of a message with no
header, all MTAs would have to look at the message content
(to determine whether or not there is a header) and require
logic to handle the two distinct cases (message with existing
header vs. message that needs to have a separator line as
well as a time-stamp line).

In practice, I suspect most MTAs assume that the DATA is a
message with at least one header field, and consequently
simply prepend a time-stamp line w/o checking.

Prepending a time-stamp line to something that isn't an
RFC [2]822 message with a header makes little sense; if
mucking about with header fields is wrong, mucking about
with body content is more wrong (gateways are permitted
to do so in very specific ways for specific reasons, e.g.
applying transfer encoding).  SMTP MTAs aren't even
theoretically capable of transfer of something which isn't
in Internet [Text] Message Format; prepending a Received
field (a component of that Message Format) precludes that
(unless transparency of transfer is immaterial) -- and that
goes all the way back to RFC 788, which also prepended a
time-stamp line (with a different name).

In practice, therefore, SMTP always transfers Internet
Message Format payloads (with modification, at least by
addition of a Received field).  Since all valid RFC [2]822
Internet Messages must contain at least one field (Date
has been mandatory since long before SMTP existed; at least
as far back as RFC 724), that leaves only non-message
(= non-compliant content) as the only (slightly) interesting
case.  As noted above, I suspect that MTA authors tend not
to worry about it; the reasonable choices are:
1. prepend a time-stamp line to the DATA and pass it along.
   if the content is already broken, it won't be more
   broken.
2. check content for lack of header, insert separator if
   there is no header, then prepend time-stamp line. If
   the previous hop screwed up (or detection is faulty),
   that can result in effectively putting the (screwup and)
   previous header in the message body (I've seen the
   results of that).
3. check for lack of header, bounce if there is none, else
   prepend time-stamp line and forward.

Note that only the first choice involves neither peeking
at content nor logic, and that is predicated on the
assumption that content is always an Internet Message with
an existing header.  The other choices involve both looking
at content and some logic based on the findings.

Date: Sun, 13 Mar 2005 22:49:04 +0000
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Sun, 13 Mar 2005, Arnt Gulbrandsen wrote:
>
> Right now, it's sensible to consider only the received fields before the first
> other field, and assume that any later fields have been processed by a
> nonstandard randomizer.
>
> The proposal seems to invalidate that simple heuristic.

It might make sense to define a subset of the header field name space
which can be used by these fields, by analogy with Resent-*.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
PORTPATRICK: STRONG OR GALE FORCE WEST WINDS IN HEBRIDES AND FAIR ISLE, WILL
DECREASE DURING MONDAY, WITH STRONG OR GALE FORCE EASTERLY WINDS THEN SETTING
IN ON TUESDAY.

Date: Sun, 13 Mar 2005 23:42:13 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Cc: Tony Finch <[email-masked]>, John Leslie <[email-masked]>
Sender: [email-masked]

Tony Finch writes:
> It might confuse software that analyses Received: fields, causing it 
> to fail to find all of them.

Yes.

> However there is plenty of software out there which already 
> interleaves other fields amongst Received: fields, so robust code 
> already has to cope with it.

Right now, it's sensible to consider only the received fields before the 
first other field, and assume that any later fields have been processed 
by a nonstandard randomizer.

The proposal seems to invalidate that simple heuristic.

Arnt

Date: Sun, 13 Mar 2005 21:57:55 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Sun, 13 Mar 2005, Bruce Lilly wrote:
>
> Any software that attempts to analyze Received fields is the work
> of a misguided author and is unlikely to be useful.

That about wraps it up for SpamAssassin then.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
VIKING: WEST 5 OR 6, BECOMING CYCLONIC 6 TO GALE 8, THEN NORTH OR NORTHWEST,
DECREASING 4 OR 5 LATER. WINTRY SHOWERS. MODERATE OR GOOD.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Sun, 13 Mar 2005 16:42:29 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Sun March 13 2005 15:09, Tony Finch wrote:

> >    OTOH, there _might_ be some existing MTA this would break, though I
> > can't think why...
> 
> It might confuse software that analyses Received: fields, causing it to
> fail to find all of them. However there is plenty of software out there
> which already interleaves other fields amongst Received: fields, so robust
> code already has to cope with it.

Any software that attempts to analyze Received fields is the work
of a misguided author and is unlikely to be useful.  RFC 1123 notes
that:
              Received: lines are primarily intended for humans tracing
              mail routes, primarily of diagnosis of faults.  See also
              the discussion under 5.3.7.
and
                   Received: fields of messages originating from other
                   environments may not conform exactly to RFC822.
                   However, the most important use of Received: lines is
                   for debugging mail faults, and this debugging can be
                   severely hampered by well-meaning gateways that try
                   to "fix" a Received: line.

Aside from the "other environments", there exist widely-deployed
agents that generate Received fields which do not conform to even
the very liberal syntax of RFC 2822.  E.g:
   Received: from mail pickup service by crm with Microsoft SMTPSVC;
         Fri, 4 Mar 2005 16:55:49 -0800

Date: Sun, 13 Mar 2005 16:41:04 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

Hector,

Some of the issues here are connected to the reasoning behind
wanting a Message Submission protocol.  Those have been kicked
to death already; I don't want to go there again.

To identify intent, rather than specific text,

	No non-gateway MTA should be changing the order of
	message headers, or even reading and interpreting the
	headers it received, unless it is acting in a "gateway"
	role and, then, only if necessary.

I don't know how to be more clear about that.

More below.

--On Saturday, 12 March, 2005 19:20 -0500 Hector Santos
<[email-masked]> wrote:

> 
> 
> ----- Original Message -----
> From: "John C Klensin" <[email-masked]>
> To: "John Leslie" <[email-masked]>; "Tony Finch" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, March 11, 2005 6:10 PM
> Subject: Re: (lack of) message header field ordering
> 
> 
>> What it breaks is at least one detector for bogus or
>> fraudulent mail.  If the rules of 2821 are followed, then
>> there is no mechanism by which headers can be injected into a
>> message other than at the beginning, making a message with
>> fields ordered as
>>...
>> a little dubious. Not nearly dubious enough, given the ways in
>> which things can be reordered, to reject or dump the message,
>> but perhaps worth a few points on an subjective probability
>> estimate that the message is at least partially faked.
> 
> Or perhaps the software is well, not too smart.

If what you mean by "smart" is that it reorders headers, then
your term "smart" is a synonym for "broken" or "non-conforming".
The text in 2821 that bars tampering with messages by MTAs that
are neither in "submission" nor in "final delivery" roles is as
clear as I knew how to make it.   But that text did not change
the clear intent of 821, just repeated it more strongly.

> I'm still trying to figure out in a pure 2821 sense, how (or
> when) would it be possible for a reordering can take place.

Really simple: never.

> Off hand,  the only reason I can see it happen is when the
> 2821 receiver has determine there is a missing element or a
> message lacking a RFC header.

2821 allows some flexibility for a submission MTA, but the
problems and issues with that were part of what motivated the
Message Submission Protocol.   Even then, the only reordering
should be to insert missing headers below the first Received
field.  There is no requirement or justification for reordering
existing fields.

> For example, depending on which is followed 822 or 2822, the
> receiver may determine there is no valid RFC header and hence
> it will add its own.  Or it may add individual required
> headers, i.e, Message-Id:   The latter is possibly a mode
> where an reordering is more possible.

Still should not require reordering.  MTAs that are not the
submission server should not even be looking at the headers to
determine that the information is missing.   The delivery MTA
may look (although it doesn't have to), but there is a strong
case to be made that, if it doesn't like what it receives, it
should bounce the message rather than trying to fix it up.  The
standard doesn't require that-- it encourages delivery on the
basis of the envelope only, i.e., not looking at the headers to
determine what might be missing.

> Which begs the question, what is the minimum requirement for
> an valid RFC header required by a x821 process?

There is an old controversy about this.  One interpretation is
that, unlike 2821 with MIME, 821 does not require any particular
message format at all and, in particular, does not require any
822-style headers.   Some people have taken the position for
many years that a header-free message was essentially required
for an implementation of SEND to make any sense at all.  

If one takes a strong version of the "MTAs don't look at the
headers" principle, and states your question as above, the
answer is "none".   Of course, one might then claim that
delivery from the delivery MTA into the mail store (or to the
receiving MUA) is not part of the "x821 process".

> For a MUA,  none.
> For a MTA,  Received: (before the server adds its own) + x822
> 
> I am not sure if this is optional for other servers, but ours,
> like most servers will accept a DATA block that does not
> contain a RFC header. Definitely a legacy issue.

See comments about 821 and SEND, above.

> So what logic should a x821 receiver have to determine whether
> there is a valid RFC header?

If it is not the final delivery MTA, it doesn't need any
"logic", because it should not be asking.

> Of course, starting with RFC 822 Section 4.1, the original
> requirement was:
> 
>     Date:
>     From:
>     Destination fields: (To: | CC: )
> 
> We jump to RFC 2822 Section 3.5, and the current minimum
> requirement is:
> 
>     Origination Date:     Date:
>     Originator Field(s):  From: | Sender: | Reply-To:

None of those requirements apply to the MTA.  Under 821, it was
clearly possible for a mail system to be 821-conforming without
being 822-conforming.  Under 2821, the rules are different, but
the MTA is not supposed to be acting in a police function.

> But I believe we do have a baseline requirement according to
> RFC x821.  A missing RFC header will cause the MSA to generate
> a RFC header for the transaction:
> 
>     Return-Path: <--- required by the MDA
>     Received:    <--- required by the MTA hop
>     Date:        <--- current time stamp
>     From:        <--- from x821.Mail From command
>     To:          <--- from x821.Rcpt To command
>     Message-Id: <--- generated by the host.
>     Sender:   <--- from x821.Mail From command (redundant)

Yes, or to bounce the broken message, which is what it is
supposed to do if it can't figure out, with considerable
authority, what should be in those fields.  It is doubtful that
Sender should be generated, and the MSA is forbidden to generate
Return-path unless it is somehow also the MDA (final delivery
MTA).

> This should conform with RFC 821/822 and RFC 2821/2822 as well.

Yes
 
> On a related note, I believe Bruce's new proposal which
> relaxes the From: 2822 requirement, will change the logic
> decision on how to fit it in.  If the MDA supports Bruce's
> proposal, then should the From: header not be added?  If
> Sender: is supported, then it should not be an issue from a
> 2821 standpoint.
> 
> However, at a minimum, for the sake of the gateway and the
> final MUA, the Date:, From: and To: should be made available
> too for presentation reasons. So this is another area where
> reordering may take place.

It does not require reordering.  It may require insertion below
the trace fields, or may not.

> These are some issues that I am currently researching simply
> because the RFC header is currently technically not required.
> 
> What's the BCP in this area?

In an MTA, do not tamper with headers, other than adding trace
fields, without explicit sender authorization.  If you must
tamper with the headers, do so as little as possible.

> Is this on topic, important? a non-issue?

Mostly a non-issue, I think.

      john

Date: Sun, 13 Mar 2005 20:09:58 +0000
From: Tony Finch <[email-masked]>
To: John Leslie <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Fri, 11 Mar 2005, John Leslie wrote:
> Tony Finch <[email-masked]> wrote:
> >
> > I'm proposing that the 2822 syntax should be extended to allow arbitrary
> > fields to be appended to the start of the header, in a similar manner to
> > Received: and Resent-* fields.
>
>    At first blush, this seems like a very good idea. There are a number
> of things which are trace-like; and forcing them into the non-trace area
> tends to confuse the ordering of the non-trace headers.
>
>    OTOH, there _might_ be some existing MTA this would break, though I
> can't think why...

It might confuse software that analyses Received: fields, causing it to
fail to find all of them. However there is plenty of software out there
which already interleaves other fields amongst Received: fields, so robust
code already has to cope with it.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER GERMAN BIGHT: WEST OR NORTHWEST 5 OR 6, OCCASIONALLY 7 LATER, BUT SOUTH
5 OR 6 IN EAST AT FIRST. WINTRY SHOWERS. MAINLY GOOD.

Date: Sat, 12 Mar 2005 17:33:58 -0500
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

At 09:22 -0500 on 03/11/2005, Bruce Lilly wrote about Re: Can 
reputable domains stop outgoing spam?:

>... and the fact that AOL is one of the largest senders of unsolicited
>commercial mail (floppy disks and CD-ROMs as well as email)

There is one MAJOR difference between the AoL unsolicited commercial 
mailing of floppy disks and CD-ROMs and that of email. In the Disk/CD 
case AoL not the Recipient are doing the paying the full cost for 
delivery (as with all "Junk Snailmail"). With Email, aside from the 
cost of the injection of the Email into the Internet Email System, it 
is the Recipient (or their ISP acting as their agent) who pays the 
cost of receipt of the email. This is the same as if the 
Floppy/CD-ROM came "Postage Due" and you had to pay (sight unseen) 
the USPS to accept the Floppy/CD-ROM.

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: (lack of) message header field ordering
Date: Sat, 12 Mar 2005 19:20:56 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: "John C Klensin" <[email-masked]>
To: "John Leslie" <[email-masked]>; "Tony Finch" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, March 11, 2005 6:10 PM
Subject: Re: (lack of) message header field ordering

> What it breaks is at least one detector for bogus or fraudulent
> mail.  If the rules of 2821 are followed, then there is no
> mechanism by which headers can be injected into a message other
> than at the beginning, making a message with fields ordered as
>
>    Return-path
>    Received
>    Received
>    To:
>    From:
>    Date:
>    Received
>    Received
>    MIME-version
>
> a little dubious. Not nearly dubious enough, given the ways in
> which things can be reordered, to reject or dump the message,
> but perhaps worth a few points on an subjective probability
> estimate that the message is at least partially faked.

Or perhaps the software is well, not too smart.

I'm still trying to figure out in a pure 2821 sense, how (or when) would it
be possible for a reordering can take place.

Off hand,  the only reason I can see it happen is when the 2821 receiver has
determine there is a missing element or a message lacking a RFC header.

For example, depending on which is followed 822 or 2822, the receiver may
determine there is no valid RFC header and hence it will add its own.  Or it
may add individual required headers, i.e, Message-Id:   The latter is
possibly a mode where an reordering is more possible.

Which begs the question, what is the minimum requirement for an valid RFC
header required by a x821 process?

For a MUA,  none.
For a MTA,  Received: (before the server adds its own) + x822

I am not sure if this is optional for other servers, but ours, like most
servers will accept a DATA block that does not contain a RFC header.
Definitely a legacy issue.

So what logic should a x821 receiver have to determine whether there is a
valid RFC header?

Of course, starting with RFC 822 Section 4.1, the original requirement was:

    Date:
    From:
    Destination fields: (To: | CC: )

We jump to RFC 2822 Section 3.5, and the current minimum requirement is:

    Origination Date:     Date:
    Originator Field(s):  From: | Sender: | Reply-To:

But I believe we do have a baseline requirement according to RFC x821.  A
missing RFC header will cause the MSA to generate a RFC header for the
transaction:

    Return-Path: <--- required by the MDA
    Received:    <--- required by the MTA hop
    Date:        <--- current time stamp
    From:        <--- from x821.Mail From command
    To:          <--- from x821.Rcpt To command
    Message-Id: <--- generated by the host.
    Sender:   <--- from x821.Mail From command (redundant)

This should conform with RFC 821/822 and RFC 2821/2822 as well.

On a related note, I believe Bruce's new proposal which relaxes the From:
2822 requirement, will change the logic decision on how to fit it in.  If
the MDA supports Bruce's proposal, then should the From: header not be
added?  If Sender: is supported, then it should not be an issue from a 2821
standpoint.

However, at a minimum, for the sake of the gateway and the final MUA, the
Date:, From: and To: should be made available too for presentation reasons.
So this is another area where reordering may take place.

These are some issues that I am currently researching simply because the RFC
header is currently technically not required.

What's the BCP in this area?

Is this on topic, important? a non-issue?

Thanks

---
Hector Santos, CTO
WINSERVER "Wildcat! Interactive Net Server"
http://www.winserver.com (support)
http://www.santronics.com

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Fri, 11 Mar 2005 19:40:51 -0500
Cc: John C Klensin <[email-masked]>
Sender: [email-masked]

On Fri March 11 2005 18:10, John C Klensin wrote:

> If the rules of 2821 are followed, then there is no
> mechanism by which headers can be injected into a message other
> than at the beginning, making a message with fields ordered as
>    Return-path 
>    Received
>    Received
>    To:
>    From:
>    Date:
>    Received
>    Received
>    MIME-version
> 
> a little dubious.

However,

    Return-path 
    Received
    Received
    Resent-To:
    Resent-From:
    Resent-Date:
    Received
    Received
    To:
    From:
    Date:
    MIME-version

is not at all dubious (ignoring the missing colons and empty field
bodies, of course).  I believe that's what's behind the 2822 ABNF
field grouping.

I believe, however, that 2821 section 3.8 and its subsections
provides a mechanism whereby all sorts of things, including
reordering of fields within the message header, can happen.

Date: Fri, 11 Mar 2005 18:10:59 -0500
From: John C Klensin <[email-masked]>
To: John Leslie <[email-masked]>, Tony Finch <[email-masked]>
cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

--On Friday, 11 March, 2005 12:36 -0500 John Leslie
<[email-masked]> wrote:

> Tony Finch <[email-masked]> wrote:
>> 
>> I'm proposing that the 2822 syntax should be extended to
>> allow arbitrary fields to be appended to the start of the
>> header, in a similar manner to Received: and Resent-* fields.
> 
>    At first blush, this seems like a very good idea. There are
> a number of things which are trace-like; and forcing them into
> the non-trace area tends to confuse the ordering of the
> non-trace headers.
> 
>    OTOH, there _might_ be some existing MTA this would break,
> though I can't think why...

What it breaks is at least one detector for bogus or fraudulent
mail.  If the rules of 2821 are followed, then there is no
mechanism by which headers can be injected into a message other
than at the beginning, making a message with fields ordered as
   Return-path 
   Received
   Received
   To:
   From:
   Date:
   Received
   Received
   MIME-version

a little dubious. Not nearly dubious enough, given the ways in
which things can be reordered, to reject or dump the message,
but perhaps worth a few points on an subjective probability
estimate that the message is at least partially faked.

  john

From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball
Date: Fri, 11 Mar 2005 12:41:08 -0500
Cc: [email-masked]
Sender: [email-masked]

On Fri March 11 2005 11:53, David MacQuigg wrote:

> I think they could take a small bit of initiative, without getting into 
> R&D

In case you don't know:  there is an Internet Research Task Force
(IRTF) and an Anti-Spam Research Group (ASRG) with a mailing list:
https://www1.ietf.org/mailman/listinfo/asrg

I'm not suggesting that you'll have any better results there, but
whining here (ietf-smtp) isn't going to do any good.

Date: Fri, 11 Mar 2005 12:36:21 -0500
From: John Leslie <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

Tony Finch <[email-masked]> wrote:
> 
> I'm proposing that the 2822 syntax should be extended to allow arbitrary
> fields to be appended to the start of the header, in a similar manner to
> Received: and Resent-* fields.

   At first blush, this seems like a very good idea. There are a number
of things which are trace-like; and forcing them into the non-trace area
tends to confuse the ordering of the non-trace headers.

   OTOH, there _might_ be some existing MTA this would break, though I
can't think why...

--
John Leslie <[email-masked]>

Date: Fri, 11 Mar 2005 17:10:56 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Fri, 11 Mar 2005, Bruce Lilly wrote:
>
> Then when *generating* a message with extension (i.e. not defined
> in [2]822) or user-defined (beginning with X-) fields, optional-field
> is what applies.
>
> When *parsing* a message, order should not be taken as significant
> (because it isn't).
>
> That leaves *modification* of an existing message, which is at best
> a gray area.

The latter is where the restrictions on the location of optional-field are
a problem, especially in conjunction with proposals like DomainKeys which
make (usually incorrect) assumptions about where fields are added. I'm
proposing that the 2822 syntax should be extended to allow arbitrary
fields to be appended to the start of the header, in a similar manner to
Received: and Resent-* fields.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NITON: STRONG WINDS AT TIMES IN ALL AREAS, OCCASIONALLY GALE FORCE IN FITZROY,
SOLE AND SHANNON ON TUESDAY.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Fri, 11 Mar 2005 11:54:08 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Fri March 11 2005 10:56, Tony Finch wrote:
> 
> On Fri, 11 Mar 2005, Bruce Lilly wrote:
> >
> > In light of the fact that ordering is not significant and
> > cannot be guaranteed, why does it matter -- how does:
> >
> >                         optional-field)
> >
> > from the same section not work?
> 
> Because when generating messages I want to be compliant with the
> specification. Robustness principle and all that.

Then when *generating* a message with extension (i.e. not defined
in [2]822) or user-defined (beginning with X-) fields, optional-field
is what applies.

When *parsing* a message, order should not be taken as significant
(because it isn't).

That leaves *modification* of an existing message, which is at best
a gray area.  One of the deficiencies of the messaging architecture
as it has evolved is that modification of the message header in
transit is really a layering violation; there really should be a
separate place (call it an "envelope", perhaps) where transport
agents (including list expanders, forwarders, etc.) could place
their markings, separate from the end-to-end, user-to-user message
content (header and body).  Something like that could be retrofitted
onto the existing architecture (by wrapping the user-to-user part
in a MIME message/rfc822 wrapper and (MIME) signing that to prevent
modification).

Date: Fri, 11 Mar 2005 09:53:17 -0700
To: Russ Allbery <[email-masked]>, [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Has the IETF dropped the ball
Sender: [email-masked]

At 07:38 PM 3/10/2005 -0800, Russ Allbery wrote:

>The IETF should be doing what the IETF has always done, namely provide a
>forum for publication of interoperable network standards, provide vetting
>by experienced standards authors to make sure that those standards are
>interoperable and work well with the rest of the Internet infrastructure,
>and document and standardize solutions that have been successful in the
>real world.

I think they could take a small bit of initiative, without getting into 
R&D, and without taking sides, by saying "Give us a proposal for just the 
parts all sides can live with."  That would not be a complete 
authentication system, but could allow the different systems to 
inter-operate, and most importantly, the non-combatants to start work on 
better spam filters, domain-rating lists, etc.

[snip]
>If you see some useful lower-level interoperable substrate that you think
>should be standardized independent of any specific system, then by all
>means write that up as an I-D so that people can see what it looks like
>and see if you can get the people who are working on the different systems
>to agree on it.
[snip]

I think writing a draft at this point would be a waste of time.  I took the 
first step by writing a set of fundamental requirements, ( 
ece.arizona.edu/~edatools/etc ) but I can't even get agreement on the 
question - should requirements be discussed before implementation 
details.  It looks to me like just about everyone in the technical 
community involved in this issue is pushing one or another system, and 
really doesn't want any compromise.  If you are neutral, you are against 
us.  Our side will win.  The other side doesn't have a chance, so why 
should we spend even five minutes thinking about how to solve some 
technical problem in a way that works for everyone.  Much better for our 
eventual victory if we use our expertise to shoot down any idea that helps 
the other side, even if it helps us too.

I see the result of this deadlock as at least another year of delay, and 
possibly eventual failure.  If the public and the technical community not 
directly involved in the issue does not have confidence in the eventual 
success of email authentication, then nobody will take the next step, and 
it won't happen.  Confidence could be built, while still providing room for 
competition, if we had a simple standard with such things as a standard 
form for an authentication header, or a standard query to get whatever 
authentication information is provided by a domain.

If the internet technical community doesn't provide a solution, the 
government will.  This is a lot less difficult politically than air 
pollution or broadcast regulation.  Similar problems have already been 
solved by government regulation ( Do Not Call, Junk faxes ).  The public is 
expecting spam-free email, and they will get it, even if it takes 500 pages 
of regulations governing the operation of public mail servers.

I welcome constructive criticism, and I will continue pursuing this project 
if it looks like it might eventually help with a solution.  I have no 
financial or intellectual "vested interest".  I don't even care about 
getting credit.  If some of what I propose gets copied into other 
proposals, that will save having to present the IETF with yet another draft.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Fri, 11 Mar 2005 15:56:07 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Fri, 11 Mar 2005, Bruce Lilly wrote:
>
> In light of the fact that ordering is not significant and
> cannot be guaranteed, why does it matter -- how does:
>
>                         optional-field)
>
> from the same section not work?

Because when generating messages I want to be compliant with the
specification. Robustness principle and all that.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
PORTPATRICK: STRONG TO GALE FORCE WINDS IN ALL AREAS AT TIMES, OCCASIONALLY
SEVERE GALE FORCE IN BAILEY, FAIR ISLE, FAEROES AND SOUTHEAST ICELAND AND
PERHAPS STORM FORCE IN FAEROES AND SOUTHEAST ICELAND ON MONDAY .

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Date: Fri, 11 Mar 2005 10:43:51 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Fri March 11 2005 09:33, Tony Finch wrote:

> I'm talking about the following part of the syntax, which gives the
> resent- fields a special status w.r.t. the trace fields, and does not
> allow anyone to extend the message format with new header fields which
> (like resent-*) appear amongst the Received: fields.
> 
> fields          =       *(trace
>                           *(resent-date /
>                            resent-from /
>                            resent-sender /
>                            resent-to /
>                            resent-cc /
>                            resent-bcc /
>                            resent-msg-id))
>                         *(orig-date /
>                           ...

In light of the fact that ordering is not significant and
cannot be guaranteed, why does it matter -- how does:

                        optional-field)

from the same section not work?

From: "Hector Santos" <[email-masked]>
To: "Glenn Anderson" <[email-masked]>, <[email-masked]>, "David MacQuigg" <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Date: Fri, 11 Mar 2005 10:24:01 -0500
Organization: Santronics Software, Inc.
Sender: [email-masked]

----- Original Message -----
From: "David MacQuigg" <[email-masked]>
To: "Glenn Anderson" <[email-masked]>; <[email-masked]>
Sent: Friday, March 11, 2005 8:10 AM
Subject: Re: Can reputable domains stop outgoing spam?

> I'm interested in what impairment AOL has suffered as a result of their
> policy to eliminate *outgoing* spam from *AOL systems*.  See
>
<http://www.circleid.com/article/917_0_1_0_C/>http://www.circleid.com/articl
e/917_0_1_0_C
> for a discussion of their policy and claims "Spam is a completely solvable
> problem." and "We do not send it anymore."
>

Hate to beat on the proverbial "dead horse"  but AOL, like small vendors
like us,  have seen the major benefits gained by tightening up the screws a
bit.   The problem is so large, it is clear the benefits outweigh the
initial pains anyone might feel.  Pains where a medicine fix exist.

With that said,  in our support experience, the "impairment"  is that it
puts pressure on ISPs to:

a) Clean up/properly setup the site (yes, SMTP compliant)
b) Add their own enforcements,
c) Get something that whitelist/authorized AOL machines, and
d) Get their own sites whitelisted with AOL.

We had two AOL related ISP incidents to work out in the past 2 days.  The
first was solved with C and the second one would probably be solved with D.
That incident is still open. (See below)

With the second incident, the ISP customer reported AOL blocked them, but
more importantly AOL hide/strip the recipient information in the DSN/Bounce
making it difficult for them to analyze the report.   I am still looking at
this one but told the customer to check out AOL's WhiteList policy stuff.

In  regards to SPF, until the transition domain change concept is
implemented across the board to handle the forwarding problem, such as by
using SUBMITTER,   the user's themselves might be impaired by enforce them
to not use aliases even in authorized environments.   The SUBMITTER and PRA
exemplifies this potential user privacy problem.

In case others are not aware of submitter,  since it is SMTP related,  I
will briefly describe it. This is an ESMTP MAIL FROM modifier which
addressed the SPF forwarding problem.   When the responsible domain changes
in relationship to the sender machine IP address,   the SUBMITTER can be
used to  satisfy SPF lookups.

If SPF sites do not have "forwarding mail" solution on both ends
(sending/receiving),  the MUA user will be restricted in alias name usage.

What is surprising to me is that there is a large ISP (bellsouth.net) who
thinks that might not be a bad idea in my discussions their recently
installed SPF policy-only ready site and how bellsouth.net users will not be
able to use aliases if the email is destined to a SPF ready site.

SUBMITTER can solve this, but the SPF-ready MDA must support it too.

--
Hector Santos
http://www.santronics.com

Date: Fri, 11 Mar 2005 14:33:52 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: (lack of) message header field ordering
Sender: [email-masked]

On Thu, 10 Mar 2005, Bruce Lilly wrote:
> On Thu March 10 2005 13:21, Tony Finch wrote:
>
> > Good point. I need a word to describe the set of fields that are permitted
> > to appear at the start of the message header interleaved with the trace
> > fields. At the moment this is only Resent- fields, which is obviously not
> > enough.
>
> Resent- fields are not required to be interleaved with other fields.

I'm talking about the following part of the syntax, which gives the
resent- fields a special status w.r.t. the trace fields, and does not
allow anyone to extend the message format with new header fields which
(like resent-*) appear amongst the Received: fields.

fields          =       *(trace
                          *(resent-date /
                           resent-from /
                           resent-sender /
                           resent-to /
                           resent-cc /
                           resent-bcc /
                           resent-msg-id))
                        *(orig-date /
                          ...

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ARDNAMURCHAN POINT TO CAPE WRATH INCLUDING THE OUTER HEBRIDES: NORTHWEST 5 TO
7, VEERING NORTH 6 TO GALE 8. RAIN THEN SHOWERS, TURNING WINTRY. GOOD OR
MODERATE. ROUGH TO VERY ROUGH BECOMING HIGH TO VERY HIGH.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Can reputable domains stop outgoing spam?
Date: Fri, 11 Mar 2005 09:22:02 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Fri March 11 2005 08:10, David MacQuigg wrote:

> <http://www.circleid.com/article/917_0_1_0_C/>http://www.circleid.com/article/917_0_1_0_C 

Well, if nothing else, the remark about "ISPs who don't even authenticate
their members" was humorous...

Much of the rest sounds like unwarranted puffery ... considering AOL's
spectacular failures:
http://www.usatoday.com/life/cyber/tech/2002/01/02/harvard-spam.htm
... and the fact that AOL is one of the largest senders of unsolicited
commercial mail (floppy disks and CD-ROMs as well as email).

Date: Fri, 11 Mar 2005 06:10:43 -0700
To: Glenn Anderson <[email-masked]>, [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

Still zeroing in on the question.  Sorry if this sounds like a 
cross-examination, but I want to make sure I'm not basing my whole strategy 
on a faulty assumption.

>>>>>>   Reputable domains will eliminate 99% of their outgoing spam, as 
>>>>>> AOL has done.
>>>>>
>>>>>this doesn't happen without impairing the ability of the mail system 
>>>>>to carry legitimate mail.
>>>>
>>>>Has AOL suffered any impairment?
>>>
>>>I have had a couple of reports of legitimate email being bounced due to 
>>>AOL using SPF. The mail was sent from an AOL user to an address on a 
>>>non-AOL server, and that address was set to forward to an AOL account.
>>
>>Do you know what caused the problem?  Some other replies have confused 
>>the issue with problems on the *incoming* side.  AOL is leaning on other 
>>domains to clean up their spam, and it may be that the forwarder has a problem.
>
>SPF breaks forwarding <http://spf.pobox.com/faq.html#forwarding>.
>
>>I'm interested in what impairment AOL has suffered as a result of their 
>>policy to eliminate *outgoing* spam.
>
>Outgoing spam from AOL systems, or outgoing spam from other systems with 
>AOL domains in the email addresses?
>
>If the former, what is their policy to eliminate outgoing outgoing spam 
>from their systems, and what are the relevant technical issues?

I'm interested in what impairment AOL has suffered as a result of their 
policy to eliminate *outgoing* spam from *AOL systems*.  See 
<http://www.circleid.com/article/917_0_1_0_C/>http://www.circleid.com/article/917_0_1_0_C 
for a discussion of their policy and claims "Spam is a completely solvable 
problem." and "We do not send it anymore."

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

To: [email-masked]
Subject: Re: Can reputable domains stop outgoing spam?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 10 Mar 2005 19:41:26 -0800
Sender: [email-masked]

David MacQuigg <[email-masked]> writes:

> Has AOL suffered any impairment?

Yes, absolutely.  AOL has thrown resources and clout behind their spam
filtering mechanism to make what they're doing work for them, but they're
doing that at a scale that very few other Internet sites would be able to
manage.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: Impossibility of Stopping Spam
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 10 Mar 2005 19:38:52 -0800
Sender: [email-masked]

David MacQuigg <[email-masked]> writes:

> Gentlemen, This sub-thread went off-topic long ago.  I like Dave
> Crocker's suggestion of starting a new thread, and keeping each one on a
> narrow topic.  The topic here is what should the IETF be doing?

The IETF should be doing what the IETF has always done, namely provide a
forum for publication of interoperable network standards, provide vetting
by experienced standards authors to make sure that those standards are
interoperable and work well with the rest of the Internet infrastructure,
and document and standardize solutions that have been successful in the
real world.

As near as I can tell, this is exactly what the IETF is doing.  Yes, there
is a great deal of chaos right now around spam within the IETF, with
multiple different solutions being proposed.  I submit that's partly
because, and for exactly the same reasons, that there is even more chaos
around spam outside the IETF.  It's a chaotic subject.  We don't know what
we're doing yet, for a very broad definition of "we."

If you see some useful lower-level interoperable substrate that you think
should be standardized independent of any specific system, then by all
means write that up as an I-D so that people can see what it looks like
and see if you can get the people who are working on the different systems
to agree on it.  But telling other people that they should go work on that
instead of what they're currently working on probably isn't going to
produce any useful results.  Maybe you're in a better position to see the
big picture than they are; I honestly don't know.  But if they felt like
that was the most productive approach, I expect they'd already be working
on it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Has the IETF dropped the ball?
Date:  Fri, 11 Mar 2005 04:14:26 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 72
Sender: [email-masked]

David MacQuigg wrote:

> Is this failure due to fundamentally incompatible methods,
> or due to a failure of the standards process?

Neither nor.  MARID found a clean way to solve this problem,
sp2-0/mfrom IsNot (pat.pend.) spf2.0/pra (pat. pend.)

spf2.0/mfrom,pra or spf2.0/pra,mfrom are allowed.  Of course
there are many arguments against PRA, e.g. what Bruce said
here about reordered Resent-* header fields.

> What if the standard said each method could have its own
> label and record format

They (MARID) came to this conclusion and then closed the WG.
The "bug" in the standards process is RfC 3710 chapter 4.3
(WG Termination) which does not reflect RfC 2418 chapter 4.

That's where the IETF dropped the ball, but there were a
lot of fouls before.  Later Sender-ID tried again to abuse
v=spf1 policies, and SPF said again "thanks but no thanks".

The LMAP methods like SPF, CSV, BATV, etc. were "off topic",
MARID wasted its time with XML-over-DNS and patent issues.
At least "mfrom IsNot pra" was clear in the last MARID days.

> If the dk1 folks are sensible, they won't make their record
> formats too difficult for an spf2 process to use.

The best DK and others can do is to stay away from this mine
field.  SPF can offer an op=dk to indicate "I always use DK"
in a sender policy, but that's not essential.

> Could all methods live within this standard?

Several independent non-conflicting standards, which could be
used simultaneously, sure, that's possible.  If you use CSV
and it says "bad", then you don't need SPF for a 2nd opinion.

The MTAs don't do these checks because they love to burn CPU
cycles on DNS servers, they want clear reasons to reject mails
a.s.a.p. (most mail is spam today).

> Spam Bounces in an IP-authenticated transfer must go back
> the path they came, not to some header address that might be
> forged.

Welcome to the <quote> quixotic quest </quote> of reviving
822 reverse paths disguised as SPF (formerly known as RMX).

> Is this something all sides could live with?

Depends, "bounces-to" mail-arch and "anti-phishing" PRA won't
acknowledge the underlying principle of a literal MAIL FROM.
They're free to ignore it, but is that "live with" ?

> Is this issue really something that must be worked out in a
> shared standard?

IMHO we can only tell it as it is, RfC 1123 5.3.6 (b) "alias"
is a part of the problem.  RfC 2476 6.1 "enforced submission
rights" is a good step forward, RfC 2821 7.1 is again a step
backwards, SPF is again a step forward, Sender-id backwards,
RfC 3834 chapter 4 is IMO forward, mail-arch "bounces-to" is
backwards, etc.

We'll end with mail as a minor Jabber-function if neither SPF
nor MASS fly.   CSV is IMHO a long shot, the clumsy SPF works
now (for those who want it).
                            Bye, Frank

Date: Fri, 11 Mar 2005 16:20:07 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

>>>Has AOL suffered any impairment?
>>
>>I have had a couple of reports of legitimate email being bounced 
>>due to AOL using SPF. The mail was sent from an AOL user to an 
>>address on a non-AOL server, and that address was set to forward to 
>>an AOL account.
>
>Do you know what caused the problem?

SPF breaks forwarding <http://spf.pobox.com/faq.html#forwarding>.

>I'm interested in what impairment AOL has suffered as a result of 
>their policy to eliminate *outgoing* spam.

Outgoing spam from AOL systems, or outgoing spam from other systems 
with AOL domains in the email addresses?

If the former, what is their policy to eliminate outgoing outgoing 
spam from their systems, and what are the relevant technical issues?

Glenn.

Date: Thu, 10 Mar 2005 20:03:11 -0700
To: Glenn Anderson <[email-masked]>, [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

At 01:54 PM 3/11/2005 +1300, Glenn Anderson wrote:

>>>>   Reputable domains will eliminate 99% of their outgoing spam, as AOL 
>>>> has done.
>>>
>>>this doesn't happen without impairing the ability of the mail system to 
>>>carry legitimate mail.
>>
>>Has AOL suffered any impairment?
>
>I have had a couple of reports of legitimate email being bounced due to 
>AOL using SPF. The mail was sent from an AOL user to an address on a 
>non-AOL server, and that address was set to forward to an AOL account.

Do you know what caused the problem?  Some other replies have confused the 
issue with problems on the *incoming* side.  AOL is leaning on other 
domains to clean up their spam, and it may be that the forwarder has a problem.

I'm interested in what impairment AOL has suffered as a result of their 
policy to eliminate *outgoing* spam.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Fri, 11 Mar 2005 13:54:28 +1300
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

>>>   Reputable domains will eliminate 99% of their outgoing spam, as 
>>>AOL has done.
>>
>>this doesn't happen without impairing the ability of the mail 
>>system to carry legitimate mail.
>
>Has AOL suffered any impairment?

I have had a couple of reports of legitimate email being bounced due 
to AOL using SPF. The mail was sent from an AOL user to an address on 
a non-AOL server, and that address was set to forward to an AOL 
account.

Glenn.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: (lack of) message header field ordering
Date: Thu, 10 Mar 2005 17:21:41 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Thu March 10 2005 13:21, Tony Finch wrote:

> Good point. I need a word to describe the set of fields that are permitted
> to appear at the start of the message header interleaved with the trace
> fields. At the moment this is only Resent- fields, which is obviously not
> enough.

Resent- fields are not required to be interleaved with other fields.
RFC 2822, at Proposed Standard, introduces a recommendation (N.B. not
a requirement) that Resent- fields, when used, be maintained in a block.
However, 2822 notes:

"  It is important to note that the header fields are not guaranteed to
   be in a particular order.  They may appear in any order, and they
   have been known to be reordered occasionally when transported over
   the Internet."

So a message-parsing application cannot justifiably assume that
fields appear in any particular order, or that fields have not been
reordered, both for the reasons noted in 2822 and because there exist
(a large number of diverse) applications that have not been revised to
conform to 2822.

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Date: Thu, 10 Mar 2005 15:19:21 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, March 10, 2005 2:23 PM
Subject: Re: Can reputable domains stop outgoing spam?

> 
> > Has AOL suffered any impairment?
> 
> their users certainly have.
> 

Where is the proof  Keith?  Do you have specifics?

I'm interested in your inherent knowledge so that it isn't repeated.

I think what I see here is a new term called "Mail Blabbing." 

Sincerely, 

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "ietf-smtp" <[email-masked]>
Subject: Re: one use for mail folders
Date: Thu, 10 Mar 2005 15:15:24 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "ietf-smtp" <[email-masked]>
Sent: Thursday, March 10, 2005 2:31 PM
Subject: one use for mail folders

> Just in case this is useful information for people in this discussion -
> there are a small number of individuals whose input has repeatedly and
> consistently been demonstrated to be of so little value (worse than
> most spam)

You mean like this stupid moronic input has yours?

Keith, stop trying get around it.

Show us the proof why SMTP compliancy will not work?

All this has meaningful value for vendors like us who build and sell total
mail and file systems.  I believe I provide insight and input.  If they
don't agree, please,  let me hear it from them, not you and I will quickly
stop wasting time in IETF mail forums.

But you on the other hand have indeed contributed nothing but 100% pessimism
in the entire process.   It doesn't take anyone that long to see what's real
or not here.  It is not hard to see how much of a moron you are..

To John:  John please, if you think I am totally off base, cut me out. I
have been looking for a reason. No one in their right mind have time for
this baloney.  I still have hope that people who have a real stake in all
this can get together and do something.  But I will not allow the twins:
Moores and Lilly ruin it for me or anyone for that matter.

I guess Keith, if anyone takes you up on your recommendation, then maybe I
don't really belong here.  It will no doubt show the sad state of affairs.

But it makes me wonder:

What is it that I am saying that is "wrong?"

What don't have any respect at all for my experience?

What did I do to you to make me hate me so much?

If you don't agree with me, find, but do you have to flame me and put me
down?

Is it my surname?  That I am Hispanic? I certainly hope not. I am as rotten
as a American you can get.  Boggle down South Bronx!  Born and raise!

If I am wrong, show me how I am wrong. Blabbing it out doesn't cut it. And
yes, you need to SPENT the time to show how I am wrong.  If you have
references, data, statistics, web sites that shows I am wrong, by all means,
please show it.

If not, then...... KISS MY ASS!

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: New Authenticated: header?
Date: Thu, 10 Mar 2005 15:16:41 -0500
Sender: [email-masked]

On Thu March 10 2005 13:25, David MacQuigg wrote:

> The key concept here is that these forgeries will be only in headers 
> *below* the authenticated header of the trusted forwarder.

Sigh.

Repeat until it sinks in:

"Header fields are NOT required to occur in any particular order"
[STD 11]

> >2. There has still been no concrete definition of precisely
> >    who or what is supposedly being authenticated, by whom,
> >    according to what criteria, or for what purpose.
> 
> These are implementation details

No, they are fundamentals.

> >First things first: first come up with a detailed definition for
> >the supposed "authentication", addressing the issues above.
> >
> >Then, if and only if there is some reasonable purpose, there
> >might be some point in discussing how to pass information from
> >point to point.
> 
> Fundamental requirements first, then details of specific proposed 
> implementations.

That's what I said.  So first state what it is that you intend
to authenticate, vs. what other information, who you propose
should perform the authentication and how, why, and what you
think it will accomplish.

There are few too many half-baked ideas that begin with "let's
use a header [field] to do ..."; this one looks like it's
completely unbaked.

Date: Thu, 10 Mar 2005 15:09:03 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

It's been proposed in a few different forums by a number of different 
people. Each version has had the problems that Tony Finch noted. I think 
the most recent example of this idea was posted to the [email-masked] 
list about 3 months ago. (No, I'm not going to go looking for it.) There 
was a similar proposal at the FTC summit (patent pending, no less!) last 
fall.

	Tony Hansen
	[email-masked]

Bruce Lilly wrote:
> A two-stage transfer whereby the message stays on the sender's host
> until retrieved by the recipient (who is sent some sort of
> notification that a message is pending) [such as one of Bernstein's
> schemes] might balance the desires of sender and recipient.
> However, that would be an entirely new architecture, has some
> serious drawbacks for mobile users and other users w/o full-time
> connectivity, and has not (to my knowledge) been written up even
> in draft form.

...

> I believe you are making unwarranted assumptions about a non-existent
> design of an unspecified protocol.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Spam Bounces - was What is Spam
Date: Thu, 10 Mar 2005 15:04:27 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Thu March 10 2005 14:31, David MacQuigg wrote:

> I think it is not a good idea to have users compose any message in response 
> to a spam.  Much better if they can just hit a "Reject as Spam" button,

"Forward", "Reply", whatever; it's a new message.

> Do we 
> agree that Spam Bounces should not go to the Return-Path?  Do we agree that 
> reputable forwarders should want to receive these bounces?

Evidently we're not agreed that they're not "bounces".

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 14:59:49 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

>> No.  It's not acceptable for any of these methods to use existing 
>> protocol elements in a way that contradicts either their definitions 
>> or their use in practice.
>
> If a method abuses an existing protocol or practice, seems to me it 
> will suffer in the competition with other methods that don't.

Tell that to the folks at Microsoft who deliberately violated the MIME 
specification's rules about presentation of unsafe content, and who 
have cost the network community several billion dollars by increasing 
their customers' vulnerabilities to viruses and worms.

> Please give me an example of something the IETF needs to make a 
> *requirement* for every authentication method.

I've already given one.

IETF's job is to write a set of rules that, if followed, will result in 
interoperation.  If our rules are not sufficient to do that, it's a 
flaw in our work that needs to be corrected.  If an authentication 
method causes legitimate use cases to break, it doesn't meet the 
requirements for standard.  However I do think that it's possible to 
tweak most of the proposals that exist now so that they don't break 
things.

Keith

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 14:50:54 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Tony Finch" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, March 10, 2005 2:04 PM
Subject: Re: Has the IETF dropped the ball?

> Spam is an inevitable consequence of
> accepting communication from people you don't already know.

That is the top cause, but remember there are two forms of acceptance:

    SMTP acceptance
    USER acceptance

Yuns choose to ignore the issues surrounding SMTP anonymous sender
acceptance. That isn't a social problem. Thats a technical issue.  Well,
maybe it is a social problem - an IETF social problem because isn't
certainly a user social problem or how did Keith put it? 'User desire" that
the SMTP process is a playing blind mute!

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Thu, 10 Mar 2005 11:59:12 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: [email-masked]
Subject: Re: Spam Bounces - was What is Spam
Sender: [email-masked]

David MacQuigg wrote:

> I think it is not a good idea to have users compose any message in 
> response to a spam.  Much better if they can just hit a "Reject as 
> Spam" button, and know that the "Spam Bounce" will be handled promptly 
> and correctly. 

All of the big end-user ISPs have this is some form or another, and the 
actions they take vary a lot. There are also non-ISP "this is spam" 
systems like Razor/Cloudmark and shared Bayesian systems, and of course 
there's SpamCop. There is absolutely no consensus among these about what 
"prompt and correct" handling means. For example, the mindset at AOL is 
to stop their users from receiving unwanted messages. The mindset of 
Razor/Cloudmark is to choke off spam messages from the whole community. 
The mindset of most SpamCop users is to get spammer's kicked off their 
hosting services.

AOL is driving an effort to standardize "this is spam" end-user 
feedback, based on the considerable experience they have with their 
existing FBL (feedback loop) mechanism. The BOF that AOL hosted at MAAWG 
was pretty firey -- lots of strong feelings on specific issues, and 
markedly different needs from different senders.

> Do we agree that Spam Bounces should not go to the Return-Path?  Do we 
> agree that reputable forwarders should want to receive these bounces? 

Nope, we don't. Handling of "spam bounces" is a matter of local policy 
and is an evolving subject.

And totally out of scope for the E-mail architecture document.

<csg>

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Thu, 10 Mar 2005 13:55:44 -0600
Subject: smtp mailing list activity
Sender: [email-masked]

Folks,

Howdy.

There are a series of threads underway on this list, right now, each appearing to have solved the problem of pertual motion.

This mailing list has a good history of productivity, so I thought it worth asking what is the productive basis for continuing the postings on those threads.

(And, yes, this is implicitly a suggestion that you stop posting, unless you are extremely clear that the posting has significant benefit to the population of list readers.)

 d/
 --
 Dave Crocker
 Brandenburg InternetWorking
 +1.408.246.8253
 dcrocker  a t ...
 WE'VE MOVED to:  www.bbiw.net

Date: Thu, 10 Mar 2005 12:48:07 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Impossibility of Stopping Spam
Sender: [email-masked]

> > > It won't solve the spam problem either; in fact it'll just make the
> > > spammer's job easier. Instead of wading through a load of spam messages,
> > > recipients will have to wade through a load of spam message 
> notifications.
> >
> > I believe you are making unwarranted assumptions about a non-existent
> > design of an unspecified protocol.
>
>Maybe about the details, but my main point stands. Throwing away SMTP and
>starting again (with IM2000, or with RSS) will not solve the spam problem:
>it will just move it about. (I say RSS because of Andy Newton's blog:
>http://hxr.us/blojsom/blog/grumpops/) Spam is an inevitable consequence of
>accepting communication from people you don't already know.

Gentlemen, This sub-thread went off-topic long ago.  I like Dave Crocker's 
suggestion of starting a new thread, and keeping each one on a narrow 
topic.  The topic here is what should the IETF be doing?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Thu, 10 Mar 2005 11:42:37 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: [email-masked]
Subject: Re: Can reputable domains stop outgoing spam?
Sender: [email-masked]

David MacQuigg wrote:

>> this doesn't happen without impairing the ability of the mail system 
>> to carry legitimate mail.
>
> Has AOL suffered any impairment? 

Considerably, yes. AOL's reliance at the router layer on the MAPS-DUL 
has resulted in blocked mail from legitimate senders using recently 
allocated IP addresses. AOL users accidentally clicking on the "this is 
spam" button -- and then trying to figure out why they aren't getting 
their bank statements -- has created a whole new line of business for 
help desks. If you are a small business and you want to be sure your 
customers get your mail, it's vital that you put yourself onto AOL's 
white lists; but the volume of mail coming back on AOL's feedback loops 
has been known to overwhelm small servers; when the legitimate business 
finds itself unable to deal with the FBL volume they get removed from 
the whitelist or blocked entirely. AOL's gentle (but real) threat to 
downgrade or block mail from ISPs that don't implement anti-zombie 
measures have sent some ISPs scrambling to implement port 25 blocking 
across the board, resulting in legitimate senders suddenly and without 
warning being unable to send E-mail anywhere.

These are all issues I discussed with live users at MAAWG last week. 
Plenty more where that comes from.

All of these problems do get resolved -- AOL is really good about 
working with legitimate senders to get the mail through -- but managing 
the "impairment" is a big job.

<csg>

Date: Thu, 10 Mar 2005 12:40:00 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

>
>>>SMTP should be "simple", but obviously we are unable to agree
>>>on a concept for MAIL FROM, the 1123 / 2821 fans insist on a
>>>"feature" where the 821 / SPF fans cry "bug".  "Sender-ID" was
>>>a foul compromise, beaten to death from both camps.  I could
>>>live with it if it would only leave v=spf1 alone.
>>
>>Is this issue really something that must be worked out in a shared 
>>standard?  Could we not leave it to each competing method exactly which 
>>header it chooses for the domain name to authenticate?
>
>No.  It's not acceptable for any of these methods to use existing protocol 
>elements in a way that contradicts either their definitions or their use 
>in practice.

If a method abuses an existing protocol or practice, seems to me it will 
suffer in the competition with other methods that don't.  You can be sure 
the fault will be pointed out by their competitors, and maybe even 
exaggerated in a FUD campaign.

Please give me an example of something the IETF needs to make a 
*requirement* for every authentication method.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: Keith Moore <[email-masked]>
Subject: one use for mail folders
Date: Thu, 10 Mar 2005 14:31:56 -0500
To: ietf-smtp <[email-masked]>
Sender: [email-masked]

Just in case this is useful information for people in this discussion - 
there are a small number of individuals whose input has repeatedly and 
consistently been demonstrated to be of so little value (worse than 
most spam) that anything they send to me is automatically filed into an 
appropriately-named folder rather than the folder which is assigned to 
this list.  I check this folder only when I see some list or another 
starting to rathole on a particular discussion and the rathole seems to 
have been caused by one of those individuals.

I find this facility quite useful, and I highly recommend it for those 
of you who aren't using it already.

Keith

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Can reputable domains stop outgoing spam?
Date: Thu, 10 Mar 2005 14:23:41 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

> Has AOL suffered any impairment?

their users certainly have.

Date: Thu, 10 Mar 2005 12:31:57 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Spam Bounces - was What is Spam
Sender: [email-masked]

At 12:24 PM 3/10/2005 -0500, Bruce Lilly wrote:

>On Wed March 9 2005 18:27, David MacQuigg wrote:
>
> > I was trying to move the discussion away from "what is spam" ( Irrelevant
> > question, in my opinion ) back to what to do with "Spam Bounces", assuming
> > the subjective judgement has already been made by the user.
>
>If a user has already seen the message, there is no "bounce" (a bounce
>would be a transport-related notification).  A user *might* elect to
>compose an entirely new message (e.g. to an abuse contact), but that
>situation (new messages) is already covered in the architecture draft.

I think it is not a good idea to have users compose any message in response 
to a spam.  Much better if they can just hit a "Reject as Spam" button, and 
know that the "Spam Bounce" will be handled promptly and correctly.  Do we 
agree that Spam Bounces should not go to the Return-Path?  Do we agree that 
reputable forwarders should want to receive these bounces?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 14:18:20 -0500
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> Spam is an inevitable consequence of accepting communication from 
> people you don't already know.

someone needs to set that in 72pt type and frame it.

Date: Thu, 10 Mar 2005 12:15:47 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Can reputable domains stop outgoing spam?
Sender: [email-masked]

>
>>   Reputable domains will eliminate 99% of their outgoing spam, as AOL 
>> has done.
>
>this doesn't happen without impairing the ability of the mail system to 
>carry legitimate mail.

Has AOL suffered any impairment?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "David MacQuigg" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Subject: What is the impact of enforcing SMTP compliancy?
Date: Thu, 10 Mar 2005 14:09:37 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I'm still waiting for your proof Keith.  Teach me please. Even "superior
minds" are always willing to learn, that is what makes us "superior"
especially from the 1 time wonder sage "Keith Moore"

In the mean time, here is our 1.5 years worth of proof:

        http://www.winserver.com/spamstats

If you can't show any proof showing the opposite, and I don't mean just
"blabbing it out", then maybe it is you who....how did you put it?

        "don't know what the hell they are talking about"

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

----- Original Message -----
From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>; "David MacQuigg" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Sent: Thursday, March 10, 2005 11:20 AM
Subject: Re: Do we need SMTP compliance?

>
>
> ----- Original Message -----
> From: "Keith Moore" <[email-masked]>
> To: "David MacQuigg" <[email-masked]>
> Cc: "ietf-smtp" <[email-masked]>
> Sent: Thursday, March 10, 2005 10:35 AM
> Subject: Re: Do we need SMTP compliance?
>
>
> >
> > > As I understand the arguments pro and con:
> > >
> > > Pro:  Rejecting messages with errors in the SMTP commands will
> > > eliminate most spam.
> > > Con:  Spammers can correct these errors, and will do so as soon as the
> > > need arises.
> > >
> > > I've probably missed something, but please restate the arguments
> > > (briefly), so we can start with the right understanding.
> >
> > The Pro argument isn't true.  The Con argument is.
> >
> > There's no need to have a discussion with people who don't know what
> > the hell they are talking about, and who are so convinced of their own
> > superior knowledge that they cannot learn anything.  Spam is not the
> > only kind of mail that should be discarded without reading it.
>
> Keith you have no proof in your statements.  You speak mainly on
conjecture.
> Show the proof in each?
>
>
> I have proof SMTP compliancy works, and it doesn't take much of a brain to
> realize that.
>
> The issue is the impact it will have and even if adaptation does occur, it
> can be a benefit, not a negative.
>
> You are showing nothing, Keith.
>
> Sincerely,
>
> Hector Santos, CTO
> Santronics Software, Inc.
> http://www.santronics.com
> 305-431-2846 Cell
> 305-248-3204 Office
>
>
>
>

Date: Thu, 10 Mar 2005 19:04:48 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

On Thu, 10 Mar 2005, Bruce Lilly wrote:
> On Thu March 10 2005 13:31, Tony Finch wrote:
>
> > It won't solve the spam problem either; in fact it'll just make the
> > spammer's job easier. Instead of wading through a load of spam messages,
> > recipients will have to wade through a load of spam message notifications.
>
> I believe you are making unwarranted assumptions about a non-existent
> design of an unspecified protocol.

Maybe about the details, but my main point stands. Throwing away SMTP and
starting again (with IM2000, or with RSS) will not solve the spam problem:
it will just move it about. (I say RSS because of Andy Newton's blog:
http://hxr.us/blojsom/blog/grumpops/) Spam is an inevitable consequence of
accepting communication from people you don't already know.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: EAST OR NORTHEAST 4, OCCASIONALLY 5 IN WEST, BECOMING VARIABLE 4.
FAIR. GOOD.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 13:58:29 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Thu March 10 2005 13:31, Tony Finch wrote:

> It won't solve the spam problem either; in fact it'll just make the
> spammer's job easier. Instead of wading through a load of spam messages,
> recipients will have to wade through a load of spam message notifications.

I believe you are making unwarranted assumptions about a non-existent
design of an unspecified protocol.

Date: Thu, 10 Mar 2005 18:31:41 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

On Thu, 10 Mar 2005, Bruce Lilly wrote:
>
> A two-stage transfer whereby the message stays on the sender's host
> until retrieved by the recipient (who is sent some sort of
> notification that a message is pending) [such as one of Bernstein's
> schemes] might balance the desires of sender and recipient.

It won't solve the spam problem either; in fact it'll just make the
spammer's job easier. Instead of wading through a load of spam messages,
recipients will have to wade through a load of spam message notifications.
Spammers will concentrate the important part of their advert into the
notification instead of the message, in the manner of bluetooth spam and
spim. They can just spew UDP which is much cheaper than SMTP and much
easier to forge. DNS blacklists would become useless.

The problem is not network bandwidth or disk space or CPU, the problem is
people's time. IM2000 optimises the former 3 at the expense of the latter.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HEBRIDES BAILEY: WEST VEERING NORTHWEST 6 TO GALE 8 DECREASING 4 OR 5 IN WEST
BAILEY. RAIN THEN SHOWERS. MODERATE OR GOOD.

Date: Thu, 10 Mar 2005 11:25:49 -0700
To: [email-masked], [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: New Authenticated: header?
Cc: John Leslie <[email-masked]>
Sender: [email-masked]

At 12:14 PM 3/10/2005 -0500, Bruce Lilly wrote:

>On Wed March 9 2005 16:18, John Leslie wrote:
>
> >    If there were a header prepended at the time Authentication is done,
> > it would make it possible to use its result as input to (possibly
> > heuristic) filtering executed later; and _might_ open the door to a
> > future in which a limited trust could be given to Received: lines after
> > the first.
> >
> >    What do other folks think?
>
>Some observations:
>1. Message header fields can be trivially forged.

The key concept here is that these forgeries will be only in headers 
*below* the authenticated header of the trusted forwarder.  The 
authentication results passed on by that forwarder could be very useful in 
spam filtering.

>2. There has still been no concrete definition of precisely
>    who or what is supposedly being authenticated, by whom,
>    according to what criteria, or for what purpose.

These are implementation details, and are different for each authentication 
method.  For now, I'm focused on the fundamental requirements shared by all 
methods.

5) If the message is forwarded, the results of authentication must be made
available to all subsequent receiving MTAs.  The format should be simple and
standardized to facilitate later blocking and filtering.

I do have a suggested implementation, appropriate for a chain-of-trust 
authentication method, and it may help clarify this discussion, but I don't 
want to get bogged down in debate over implementation details.  John's 
suggestion could work equally well.
http://www.ece.arizona.edu/~edatools/etc/Email%20Forwarding%20Protocol.htm 
- Implementation Examples (5)

>First things first: first come up with a detailed definition for
>the supposed "authentication", addressing the issues above.
>
>Then, if and only if there is some reasonable purpose, there
>might be some point in discussing how to pass information from
>point to point.

Fundamental requirements first, then details of specific proposed 
implementations.

Does the above requirement capture what needs to be done in forwarding a 
message?  Does it require too little? e.g. missing something that cannot be 
left to the specific authentication methods?  Does it require too much? 
e.g. Does it place an unnecessary burden on an end-to-end authentication 
method like DomainKeys?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Thu, 10 Mar 2005 11:25:49 -0700
To: [email-masked], [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: New Authenticated: header?
Cc: John Leslie <[email-masked]>
Sender: [email-masked]

At 12:14 PM 3/10/2005 -0500, Bruce Lilly wrote:

>On Wed March 9 2005 16:18, John Leslie wrote:
>
> >    If there were a header prepended at the time Authentication is done,
> > it would make it possible to use its result as input to (possibly
> > heuristic) filtering executed later; and _might_ open the door to a
> > future in which a limited trust could be given to Received: lines after
> > the first.
> >
> >    What do other folks think?
>
>Some observations:
>1. Message header fields can be trivially forged.

The key concept here is that these forgeries will be only in headers 
*below* the authenticated header of the trusted forwarder.  The 
authentication results passed on by that forwarder could be very useful in 
spam filtering.

>2. There has still been no concrete definition of precisely
>    who or what is supposedly being authenticated, by whom,
>    according to what criteria, or for what purpose.

These are implementation details, and are different for each authentication 
method.  For now, I'm focused on the fundamental requirements shared by all 
methods.

5) If the message is forwarded, the results of authentication must be made
available to all subsequent receiving MTAs.  The format should be simple and
standardized to facilitate later blocking and filtering.

I do have a suggested implementation, appropriate for a chain-of-trust 
authentication method, and it may help clarify this discussion, but I don't 
want to get bogged down in debate over implementation details.  John's 
suggestion could work equally well.
http://www.ece.arizona.edu/~edatools/etc/Email%20Forwarding%20Protocol.htm 
- Implementation Examples (5)

>First things first: first come up with a detailed definition for
>the supposed "authentication", addressing the issues above.
>
>Then, if and only if there is some reasonable purpose, there
>might be some point in discussing how to pass information from
>point to point.

Fundamental requirements first, then details of specific proposed 
implementations.

Does the above requirement capture what needs to be done in forwarding a 
message?  Does it require too little? e.g. missing something that cannot be 
left to the specific authentication methods?  Does it require too much? 
e.g. Does it place an unnecessary burden on an end-to-end authentication 
method like DomainKeys?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Thu, 10 Mar 2005 18:21:36 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: New Authenticated: header?
Sender: [email-masked]

On Thu, 10 Mar 2005, Bruce Lilly wrote:
> On Wed March 9 2005 17:25, Tony Finch wrote:
>
> > It's probably worth doing. However 2822 needs to be revised so that the
> > set of valid trace header fields is extended to more than Received: and
> > Resent-*.
>
> Resent- fields are not trace fields.  Return-Path is a trace field.

Good point. I need a word to describe the set of fields that are permitted
to appear at the start of the message header interleaved with the trace
fields. At the moment this is only Resent- fields, which is obviously not
enough.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FITZROY: EAST 4 OR 5, OCCASIONALLY 6 AT FIRST. RAIN IN SOUTH. MAINLY GOOD.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 12:32:57 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Wed March 9 2005 22:03, Keith Moore wrote:

> > The key problems are social, not technical.
> 
> while I don't disagree, that's not a helpful statement.  spam is a 
> social problem because it involves a conflict between the desire of the 
> sender and the desire of the recipient.  but because the two are at a 
> distance, you can't solve the problem by purely social means.  nor can 
> you solve the problem by purely technical means, or purely legal means. 

A two-stage transfer whereby the message stays on the sender's host
until retrieved by the recipient (who is sent some sort of
notification that a message is pending) [such as one of Bernstein's
schemes] might balance the desires of sender and recipient.
However, that would be an entirely new architecture, has some
serious drawbacks for mobile users and other users w/o full-time
connectivity, and has not (to my knowledge) been written up even
in draft form.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Spam Bounces - was What is Spam
Date: Thu, 10 Mar 2005 12:24:23 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Wed March 9 2005 18:27, David MacQuigg wrote:

> I was trying to move the discussion away from "what is spam" ( Irrelevant 
> question, in my opinion ) back to what to do with "Spam Bounces", assuming 
> the subjective judgement has already been made by the user.

If a user has already seen the message, there is no "bounce" (a bounce
would be a transport-related notification).  A user *might* elect to
compose an entirely new message (e.g. to an abuse contact), but that
situation (new messages) is already covered in the architecture draft.

Abuse handlers *might* get together and define a structured format for
such messages based on the RFC 3462 multipart/report MIME media type
(as used by DSNs, MDNs, and MTSNs), but they haven't.  And that would
be outside of the scope of the architecture document.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: New Authenticated: header?
Date: Thu, 10 Mar 2005 12:16:04 -0500
Cc: Tony Finch <[email-masked]>
Sender: [email-masked]

On Wed March 9 2005 17:25, Tony Finch wrote:
 
> It's probably worth doing. However 2822 needs to be revised so that the
> set of valid trace header fields is extended to more than Received: and
> Resent-*.

Resent- fields are not trace fields.  Return-Path is a trace field.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: New Authenticated: header?
Date: Thu, 10 Mar 2005 12:14:27 -0500
Cc: John Leslie <[email-masked]>, David MacQuigg <[email-masked]>
Sender: [email-masked]

On Wed March 9 2005 16:18, John Leslie wrote:

>    If there were a header prepended at the time Authentication is done,
> it would make it possible to use its result as input to (possibly
> heuristic) filtering executed later; and _might_ open the door to a
> future in which a limited trust could be given to Received: lines after
> the first. 
> 
>    What do other folks think?

Some observations:
1. Message header fields can be trivially forged.
2. There has still been no concrete definition of precisely
   who or what is supposedly being authenticated, by whom,
   according to what criteria, or for what purpose.

First things first: first come up with a detailed definition for
the supposed "authentication", addressing the issues above.

Then, if and only if there is some reasonable purpose, there
might be some point in discussing how to pass information from
point to point.

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "David MacQuigg" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Subject: Re: Do we need SMTP compliance?
Date: Thu, 10 Mar 2005 11:20:48 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Sent: Thursday, March 10, 2005 10:35 AM
Subject: Re: Do we need SMTP compliance?

>
> > As I understand the arguments pro and con:
> >
> > Pro:  Rejecting messages with errors in the SMTP commands will
> > eliminate most spam.
> > Con:  Spammers can correct these errors, and will do so as soon as the
> > need arises.
> >
> > I've probably missed something, but please restate the arguments
> > (briefly), so we can start with the right understanding.
>
> The Pro argument isn't true.  The Con argument is.
>
> There's no need to have a discussion with people who don't know what
> the hell they are talking about, and who are so convinced of their own
> superior knowledge that they cannot learn anything.  Spam is not the
> only kind of mail that should be discarded without reading it.

Keith you have no proof in your statements.  You speak mainly on conjecture.
Show the proof in each?

I have proof SMTP compliancy works, and it doesn't take much of a brain to
realize that.

The issue is the impact it will have and even if adaptation does occur, it
can be a benefit, not a negative.

You are showing nothing, Keith.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Cc: [email-masked], David MacQuigg <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Do we need SMTP compliance?
Date: Thu, 10 Mar 2005 11:13:54 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>>> Pro:  Rejecting messages with errors in the SMTP commands will 
>>> eliminate most spam.
>>
>> The Pro argument isn't true.
>
> Have a look at http://archive.iecc.com/article/spamtools/20050223008 
> or any page about greylisting. Even if the spammers' commands are 
> correct, other aspects of their SMTP implementations appear to have 
> certain defects.

no doubt.  but that doesn't mean that stricter checking will eliminate 
most spam.

Date: Thu, 10 Mar 2005 16:53:52 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Do we need SMTP compliance?
Cc: David MacQuigg <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

>> Pro:  Rejecting messages with errors in the SMTP commands will 
>> eliminate most spam.
>
> The Pro argument isn't true.

Have a look at http://archive.iecc.com/article/spamtools/20050223008 or 
any page about greylisting. Even if the spammers' commands are correct, 
other aspects of their SMTP implementations appear to have certain 
defects.

Arnt

Cc: ietf-smtp <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Do we need SMTP compliance?
Date: Thu, 10 Mar 2005 10:35:23 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

> As I understand the arguments pro and con:
>
> Pro:  Rejecting messages with errors in the SMTP commands will 
> eliminate most spam.
> Con:  Spammers can correct these errors, and will do so as soon as the 
> need arises.
>
> I've probably missed something, but please restate the arguments 
> (briefly), so we can start with the right understanding.

The Pro argument isn't true.  The Con argument is.

There's no need to have a discussion with people who don't know what 
the hell they are talking about, and who are so convinced of their own 
superior knowledge that they cannot learn anything.  Spam is not the 
only kind of mail that should be discarded without reading it.

Date: Thu, 10 Mar 2005 08:18:07 -0700
To: "Hector Santos" <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Do we need SMTP compliance?
Cc: <[email-masked]>
Sender: [email-masked]

At 08:04 AM 3/10/2005 -0500, Hector Santos wrote:

>"We build up whole cultural intellectual patterns based on past 'facts'
>which are extremely selective. When a new fact comes in that does not fit
>the pattern we don't throw out the pattern. We throw out the fact. A
>contradictory fact has to keep hammering and hammering and
>hammering,sometimes for centuries, before maybe one or two people see it.
>And then these one or two have to start hammering on others for a long time
>before they see it too . . . Seeing is not believing. Believing is seeing."
>'
>                       "Zen And The Art Of Motorcycle Maintenance"

I like the quote.  We've got to move this discussion off the other threads, 
however.  So here is a new thread where we can discuss all issues related 
to SMTP compliance.

As I understand the arguments pro and con:

Pro:  Rejecting messages with errors in the SMTP commands will eliminate 
most spam.
Con:  Spammers can correct these errors, and will do so as soon as the need 
arises.

I've probably missed something, but please restate the arguments (briefly), 
so we can start with the right understanding.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Cc: Frank Ellermann <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 09:34:31 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

>> SMTP should be "simple", but obviously we are unable to agree
>> on a concept for MAIL FROM, the 1123 / 2821 fans insist on a
>> "feature" where the 821 / SPF fans cry "bug".  "Sender-ID" was
>> a foul compromise, beaten to death from both camps.  I could
>> live with it if it would only leave v=spf1 alone.
>
> Is this issue really something that must be worked out in a shared 
> standard?  Could we not leave it to each competing method exactly 
> which header it chooses for the domain name to authenticate?

No.  It's not acceptable for any of these methods to use existing 
protocol elements in a way that contradicts either their definitions or 
their use in practice.

From: "Hector Santos" <[email-masked]>
To: "Frank Ellermann" <[email-masked]>, <[email-masked]>, "David MacQuigg" <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 08:19:43 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "David MacQuigg" <[email-masked]>
To: "Frank Ellermann" <[email-masked]>; <[email-masked]>
Sent: Thursday, March 10, 2005 6:55 AM
Subject: Re: Has the IETF dropped the ball?

> I'm talking about Spam Bounces, not SPF Fails.  I don't see a problem with
> authentication failures generating an immediate reject.  An SPF reject
> should be treated like a one-time failure of the communication equipment,
> and fixed promptly.  Mis-routed Spam Bounces are an ongoing problem,
however.
>
> Here is what I have so far in my Email Forwarding Protocol
> http://www.ece.arizona.edu/~edatools/etc/Email%20Forwarding%20Protocol.htm
> Fundamental Requirements
> 7) Spam Bounces in an IP-authenticated transfer must go back the path they
> came,
> not to some header address that might be forged.  Forwarders must accept
Spam
> Bounces resulting from messages they have forwarded within a reasonable
time,
> and must attempt to forward these bounces to the next upstream domain.
>
> Is this something all sides could live with?

This is perfect illustration of the administrative vs developer conflicts
that exist here, and in all other IETF areas that is addressing the "spam
issue."

You are thinking POST-SMTP processing as a bounce is only generated in ths
matter.

SORBIG-generation based hackers will lick their chops for sites running your
protocol. They will home in and seek your sites.

The concentration should be in eliminating the need for the bounce in the
first place - that is SMTP level.  Not POST-SMTP.

Until these key differences are understood,  we will continue to not get any
significant progress made.

Just look at this way:

If I have to put in resources dollars to change software to address 2822
analysis, what logic do you have that isn't going to work at 2821?

The bounce gos to the return path defined at 2821.  So why do I need to read
the 2822 to get to the same conclusion you are a "bad sender."

What is the return path is invalid?  You just increased the outbound queue
overhead to send the "get no where" bounce.

>> .... SENDERID...

> Is this issue really something that must be worked out in a shared
> standard?

The only share standard that can exist is a SMTP compliancy understanding.

Just look it this way:

If SMTP compliancy is not a requirement, then why should a spammer adapt?
He has no reason to do so if host systems must still honor their
non-compliant transaction.

> Could we not leave it to each competing method exactly which
> header it chooses for the domain name to authenticate?  There is even room
> here for a patented algorithm!!  Just not in the shared standard.

How does this apply to SMTP?

Are you suggesting some ESMTP attribute exposure of a requirement in the
2822 header in order to proceed?

What is the "header" doesn't exist?

Should the SMTP process accept just to check the header?

Should this be done at the DATA stage?

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Thu, 10 Mar 2005 08:04:33 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, March 09, 2005 6:39 PM
Subject: Re: Has the IETF dropped the ball?

> That is my view of the future, anyway.  For the topic of this thread,
> however, it doesn't matter if I am wrong on this.  If all we get is the
> elimination of phishing scams, that is reason enough to move ahead, and
> work out a standard that everyone can live with.
>
> -- Dave

My view is that you, among many here, are missing the boat of the real
problem.

I think this says it best:

"We build up whole cultural intellectual patterns based on past 'facts'
which are extremely selective. When a new fact comes in that does not fit
the pattern we don't throw out the pattern. We throw out the fact. A
contradictory fact has to keep hammering and hammering and
hammering,sometimes for centuries, before maybe one or two people see it.
And then these one or two have to start hammering on others for a long time
before they see it too . . . Seeing is not believing. Believing is seeing."
'
                      "Zen And The Art Of Motorcycle Maintenance"

What is the fact?

     "60-80% of the transactions are non-SMTP compliant transactions"

We choose to ignore this fact because we still wish to allow non-SMTP
compliant senders to exist: The backward SMTP compatibility dilemma.

Every idea will work to a "degree" but it is completely useless if you still
have to deal with the non-compliant sender.   Note I said, sender, not
spammer.  That is the key.  SMTP must view all the same - even if you are
not spammer, if you are non-SMTP compliant, you need to be rejected.  That
is the only way will get adaptation.  Only with chaos comes change, and
hence a new stability.

A few years back, that would be an unrealistic situation.

Today, it is more realistic.  Legitimate senders with poor setups know today
what is at stake.  For many, it will be a simple matter of a configuration
change.  For others, they might need to update their software.   Those who
resist, well, they will feel the pain.  The point is, there is no reason to
resist.  They know what is at stake.

What makes it even more plausible is this a very small percentage.  Today,
most people do run good setups.  The only segment, the largest one, who will
not complain to you, is the 60-80% of the senders with malicious intentions
in mind.

We all know what is at stake here. If reasonable solution existed, it would
of been done.

But it is the legacy and non-SMTP issue that is really keeping all this from
becoming a reality.

It is a mindset issue among the cogs that we need to change. I've been in
the mail business since atleast 81 and we always knew there was this SMTP
loophole.  But it wasn't deemed a big issue then and wasn't a big issue
until atleast 1992-94 and even then the mindset was still ignorant of the
nature of the problem.  Prior to 1992-94, most people was associated with an
dial up ISP at relatively low speeds.  The damage was extremely limited.
That has changed.

Mind you. I'm was part of the stubborn mindset too. My philosophy was that
it was an administrative policy to control problematic mail.  We just
delivered it to you.  I didn't do anything until I was personally affected
by the SORBIG eVirus in 2003, not on my system, but on other systems,
falsifying my identity.  That is when I said, enough is enough.  Something
had to be done at SMTP.

You really didn't see the big efforts begin until SORBIG exploited every
aspect of the SMTP process and also exploited the fact there is a vast
network of legacy SMTP sites that are still thinking in UUCP/SLIP mode with
continuing their POST SMTP/Gateway processing.

Most modern systems have move much of the gateway validation process into
the SMTP level and those system that use SMTP validation have a greater
protection from the SORBIG like attacks.

So if you want to champion a cause to get something going,  you will need to
address the legacy/non-compliant SMTP fact that is being thrown out by the
IETF cogs because it conflicts with the old patterns, a pattern that really
doesn't exist any more or isn't fixable or isn't going to be fixed today

This is very lost cost, first level approach that will have a significant
impact in helping secure the SMTP transaction.  Once it is the norm, the new
email authentication/authorization proposals will be easier to implement
with greater reliability.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Thu, 10 Mar 2005 04:55:57 -0700
To: Frank Ellermann <[email-masked]>, [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

At 04:49 AM 3/10/2005 +0100, Frank Ellermann wrote:

>David MacQuigg wrote:
>
> > a few common items for SPF and SenderID that share a need
> > to communicate through a chain of forwarders.
>
>"Sender-ID" requires a worldwide upgrade of many mailing lists
>and news servers (for moderated NGs), it's somewhat beside the
>point of RfC 3834 chapter 4, in other words it's a FUSSP with
>the decent charme of C/R-systems, and it breaks many published
>v=spf1 policies by a SHOULD, now obsoleted by a NOT RECOMMENDED
>in the SPF draft.
>
>I do not believe in worldwide upgrades, it's against my KISS
>religion.  And I do not believe in a NOT RECOMMENDED SHOULD.

Is this failure due to fundamentally incompatible methods, or due to a 
failure of the standards process?  What if the standard said each method 
could have its own label and record format ( v=spf2, v=sidf1, v=dk1, etc. 
).  A single standard query would return whatever authentication 
information was provided by the domain in question.  If a forwarder 
supports only spf2, and the domain provides only dk1, the authentication 
result passed on by that forwarder would be "None".  Alternatively, the 
spf2 process could try and do the best it can with the dk1 information it 
is given.  If the dk1 folks are sensible, they won't make their record 
formats too difficult for an spf2 process to use.  In fact, if the standard 
is done right, there will be no incentive to make things difficult at this 
point, and the warring camps might actually put some of their technical 
genius on the task of making things more compatible, not less.

Could all methods live within this standard?  This is a very simple 
question.  Please don't respond with a flood of technical detail.

> > We just need to provide a common protocol to handle the
> > results of a Recipient having decided something is spam.
>
>An SPF FAIL is not necessarily spam.  As long as the receiver
>rejects all FAILs the sender can check what's going on.

I'm talking about Spam Bounces, not SPF Fails.  I don't see a problem with 
authentication failures generating an immediate reject.  An SPF reject 
should be treated like a one-time failure of the communication equipment, 
and fixed promptly.  Mis-routed Spam Bounces are an ongoing problem, however.

Here is what I have so far in my Email Forwarding Protocol
http://www.ece.arizona.edu/~edatools/etc/Email%20Forwarding%20Protocol.htm
Fundamental Requirements
7) Spam Bounces in an IP-authenticated transfer must go back the path they 
came,
not to some header address that might be forged.  Forwarders must accept Spam
Bounces resulting from messages they have forwarded within a reasonable time,
and must attempt to forward these bounces to the next upstream domain.

Is this something all sides could live with?

> > My frustration is partly due to the lack of progress on what
> > I see as simple problems.
>
>SMTP should be "simple", but obviously we are unable to agree
>on a concept for MAIL FROM, the 1123 / 2821 fans insist on a
>"feature" where the 821 / SPF fans cry "bug".  "Sender-ID" was
>a foul compromise, beaten to death from both camps.  I could
>live with it if it would only leave v=spf1 alone.

Is this issue really something that must be worked out in a shared 
standard?  Could we not leave it to each competing method exactly which 
header it chooses for the domain name to authenticate?  There is even room 
here for a patented algorithm!!  Just not in the shared standard.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Has the IETF dropped the ball?
Date:  Thu, 10 Mar 2005 04:49:39 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 46
Sender: [email-masked]

David MacQuigg wrote:

> I agree that IETF shouldn't be doing the R&D.

The ASRG of the IRTF tried, one of the submitted papers was
http://www.simpleweb.org/ietf/internetdrafts/complete/draft-irtf-asrg-lmap-discussion-01.txt

That was one of the sources for the later MARID WG.  It was
crippled by the publication of Caller-ID and later the mixture
of Caller-ID and SPF styling itself as "Sender-ID".

> a few common items for SPF and SenderID that share a need
> to communicate through a chain of forwarders.

"Sender-ID" requires a worldwide upgrade of many mailing lists
and news servers (for moderated NGs), it's somewhat beside the
point of RfC 3834 chapter 4, in other words it's a FUSSP with
the decent charme of C/R-systems, and it breaks many published
v=spf1 policies by a SHOULD, now obsoleted by a NOT RECOMMENDED
in the SPF draft.

I do not believe in worldwide upgrades, it's against my KISS
religion.  And I do not believe in a NOT RECOMMENDED SHOULD.

> Notice I didn't say IETF should pick a winner right now
> between SPF, SenderID, and DomainKeys.

So far they picked RfC 3865.  If that convinces bureaucrats
that a subject tag [adv] is not the FUSSP it's not so bad as I
thought.

> We just need to provide a common protocol to handle the
> results of a Recipient having decided something is spam.

An SPF FAIL is not necessarily spam.  As long as the receiver
rejects all FAILs the sender can check what's going on.

> My frustration is partly due to the lack of progress on what
> I see as simple problems.

SMTP should be "simple", but obviously we are unable to agree
on a concept for MAIL FROM, the 1123 / 2821 fans insist on a
"feature" where the 821 / SPF fans cry "bug".  "Sender-ID" was
a foul compromise, beaten to death from both camps.  I could
live with it if it would only leave v=spf1 alone.  Bye, Frank

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 22:03:05 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

>> Spam is a hard problem.  If we knew of a good solution, we'd be using 
>> it.  The government wouldn't have to mandate the solution, as there 
>> are plenty of incentives already.  There are lots of half-baked 
>> non-solutions and a few good ideas that raise the bar for spam 
>> without actually stopping it.
>>
>> Authentication methods will not solve the spam problem.  They may 
>> make phishing harder, which is a good thing.   They might be useful 
>> when combined with some other facilities, but nobody understand what 
>> those facilities are.  Authentication methods are also no better than 
>> the hosts that people use to submit mail.  So if you want to reduce 
>> spam by requiring authentication, you first need to figure out how to 
>> make Windows secure and to get that secure version deployed 
>> everywhere.
>
> I'm more optimistic.  Authentication will allow us to hold domains 
> responsible for their outgoing spam.

no, it will just change the kind of spam we get.  the notion of "what 
is spam" will become grayer, but that doesn't mean there will be less 
of it.  for every nigerian scan mail that is eliminated, there will be 
an ivory soap spam to replace it.

>   Reputable domains will eliminate 99% of their outgoing spam, as AOL 
> has done.

this doesn't happen without impairing the ability of the mail system to 
carry legitimate mail.

> Success in the war on spam doesn't depend on all domains being as 
> clean as AOL.  We can rank them based on their reputations.  Most 
> email will come from domains that are clearly good or clearly bad.

I don't think so.  I think we'll see a lot of middle ground.

>> This is a non-trivial problem.
>
> The key problems are social, not technical.

while I don't disagree, that's not a helpful statement.  spam is a 
social problem because it involves a conflict between the desire of the 
sender and the desire of the recipient.  but because the two are at a 
distance, you can't solve the problem by purely social means.  nor can 
you solve the problem by purely technical means, or purely legal means. 
  it's hard enough to build technical mechanisms that are sound enough 
to carry their weight - trying to get the right balance of social, 
legal, and technical mechanisms and to get them to interact 
appropriately is much trickier.

>  Almost everyone shares your pessimism.  Nobody will change until they 
> see an immediate benefit.  The challenge is to engineer the system so 
> that it has positive feedback at every point on the growth curve. i.e. 
> the immediate benefit of change is worth the immediate cost.  Then the 
> process will go to completion, and spam will no longer be a major 
> problem.

that's probably a necessary condition, but it's not the only necessary 
condition.

Keith

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Site policy vs. HELO
Date:  Thu, 10 Mar 2005 03:28:04 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 13
Sender: [email-masked]

Hector Santos wrote:

> I dont' remember why but I must of got tired of you harassing
> me at the time sending duplicate messages to me and the
> various IETF mailing list.

Blocking based on unsolicited courtesy copies is a bad idea,
what if he sends a private mail ?  I had some simple To/Cc-
rules for this purpose, and in the one case where a poster 
used a "Bcc: list" I covinced him to change this practice.

                     Bye, Frank

Date: Wed, 09 Mar 2005 16:39:43 -0700
To: Keith Moore <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Cc: [email-masked]
Sender: [email-masked]

At 03:32 PM 3/9/2005 -0500, Keith Moore wrote:

>Spam is a hard problem.  If we knew of a good solution, we'd be using 
>it.  The government wouldn't have to mandate the solution, as there are 
>plenty of incentives already.  There are lots of half-baked non-solutions 
>and a few good ideas that raise the bar for spam without actually stopping it.
>
>Authentication methods will not solve the spam problem.  They may make 
>phishing harder, which is a good thing.   They might be useful when 
>combined with some other facilities, but nobody understand what those 
>facilities are.  Authentication methods are also no better than the hosts 
>that people use to submit mail.  So if you want to reduce spam by 
>requiring authentication, you first need to figure out how to make Windows 
>secure and to get that secure version deployed everywhere.

I'm more optimistic.  Authentication will allow us to hold domains 
responsible for their outgoing spam.  Reputable domains will eliminate 99% 
of their outgoing spam, as AOL has done.  Success in the war on spam 
doesn't depend on all domains being as clean as AOL.  We can rank them 
based on their reputations.  Most email will come from domains that are 
clearly good or clearly bad.  Only a small fraction will have to be 
filtered and processed as we do now for all email.  These will be mostly 
new domains that are trying to earn a good reputation, and a few domains 
that were ranked as good, but suddenly fell into the hands of a spammer.

>This is a non-trivial problem.

The key problems are social, not technical.  Almost everyone shares your 
pessimism.  Nobody will change until they see an immediate benefit.  The 
challenge is to engineer the system so that it has positive feedback at 
every point on the growth curve. i.e. the immediate benefit of change is 
worth the immediate cost.  Then the process will go to completion, and spam 
will no longer be a major problem.

>Filtering methods will not solve the spam problem.  They can raise the bar 
>a bit, which means we will get different kinds of spam, rather than less spam.

Filters will be needed as long as the flow of spam is too much for users to 
handle.  When it becomes rare again, the few pieces that get through will 
be "filtered" by the best judge of all, the recipient, then bounced 
upstream and used to rapidly and effectively isolate the source.

That is my view of the future, anyway.  For the topic of this thread, 
however, it doesn't matter if I am wrong on this.  If all we get is the 
elimination of phishing scams, that is reason enough to move ahead, and 
work out a standard that everyone can live with.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Wed, 09 Mar 2005 16:27:39 -0700
To: Dave Crocker <[email-masked]>, Russ Allbery <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Spam Bounces - was What is Spam
Cc: <[email-masked]>
Sender: [email-masked]

At 03:04 PM 3/9/2005 -0600, Dave Crocker wrote:

> >  The reliability of detecting spam should not be an issue for the mail
> >  handling system.  If the recipient thinks it is spam, it should be treated
> >  as spam.
>
>this discussion has ceased to be about the email-arch document.
>
>if it continues, please do so under a different subject, such as the one i 
>have used here.

I was trying to move the discussion away from "what is spam" ( Irrelevant 
question, in my opinion ) back to what to do with "Spam Bounces", assuming 
the subjective judgement has already been made by the user.

>the architecture document is not going to be adding spam constructs, for 
>the reasons russ (and lots of others) have cited.  it is a social issue.

What should we call this group of messages that may need to be handled 
differently by the mail system than any other group shown in Fig. 5?  If 
you are saying this group should *not* be handled differently than DSNs, 
then you are taking one position on a legitimate question, but not the only 
reasonable position.

I think I said this before, but I'm not suggesting any changes in the 
figures.  Current practice is to treat Spam Bounces as DSNs, so the figure 
is an accurate statement of current reality.  My only suggestion, and this 
is a small issue not worth an argument, was that the text list this 
separate category of message, maybe as a particular type of DSN.  That 
would facilitate discussion, without taking a position on the question of 
how this category should be handled.

Sorry for the distraction.  I'm OK with dropping the topic if you think it 
is not worth discussion.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Wed, 09 Mar 2005 16:25:53 -0700
To: John Leslie <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: New Authenticated: header?
Cc: [email-masked]
Sender: [email-masked]

At 04:18 PM 3/9/2005 -0500, John Leslie wrote:

>David MacQuigg <[email-masked]> wrote:
> >
> > See my "Spam Scenarios" at
> > http://www.ece.arizona.edu/~edatools/etc/
>
>    David introduces an interesting idea there: interspersing "Authenticated:"
>headers among the "Received:" headers:
>]
>] Received: ( **SaniMail 47655 invoked from network** ); 28 Feb 2005
>]         15:35:36 -0000
>] Authenticated: 208.210.124.73 pobox.com SenderID PASS
>] Received: from unknown (HELO gold.pobox.com) (208.210.124.73)
>]         by mail5.mailsystem.us with SMTP; 28 Feb 2005 15:35:36 -0000
>
>    I think it would work better as
>"
>" Authentication: <method> (<parameters>) = <result>
>
>but it's the _idea_ that I find interesting, not the syntax.

Here is a better statement of the "idea" ( from 
http://www.ece.arizona.edu/~edatools/etc/Email%20Forwarding%20Protocol.htm ).
Fundamental Requirements:
5) If the message is forwarded, the results of authentication must be made
available to all subsequent receiving MTAs.  The format should be simple and
standardized to facilitate later blocking and filtering.

I don't want to get too far into the implementation suggestions until we 
have a consensus on the fundamental requirement.

>If there were a header prepended at the time Authentication is done,
>it would make it possible to use its result as input to (possibly
>heuristic) filtering executed later; and _might_ open the door to a
>future in which a limited trust could be given to Received: lines after
>the first.

Exactly.  Given the prevalence of forwarders, extending the "boundary of 
trust" by at least one hop will be essential to success of SPF.  I'm 
surprised they haven't addressed this issue in their draft.  SenderID 
probably has some patented way to do it.  DomainKeys bypasses the problem 
by going end-to-end.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Wed, 9 Mar 2005 22:25:24 +0000
From: Tony Finch <[email-masked]>
To: John Leslie <[email-masked]>
cc: [email-masked]
Subject: Re: New Authenticated: header?
Sender: [email-masked]

On Wed, 9 Mar 2005, John Leslie wrote:
>
>    What do other folks think?

It's probably worth doing. However 2822 needs to be revised so that the
set of valid trace header fields is extended to more than Received: and
Resent-*.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTH TYNE DOGGER FISHER GERMAN BIGHT: VARIABLE 3 OR 4, BECOMING WEST OR
SOUTHWEST 4 OR 5, OCCASIONALLY 6 IN FISHER LATER. OCCASIONAL RAIN. MODERATE OR
GOOD.

From: "Hector Santos" <[email-masked]>
To: "John Leslie" <[email-masked]>, "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: New Authenticated: header?
Date: Wed, 9 Mar 2005 17:20:50 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "John Leslie" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, March 09, 2005 4:18 PM
Subject: New Authenticated: header?

> "
> " Authentication: <method> (<parameters>) = <result>
>
> but it's the _idea_ that I find interesting, not the syntax.
>
>    What do other folks think?

Any idea that requires a payload analysis is a problem.

Augment it with an ESMTP "HEAD" proposal and you might get some interest
from me.

No change that,  100% immediate support and backing for a HEAD command.
This will immediately turn on the light for all the mail header based
proposals.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 16:41:41 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Russ Allbery" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Sent: Wednesday, March 09, 2005 3:53 PM
Subject: Re: Has the IETF dropped the ball?

> > We have a lot of good ideas out there, but there all suffer with the
> > same realization:
>
> >     "What to do about the legacy and/or non-compliant transaction?"
>
> Yup.  That and the question of whether, given the degree of restrictions
> some of these protocols put on how SMTP can be used, whether what's left
> after the required slashing and burning of features is really usefully
> SMTP any more.

My point is that the minimum that must be done is SMTP compliancy because
there are two basic categories of the above:

1) The Malicious (60-80%), and
2) The legitimate,

The facts are showing the latter is a dieing breed.  The legitimate legacy
and non-compliant is getting push out, replaced or updated for the simple
reason they are indeed legitimate and today they realize there is a need for
compliancy.   We continue to see customers who are now adjusting there
setups for this very reason.  Before they didn't have to. Today they do.

This old statement in SMTP 2821 simply no longer applies:

| 7.1 Mail Security and Spoofing
|
|   ...
|
|   This specification does not further address the authentication issues
|   associated with SMTP other than to advocate that useful functionality
|   not be disabled in the hope of providing some small margin of
|   protection against an ignorant user who is trying to fake mail.

If this was still the case, then we wouldn't be here today with this
problem.

It is no longer just an "ignorant user" but an industry of malicious abusers
and there is a great margin of protection by applying pressure on the
non-compliant mail senders.

This is the mindset we need to change.  We need a new 3821 document that
focuses on security first and relaxed provisions second.  Not the other way
around.

Look. back in 1987, we all knew of the potential for Sender Spoofing.  It
was quite obvious the loop hole was there, but we also knew the abuse was
low at the time. Modem dialups was the norm and the topology was still
larger star based.  Therefore no real effort to close the loophole.

Today, we need to finally close it in a vast P2P environment to even come
close to increasing the reliability of augmented new email
authorization/authentication technology.

Until we do so, what use is any of the proposals if server still needs to
allow malicious non-compliant senders to exist in status quo form?   He has
no incentive to adopt if even to attempt to get around it.   No reason what
so ever.  That is exactly what we are seeing.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 9 Mar 2005 16:18:25 -0500
From: John Leslie <[email-masked]>
To: David MacQuigg <[email-masked]>
Cc: [email-masked]
Subject: New Authenticated: header?
Sender: [email-masked]

David MacQuigg <[email-masked]> wrote:
> 
> See my "Spam Scenarios" at 
> http://www.ece.arizona.edu/~edatools/etc/

   David introduces an interesting idea there: interspersing "Authenticated:"
headers among the "Received:" headers:
] 
] Received: ( **SaniMail 47655 invoked from network** ); 28 Feb 2005
]         15:35:36 -0000
] Authenticated: 208.210.124.73 pobox.com SenderID PASS
] Received: from unknown (HELO gold.pobox.com) (208.210.124.73)
]         by mail5.mailsystem.us with SMTP; 28 Feb 2005 15:35:36 -0000

   I think it would work better as
" 
" Authentication: <method> (<parameters>) = <result>

but it's the _idea_ that I find interesting, not the syntax.

   If there were a header prepended at the time Authentication is done,
it would make it possible to use its result as input to (possibly
heuristic) filtering executed later; and _might_ open the door to a
future in which a limited trust could be given to Received: lines after
the first. 

   What do other folks think?

--
John Leslie <[email-masked]>

From: Dave Crocker <[email-masked]>
To: David MacQuigg <[email-masked]>, Russ Allbery <[email-masked]>
CC: <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Wed, 9 Mar 2005 15:04:51 -0600
Subject: what is spam?
Sender: [email-masked]

>  The reliability of detecting spam should not be an issue for the mail
>  handling system.  If the recipient thinks it is spam, it should be treated
>  as spam.  

this discussion has ceased to be about the email-arch document.

if it continues, please do so under a different subject, such as the one i have used here.

the architecture document is not going to be adding spam constructs, for the reasons russ (and lots of others) have cited.  it is a social issue.

 d/
 --
 Dave Crocker
 Brandenburg InternetWorking
 +1.408.246.8253
 dcrocker  a t ...
 WE'VE MOVED to:  www.bbiw.net

To: "IETF-SMTP" <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 09 Mar 2005 12:53:59 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:

> Second, we need to focus on the most prevalent fact we have:

>     By far, by industry measurements, 60-80% of all transactions are
>     non-SMTP compliant.

> Therefore, using any form of mathematics, physics and/or logic, it
> should go without saying that a great benefit is theorically achievable
> at the SMTP level simply by enforcing SMTP compliancy.

Unfortunately, the 60-80% of SMTP transactions that aren't SMTP-compliant
and the 80% of transactions that are spam are not quite the same 80%, or
this would be a lot easier than it is.  Not that I disagree with you, in
general, but that's why the notion of "let's tighten SMTP down to only
what's exactly compliant" has been such a hard sell.  Like all other spam
filtering solutions, it has false positive problems, and in fact has more
false positive problems than some of the other spam filtering techniques.

> We have a lot of good ideas out there, but there all suffer with the
> same realization:

>     "What to do about the legacy and/or non-compliant transaction?"

Yup.  That and the question of whether, given the degree of restrictions
some of these protocols put on how SMTP can be used, whether what's left
after the required slashing and burning of features is really usefully
SMTP any more.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Wed, 09 Mar 2005 13:42:59 -0700
To: Russ Allbery <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Cc: [email-masked]
Sender: [email-masked]

At 05:53 PM 3/8/2005 -0800, Russ Allbery wrote:

>David MacQuigg <[email-masked]> writes:
> > At 02:37 PM 3/8/2005 -0800, Russ Allbery wrote:
>
> >> That sure sounds like a Delivery Status Notification to me.  I don't
> >> see why it should be treated differently from any other DSN.
>
> > If you send a Spam Bounce the same place you send a DSN, you are most
> > likely sending it to a forged Return-path.
>
>Given that spam is not reliably detectable in practice, that's the same
>problem that you have with any DSN, which means that one needs to look at
>how to handle DSNs on a broader basis.  It may be that part of that
>solution is to not generate DSNs, or generate DSNs differently, if one
>believes the message to be spam, but that doesn't change the fact that
>such messages are still DSNs.

The reliability of detecting spam should not be an issue for the mail 
handling system.  If the recipient thinks it is spam, it should be treated 
as spam.  It should not be treated the same as a normal DSN and routed as 
shown in Fig. 5.  It should not be treated as a normal message and sent to 
the From or Reply-To address.  Simply prohibiting Spam Bounces is an 
option, but one that should be discussed with full understanding of the 
cost and benefits.

>It's more a terminology problem.  It makes it sound like there's some new
>type of message, when it's just a DSN for a message that one believes has
>certain properties.  A better term would be "DSN for detected spam" or
>something along those lines.

It's definitely a terminology problem.  Unfortunately, the terminology 
seems to limit what we can discuss.  Even when I try to make a clear 
distinction between Spam Bounces, and normal DSNs, I still get arguments 
like "Spam Bounces should be prohibited, because the Return-path might be 
forged." or "You can't change the routing of DSNs.  The Return-path is 
expected and must be used."  As long as we cannot separate the two groups ( 
Spam Bounces vs DSNs ) we will have these difficulties in communication.

"DSN for detected spam" is a little too verbose.  I'll continue using "Spam 
Bounce" in my own writing, and just repeat what I mean when there appears 
to be some confusion.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 15:32:33 -0500
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

> The public is getting mad as hell about spam.  There *will* be a 
> solution to this problem.  If the IETF doesn't provide it, some 
> politicians or bureaucrats will.

I would restate that as: There _will_ be more and more poorly-conceived 
hacks which purport to solve the spam problem, but which mostly serve 
to decrease reliability and transparency of the mail system without 
doing anything about spam.  This applies both to hacks that come from 
the technical community and those that come from politicians and 
bureaucrats.

Spam is a hard problem.  If we knew of a good solution, we'd be using 
it.  The government wouldn't have to mandate the solution, as there are 
plenty of incentives already.  There are lots of half-baked 
non-solutions and a few good ideas that raise the bar for spam without 
actually stopping it.

Authentication methods will not solve the spam problem.  They may make 
phishing harder, which is a good thing.   They might be useful when 
combined with some other facilities, but nobody understand what those 
facilities are.  Authentication methods are also no better than the 
hosts that people use to submit mail.  So if you want to reduce spam by 
requiring authentication, you first need to figure out how to make 
Windows secure and to get that secure version deployed everywhere.  
This is a non-trivial problem.

Filtering methods will not solve the spam problem.  They can raise the 
bar a bit, which means we will get different kinds of spam, rather than 
less spam.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 15:14:31 -0500
Sender: [email-masked]

On Wed March 9 2005 14:25, Hector Santos wrote:

> > Bull. What my ISP might or might not have done is outside
> > of my control or knowledge base.
> 
> Sorry, it was all you.  See the logs below:

> 11:35:34 C: EHLO smtp02.mrf.mail.rcn.net

> 11:35:34 C: EHLO smtp02.mrf.mail.rcn.net

> ip=207.172.4.61&cdn=smtp02.mrf.mail.rcn.net&from=[email-masked]&xid=public
> &debug=10&b1=submit
> 20041021 17:35:51 0002560b version    : 2.01 / 1.10
> 20041021 17:35:51 0002560b xid        : public
> 20041021 17:35:51 0002560b calltype   : HTTP
> 20041021 17:35:51 0002560b callpath   :
> http://www.winserver.com/public/antispam/testwcsap.wct
> 20041021 17:35:51 0002560b sesstype   : system
> 20041021 17:35:51 0002560b callerip   : 65.10.108.109
> 20041021 17:35:51 0002560b cip        : 207.172.4.61
> 20041021 17:35:51 0002560b cdn        : smtp02.mrf.mail.rcn.net

> 20041021 17:36:15 0002560b sapcbv     : mx: 207.172.4.98     mx.mail.rcn.net
> 20041021 17:36:15 0002560b try mx     : mx.mail.rcn.net ip: 207.172.4.98
> 20041021 17:36:15 0002560b # connecting to 207.172.4.98
> 20041021 17:36:15 0002560b S: 220 mx19.mrf.mail.rcn.net ESMTP [NO UCE] RCN
> Exim 3.35 #7 SD NASTD Thu, 21 Oct 2004 17:30:34 -0400

> "Proof of the pudding is in the eating!"
> 
> Also, I see you ran about 5-6 testwcsap this morning.  You tried various
> combinations of input parameters.  No FALSE POSITIVES!

Dude, if you want to embarrass yourself publicly, that's your
business. {smtp02.mrf,mx.mail}.rcn.net are my "home" ISP's
servers; nothing to do with me.  In late October 2004, I was
in Indianapolis connected to the Internet via cable modem
through another ISP, so no traffic generated by me directly
would have been through the 207.172.4 block or any other IP
block registered to RCN.

I have absolutely no interest whatsoever in your "testwcsap"
and have had no connection to it, not today, not ever.

Date: Wed, 09 Mar 2005 13:07:15 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

Russ,

Thanks for your thoughtful comments.

At 02:50 PM 3/8/2005 -0800, Russ Allbery wrote:

>David MacQuigg <[email-masked]> writes:
>
> > The public is getting mad as hell about spam.  There *will* be a
> > solution to this problem.  If the IETF doesn't provide it, some
> > politicians or bureaucrats will.  I may not understand all that is going
> > on in the IETF, but it sure looks like they dropped the ball.
>
>Your confusion may be cleared up by realizing that the IETF, like most
>standards organizations, does far better at clearly documenting the right
>way to do something after that's been hashed out over time than it does at
>making up a new way of solving a brand new problem.  The latter is more
>properly termed "research and development" and is something that works
>most effectively in small teams of smart people rather than in large
>committees worried about getting all the details right.

I agree that IETF shouldn't be doing the R&D.  Their role should be more as 
impartial judges of the proposals that have been laid before them.  Where I 
think they failed in this case of email authentication methods is in 
letting the advocates take over, and not insisting that at least one of the 
proposals be the "common elements" that are needed by all of the 
methods.  That would be almost nothing for an end-to-end method like 
DomainKeys, and a few common items for SPF and SenderID that share a need 
to communicate through a chain of forwarders.  All the other details could 
left to the competing groups to do as they wish.

By leaving everything to the competing groups, we have a situation where a 
forwarder using SenderID will add the SUBMITTER parameter, and the 
Destination using SPF will not know what to do with it.  Surely there can't 
be endless debate over how to communicate this piece of information.

>Given that no clear technical solution has yet emerged from the pack on
>dealing with spam, it doesn't surprise me at all that the IETF has not
>been able to pick a clear winner and say "everyone should do this."

Notice I didn't say IETF should pick a winner right now between SPF, 
SenderID, and DomainKeys.  I certainly can't pick one.  What we need is a 
very simple framework that will allow the industry to move forward and not 
worry that they will have to change everything when a winner does 
emerge.  A company with a popular spam filter, for example, should know by 
now what authentication headers will look like.  They don't need to know 
*how* the authentication was done, just that there will be a common format 
for the results.

> > They have allowed the standards effort to disintegrate into a bunch of
> > programmers squabbling over details that are not needed in a universal
> > standard.
>
>I don't think all those people are just stupid and squabbling for no
>reason.  I don't think we've figured out a great solution and are just
>failing to implement it because we're fighting over trivia.  Rather, I
>think people have proposed a huge number of solutions that range in
>quality from hideous to barely mediocre, without a *good* solution in the
>lot, and much of the arguing that you're seeing is over whether it's
>worthwhile to try to implement one of the several barely mediocre
>solutions and deal with the resulting breakage because nothing better
>seems to be appearing, or to wait for more R&D.

I don't think anyone in these discussions is stupid.  I do think the 
problem is that many are so caught up in the heat of battle, that they 
can't even think about compromise on the few shared items that everyone 
needs.  There may also be a few who feel that such a compromise would be to 
their disadvantage, and would prefer that the deadlock continue.  The IETF 
should be above all this, and work only for the public good.  Listen to the 
advocates.  Understand their objections.  But don't give them a vote.

I don't see anything in the shared items I have in mind that would "break" 
anything.  These are simple items like how to communicate to the next 
station the domain name that was authenticated.  We could add a few words 
to existing Received: headers, but to avoid any risk of breaking something, 
the standard could call for a new Authenticated: header.  The standard 
could specify such things as when to add an Authenticated: header (when 
crossing into a new administrative domain).  This would be the same for all 
methods.

> > If we continue the present path, it may be years before there is a clear
> > "victory" in the battle between competing and incompatible standards.
>
>And how do you think that the IETF should pick among the current bunch?
>Drawing lots?  The reason why there's no victory is that none of them are
>clearly better, and they all take very different approaches.

See above.

>The fundamental problem of spam is that we want to use a system
>specifically designed to not use authentication to either reject all mail
>from "bad people" (where the definition of who's bad varies wildly from
>one person to another) or to instead reject all "bad mail" (with a similar
>wild variation in definition).  This isn't exactly an easy problem.  This
>is certainly not a problem that we understand well enough that all that's
>needed now is to get a few experts in a room and write the final draft.
>For one thing, it's not even clear that it's *possible* to retrofit
>authentication into the e-mail system in such a fashion that doesn't break
>so much of e-mail that it would be better to just use a different protocol
>entirely, and that's just one of the many issues that arises.

Mixing in the question "What is spam?" is one way to bring all progress to 
a halt.  Luckily we don't need to answer that question.  We just need to 
provide a common protocol to handle the results of a Recipient having 
decided something is spam.  Part of the challenge for the judges here is to 
recognize what is relevant, and limit debate on issues that aren't.

>Yes, people are frustrated, but there is little correlation between how
>frustrated people are and how easy the problem is to solve.

My frustration is partly due to the lack of progress on what I see as 
simple problems.  I'm also well aware that what I see as simple, may in 
fact have some hidden difficulties.  I listen very carefully to discussions 
on this and other lists, looking for those little bits of knowledge that 
will clarify something I didn't understand.  I can't just automatically 
believe it when an expert says I am wrong, however.  I much prefer to 
investigate and reach my own conclusions.

There are parts of the authentication problem that are very difficult, but 
I'm not seeing those difficulties in the part that needs to be a shared 
standard.  Establishing a chain-of-trust, for example, will be essential 
for all IP-authentication methods, and could turn out to be an 
insurmountable problem.  That problem need not be solved by the IETF, 
however.  If you focus on just the problem of how to communicate 
authentication results from each forwarder to the final destination, you 
will maximize the chance that one or another group will make their 
IP-authentication method work.  With no standard there will be endless 
finger pointing over the reason for failure of transfers involving 
forwarders with competing protocols.  With a clear standard, it will be 
possible for one contender to demonstrate success even if a competitor's 
system is a reluctant participant.

-- Dave

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 14:25:30 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, March 09, 2005 1:48 PM
Subject: Re: Site policy vs. HELO

>
> On Wed March 9 2005 08:14, Hector Santos wrote:
>
> > Bruce could of easily reported back in Oct 2004, "Yo hector, my mail is
> > being rejected."
>
> No, because you would have rejected that mail as well.

No, you would of done it publically like you eventually did, but only this
time in a silly vain attempt to prove "something" whatever that was.

> > The fact is he knew this.  He knew he was being blocked via WCSMTP
because
> > at the same date Oct 27, he tried out the TESTWCSAP directly and saw
that he
> > passed the test.
>
> Bull. What my ISP might or might not have done is outside
> of my control or knowledge base.

Sorry, it was all you.  See the logs below:

> Your inability to receive mail is your problem, not mine,
> and I'm not going to jump through hoops to attempt to
> notify you that you have a problem.

"Wheeeeeee, here we go!"

Bruce,  You were twitted!!!  PERIOD!   You proved nothing!  You were black
listed, plain and simple and as much that might pain you, you also knew you
were black listed. This report of yours is just a red-herring. You know it.

Here is the Oct 21, 2004 logs.

Here is the SMTP level rejection:

**************************************************************************
Wildcat! SMTP Server v6.0.451.2
SMTP log started at Thu, 21 Oct 2004  11:35:34
Connection Time: 20041021 11:35:34  cid: 00024941
SSL Enabled: NO
Client IP: 207.172.4.61 (unknown)
11:35:34 S: 220-winserver.com Wildcat! ESMTP Server v6.0.451.2 ready
...
11:35:34 C: EHLO smtp02.mrf.mail.rcn.net
11:35:34 S: 250-winserver.com, Pleased to meet you.
11:35:34 S: 250-SIZE 5120000
11:35:34 S: 250-ETRN
11:35:34 S: 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
11:35:34 S: 250-AUTH=LOGIN
11:35:34 S: 250 HELP
11:35:34 C: MAIL FROM:<[email-masked]> SIZE=7732
11:35:34 S: 250 <[email-masked]>... Sender validation pending. Continue.
11:35:34 C: RCPT TO:<[email-masked]>
11:35:34 S: 550 Return Path not verifiable.
11:35:34 C: QUIT
11:35:34 S: 221 closing connection

**************************************************************************
Wildcat! SMTP Server v6.0.451.2
SMTP log started at Thu, 21 Oct 2004  11:35:34
Connection Time: 20041021 11:35:34  cid: 00024942
SSL Enabled: NO
Client IP: 207.172.4.61 (unknown)
11:35:34 S: 220-winserver.com Wildcat! ESMTP Server v6.0.451.2 ready
...
11:35:34 C: EHLO smtp02.mrf.mail.rcn.net
11:35:34 S: 250-winserver.com, Pleased to meet you.
11:35:34 S: 250-SIZE 5120000
11:35:34 S: 250-ETRN
11:35:34 S: 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
11:35:34 S: 250-AUTH=LOGIN
11:35:34 S: 250 HELP
11:35:34 C: MAIL FROM:<[email-masked]> SIZE=4205
11:35:34 S: 250 <[email-masked]>... Sender validation pending. Continue.
11:35:34 C: RCPT TO:<[email-masked]>
11:35:34 S: 550 Return Path not verifiable.
11:35:34 C: QUIT
11:35:34 S: 221 closing connection

Note the time.  What was strange was no wcSAP recording. But as we found
out, you were black listed using a deprecated filtering file.  I prematurely
took the benefit of the doubt and considered a bug on our part.  Nope. Not
the case. You were twitted. Plain and simple.

Around 5 hours later, you were still trying to figure out the rejection
reason, so you tried the web-based TESTWCSAP test form:

20041021 17:35:51 0002560b -------------------------------------
20041021 17:35:51 0002560b params     :
ip=207.172.4.61&cdn=smtp02.mrf.mail.rcn.net&from=[email-masked]&xid=public
&debug=10&b1=submit
20041021 17:35:51 0002560b version    : 2.01 / 1.10
20041021 17:35:51 0002560b xid        : public
20041021 17:35:51 0002560b calltype   : HTTP
20041021 17:35:51 0002560b callpath   :
http://www.winserver.com/public/antispam/testwcsap.wct
20041021 17:35:51 0002560b sesstype   : system
20041021 17:35:51 0002560b callerip   : 65.10.108.109
20041021 17:35:51 0002560b cip        : 207.172.4.61
20041021 17:35:51 0002560b cdn        : smtp02.mrf.mail.rcn.net
20041021 17:35:51 0002560b from       : [email-masked]
20041021 17:35:51 0002560b localhost  : mail.winserver.com
20041021 17:35:51 0002560b testorder  : FLT RBL SPF CEP CBV
20041021 17:35:51 0002560b sapfilter  : pass (time:78)
20041021 17:35:51 0002560b saprbl     : testing
61.4.172.207.sbl.spamhaus.org
20041021 17:35:52 0002560b saprbl     : testing 61.4.172.207.list.dsbl.org
20041021 17:35:57 0002560b saprbl     : testing 61.4.172.207.bl.spamcop.net
20041021 17:36:02 0002560b saprbl     : pass (time:10078)
20041021 17:36:07 0002560b sapspf     : none (time:5906)
20041021 17:36:07 0002560b sapcep     : test from=erols.com
20041021 17:36:12 0002560b sapcep     : test cdn=smtp02.mrf.mail.rcn.net
20041021 17:36:14 0002560b sapcep     : none (time:6109)
20041021 17:36:14 0002560b sapcbv     : MX lookup: erols.com
20041021 17:36:15 0002560b sapcbv     : total mx records: 1
20041021 17:36:15 0002560b sapcbv     : mx: 207.172.4.98     mx.mail.rcn.net
20041021 17:36:15 0002560b try mx     : mx.mail.rcn.net ip: 207.172.4.98
20041021 17:36:15 0002560b # connecting to 207.172.4.98
20041021 17:36:15 0002560b S: 220 mx19.mrf.mail.rcn.net ESMTP [NO UCE] RCN
Exim 3.35 #7 SD NASTD Thu, 21 Oct 2004 17:30:34 -0400
20041021 17:36:15 0002560b C: NOOP WCSAP v2.01 Wildcat! Sender
Authentication Protocol http://www.santronics.com
20041021 17:36:15 0002560b S: 250 OK
20041021 17:36:15 0002560b C: HELO mail.winserver.com
20041021 17:36:15 0002560b S: 250 mx19.mrf.mail.rcn.net Hello
ntbbs.winserver.com [208.247.131.9]
20041021 17:36:15 0002560b C: MAIL FROM: <>
20041021 17:36:15 0002560b S: 250 <> is syntactically correct
20041021 17:36:15 0002560b C: RCPT TO: <[email-masked]>
20041021 17:36:15 0002560b S: 250 <[email-masked]> verified
20041021 17:36:15 0002560b C: RCPT TO:
<[email-masked]>
20041021 17:36:15 0002560b S: 550 relaying to
<[email-masked]> prohibited by administrator
20041021 17:36:15 0002560b S: 550 relaying to
<[email-masked]> prohibited by administrator
20041021 17:36:15 0002560b C: QUIT
20041021 17:36:15 0002560b sapcbv     : 250
20041021 17:36:15 0002560b result     : accept (-1)
20041021 17:36:15 0002560b wcsap finish (23984 msecs)
20041021 17:43:30 00025651 -------------------------------------

"Proof of the pudding is in the eating!"

Also, I see you ran about 5-6 testwcsap this morning.  You tried various
combinations of input parameters.  No FALSE POSITIVES!

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 09 Mar 2005 12:27:17 -0700
To: Tony Hansen <[email-masked]>, [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

At 12:25 PM 3/9/2005 -0500, Tony Hansen wrote:

>The IETF hasn't given up; it's simply regrouping.

This is good news.  I have a lot of confidence in the integrity and wisdom 
of this group.  I just worry that they won't keep the uncompromising 
advocates at arms length.  This will require great wisdom in picking out 
valid arguments from a flood of mind-boggling advocacy.  I have this mental 
image of a befuddled old judge and a glazed-eye jury listening to 
conflicting expert witnesses.

>There're groups of people actively working on each of the following:
>
>         Client SMTP Authorization (CSA)
>         Domain Name Accreditation (DNA)
>         Sender Policy Framework (SPF)
>         Sender ID
>         a combined Domain Keys and Identified Internet Mail (IIM)
>         Bounce Address Tag Validation (BATV)

These all have a lot of detail that the advocates will never agree on.  I 
don't see any thought as to how they will work together with competing 
methods in the same transfer.  I worry about experiments going on for a 
year or more, and no clear conclusions because the real tests require 
participation of a large part of the Internet population.

When a standard finally does emerge, it will than take another year for 
writers of spam filters, MTAs, MUAs, etc. to start using the 
standard.  Spamnix, for example, has no current plans to use domain 
names.  The folks at SpamCop seem strongly opposed to any use of domain 
names instead of IP addresses in their real-time lists.

We could cut a year off the adoption process if we could just agree on a 
few simple items, like what the authentication headers will look like.  I 
don't mean all the details like we see now in Received-SPF, but just the 
essentials - IP address, domain name, and result.  The rest could be 
optional parameters added by each method, maybe to be standardized later if 
there really is a need.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 13:48:05 -0500
Sender: [email-masked]

On Wed March 9 2005 08:14, Hector Santos wrote:

> Bruce could of easily reported back in Oct 2004, "Yo hector, my mail is
> being rejected."

No, because you would have rejected that mail as well.
 
> The fact is he knew this.  He knew he was being blocked via WCSMTP because
> at the same date Oct 27, he tried out the TESTWCSAP directly and saw that he
> passed the test.

Bull. What my ISP might or might not have done is outside
of my control or knowledge base.

Your inability to receive mail is your problem, not mine,
and I'm not going to jump through hoops to attempt to
notify you that you have a problem.

From: "Hector Santos" <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>, "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 13:18:36 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Carl S. Gutekunst" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, March 09, 2005 12:34 AM
Subject: Re: Has the IETF dropped the ball?

> I would also like to direct your attention to a marvelous paper by Brett
> Watson of Macquarie University detailing all the work we have to do
> *after* we have a working authentication system. Authentication is a
> necessary first step to controlling spam, but by no means sufficient:
>
>     http://www.ceas.cc/papers-2004/140.pdf
>

I read a little and what stood out was its initial focus of analysis an
"ideal solution."  I struck my interest.

Unfortunately, it failed in doing this what I expected.

The realistic ideal SMTP model considers all process parameters, overhead
and redundancy.

There are four parameters in its End Point Validation (EPV) function model.
These four parameters have a direct correlation to the validation result for
a transaction.

      Result = EPV(CIP, CDN, RP, FP)

where

    CIP is the client IP address (connection address)
    CDN is the Client Domain Name (HELO/EHLO)
    RP is the Return Path (MAIL FROM), and
    FP is the Forwarding Path (RCPT)

The ideal solution must include the forwarding path in an SMTP based
Transaction Management System (TMS).

If the FP is not valid, then it doesn't matter what an EPV(CIP,CDN, RP)
based on Dr. Watson's model may provide.

The realistic ideal model:

1) Enforce SMTP compliancy,
1.1) MUA considerations
1.2) MTA considerations
1.3) Presumes Return Path is 100% verifiable
2) Takes all process parameters into account,
3) Uses intelligent delayed verification to optimize the system,
4) Minimizes payload requirements,.
5) Uses a server attribute concept to pre-empt redundancy.

No matter what proposal you have, they ALL have to deal with the above.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 09 Mar 2005 12:58:57 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

On 3/8/2005 5:39 PM, Hector Santos wrote:

> 1) Check for syntax compliance
> 2) Check for Domain Literal IP match
> 3) Check for local domain spoofs

> 1) Check for syntax compliance
> 2) Per 2821, delay validation until Forwarding path is known.

These are pretty benign tests, and should certainly be optional for
administrators. But that doesn't change my point, experimentation shows
there are just too many poorly-written and poorly-managed SMTP boxes out
there for binary pass/fail reject to be used in the common case.

It's also a violation of the robustness principle, which is what most
folks here are probably most concerned with, although I think that's
really an operational issue in the case of SMTP in particular, and
therefore falls within administrator prerogative.

An example of the problem: RFC2821 $2.3.5 states that "A domain name that
is not in FQDN form is no more than a local alias. Local aliases MUST NOT
appear in any SMTP transaction." In a strict rule-set, therefore, mail
that uses non-FQDN domain names in HELO or MAIL-FROM or anywhere else can
be rejected because it's not allowed. Unfortunately, experimentation shows
that there's a fair chunk of legitimate mail that uses non-FQDNs.

The big realization for me here is that my world-view is biased towards
well-run networks, and that when you get into the lower-middle class of
network users, all those assumptions go out the window.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 12:44:07 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Russ Allbery" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 08, 2005 5:50 PM
Subject: Re: Has the IETF dropped the ball?

> Yes, people are frustrated, but there is little correlation between how
> frustrated people are and how easy the problem is to solve.

Good points.

I hate to be accused of over simplifying problems and its solutions. Maybe
that comes with years of integrated experiences.

Yes, the "Spam issue" is subjective in its definition but I believe it can
be broken down to its most basic elements.

First, as far as SMTP concern, we need to remove (keep) the mail
interpretation concepts out of the picture. This is one area where is much
confusion is blocking a solution.  It needs to be taken out of the first
level of security considerations because its a subjective concept.

Second, we need to focus on the most prevalent fact we have:

     By far, by industry measurements,  60-80% of all transactions
    are non-SMTP compliant.

Therefore, using any form of mathematics, physics and/or logic,  it should
go without saying that a great benefit is theorically achievable at the SMTP
level
simply by enforcing SMTP compliancy.

The obvious general consensus is that we want a solution without breaking
the current infrastructure.  Its an practical goal to reach in lieu of
considering the ideal goal.  But the R&D needs to be done for the ideal
solution first before we can even achieve a practical compromise.   We throw
a lot of good ideas out and limit the practical implementations.

We have a lot of good ideas out there, but there all suffer with the same
realization:

    "What to do about the legacy and/or non-compliant transaction?"

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "David MacQuigg" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Date: Wed, 9 Mar 2005 12:27:54 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "David MacQuigg" <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, March 09, 2005 9:30 AM
Subject: Re: Has the IETF dropped the ball?

> Can anyone give me an example of a requirement that must be included in a
> shared standard and cannot be written in a way that both sides could live
with?

In my view, all the proposals inevitably have the same inherent problem:

    "how do deal with legacy and non-compliant SMTP clients."

SMTP compliancy issues must be addressed first before any of these proposal
even have a chance to work.  It will be extremely difficult to implement a
proposal that accepts/rejects a compliant sender yet at the same time you
still have the operate in the same mode for legacy and non-compliant
senders.

The solution to this seems to be two folds:

- A different set of policies
- Stronger Client/Server negotiation handshaking.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 09 Mar 2005 12:25:37 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

The IETF hasn't given up; it's simply regrouping.

There're groups of people actively working on each of the following:

	Client SMTP Authorization (CSA)
	Domain Name Accreditation (DNA)
	Sender Policy Framework (SPF)
	Sender ID
	a combined Domain Keys and Identified Internet Mail (IIM)
	Bounce Address Tag Validation (BATV)

Is there a formal working group for this work? Not currently. Does there 
need to be? Not necessarily. Is the work being done? Absolutely.

	Tony Hansen
	[email-masked]

David MacQuigg wrote:
> 
> Regardless, IETF should not have given up because the MARID group
> broke up.

Date: Wed, 09 Mar 2005 07:30:56 -0700
To: "Carl S. Gutekunst" <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Cc: [email-masked]
Sender: [email-masked]

At 09:34 PM 3/8/2005 -0800, Carl S. Gutekunst wrote:

>In short, MARID failed because a simple, non-controversial, and 
>all-encompassing solution to Internet Mail Authentication does not seem to 
>exist.

We don't need an "all-encompassing solution" to be a useful standard.  SMTP 
doesn't encompass authentication, but it is still quite useful.  I would 
*not* expect a shared authentication standard to encompass complex 
algorithms for extracting the purported responsible domain from a set of 
headers.  I *would* expect it to encompass how the results of 
authenticating that domain are communicated downstream.

Can anyone give me an example of a requirement that must be included in a 
shared standard and cannot be written in a way that both sides could live with?

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 08:14:07 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, March 09, 2005 7:01 AM
Subject: Re: Site policy vs. HELO

> Hector Santos writes:
> > Like I said, there is a reason for everything. Nothing to do with
> > protocols or its reliability. In fact, it was a 100% trusted result.
>
> It's just that the error message did not say so, eh?

Right. Because it was an older deprecated filtering logic that was still
active on my system.

> Expecting people to report a problem (as you wrote about earlier today)
> is 100% unrealistic when your error message gives them no idea of what
> the problem is. Get real.

We did get real.  It was a built-in decision long ago to not feed malicious
senders any information that might help them adjust.

If any problem, is that we didn't log it on the server-side and that was
because it was a deprecated filtering feature.

The new system does log it on the server-side. That is what made this whole
thing strange.

Bruce could of easily reported back in Oct 2004, "Yo hector, my mail is
being rejected."

He would of found out early on he was twitted.  No, he tried to prove "some
point" here in public with no real agenda.

The fact is he knew this.  He knew he was being blocked via WCSMTP because
at the same date Oct 27, he tried out the TESTWCSAP directly and saw that he
passed the test.

The facts are if a site is experiencing problems with it users, they will
investigate it. They don't want to run a poor system any more than you and I
do.  This is a increasingly on going issue as more and more of the "layman"
is putting their trust in a working system.  We just solved a big one
yesterday which illustrated alot of whats going on.

Customer has an user that switches between the ISP's user email account and
the user's aol.com account.

The user's transactions using his AOL.COM were accepted.
The user's transactions using the ISP's domain were rejected.

In both cases, the user was sending mail via AOL's machines.

AOL.COM has a SPF record.

WCSAP accepted the AOL.COM sender transacton as SPF pass.

When the user used the ISP's domain, WCSAP softfailed the query, and for
some reason rejected the transaction.

Technically, SPF says to accept softfail results.

I deem this a big security flaw in SPF and said so from the beginning.

WCSAP has an optional setting that addresses this:

; SPF can return low trust results. A pass means the sender has
; a valid SPF record and is accepted. Softfail and Neutral means
; no match is found but rejection is not automatic.  Setting a
; true accept can provide a loophole for potential spoofers who have
; SPF records and think they will allow them in.  The options
; below allow you to control this.

Accept-SPF-Pass      True            ; if false, continue testing
Accept-SPF-SoftFail  False           ; if false, continue testing
Accept-SPF-Neutral   False           ; if false, continue testing

So by default, we reject the SOFTFAIL.

Well, why wasn't a big issue in the pass?

Well, because WCSAP is a suite of testing where a CBV is the final test
done.

WIth the option above set to false, wcSAP will continue testing.  But he had
turned off CBV, so therefore the final result was a rejection.

If we followed the SPF rule, than we just open up the door to SPF relaxed
provisions exploitation. We don't need any more relaxed provisions. That is
what caused the problem at SMTP in the first place.

By turned on CBV,  the user would of been vailidated against his own system,
thus the problem is solved.

Another issue about SPF was its other loophole of not having a provision to
check for HELO/EHLO domains unlike earlier LMAP proposals.  I argued for for
it and I believe it has been finally been put in.  It was always left as an
"option" with a recommendation that it was a redundant overhead therefore it
shouldn't be done.  That thinking has changed.

We have yet to update to this current SPF logic, but we have it built in for
Local Domain Spoof checking which is a compromise of having an open-ended
SPF HELO check.

Assuming we have this implemented,  here is the transaction parameters for
the rejection:

    Client IP address:  64.12.138.7
    Client Machine Name: rly-ip03.mx.aol.com
    From (Return Path): [email-masked]

The SPF lookup for skyhawkbbs.com (he has since changed it to a hard fail),
but he had a soft fail yesterday.

With the new SPF logic, it allows for a HELO SPF check.

Go ahead do all the DNS lookups you can on that AOL client domain.  SPF,
PTR, A whatever, it all fails.

This is why will keep it as a Local Domain Check only as this is the only
thing you can rely on. You can't trust the result of a HELO domain DNS
lookup.

The SUBMITTER proposal have promise. That would of worked here too.  Our
system at this time is SUBMITTER ready for the INBOUND side.

It has privacy security issues  It sends the entire address, when SPF only
needs the DOMAIN.

So if the AOL client send:

        mail from: <[email-masked]>  submitter=aol.com

It woud would of solved this forwarding problem.

AOL could of done some ridiculous sender rewriting scheme too.

But today, the solution here was turn on the CBV and to give the customer a
side of white listing rules to emulate a proper SPF lookup test:

Accept if %RP% in [email-masked] and  %CIP% in 152.163.225.0/24
Accept if %RP% in [email-masked] and  %CIP% in 205.188.139.0/24
Accept if %RP% in [email-masked] and  %CIP% in 205.188.144.0/24
Accept if %RP% in [email-masked] and  %CIP% in 205.188.156.0/24
Accept if %RP% in [email-masked] and  %CIP% in 205.188.159.0/24
Accept if %RP% in [email-masked] and  %CIP% in 64.12.136.0/24
Accept if %RP% in [email-masked] and  %CIP% in 64.12.138.0/24

All this is par for the course and I goal is teach the customer enough to
have them do this themselves.

Could SMTP + some methodology solve this?

Dave and Doug has CSV/DNA.   They think it will work.

Not in this case, AOL must support to.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: 
Date: Wed, 9 Mar 2005 07:33:34 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 08, 2005 5:28 PM

> On 3/8/2005 11:36 AM, Hector Santos wrote:
>
> > Over the pass year and a half, we have proved that by increasing the
> > level of SMTP compliancy required by senders, you can address an
> > extremely high rejection rate with a very low to non-existence false
> > positives.   100% based on SMTP compliancy.
>
> I'm seeing just the opposite. I've been doing some expirementation with
> some strict SMTP rules in SpamAssassin (looking for things like literal
> HELO, mismatched rDNS, etc), and am getting reports of extremely high
> error rates from testers. [1]

You are not going to get a reliable result performing DNS techniques for
HELO/EHLO.   Thats a given.

I don't see how you can get poor results with testing domain literals.
Either it is correct or its not.

Our HELO tester is quite simple:

1) Test for 2822 ABNF compliance.    - RFC 2822, RFC 2821
2) Domain Literals MUST match the IP - RFC 2821
3) Check for Local Domain Spoofs - RFC 2821

Examples:

1) No Brackets

HELO 208.131.247.9

Technically incorrect.  No brackets, before anyone stops here and says "Ah
ha! Got you!" read below.

2) IP mismatch

CLIENT IP = 205.123.343.23
HELO [208.131.247.9]

This is a clear violation   The IPs much match!  Again, read below.

3) Local Domain Spoof

CLIENT IP = 205.123.343.23
HELO mail.winserver.com

There is a clear local domain spoof.  As a compromise to having an opening
ended DNS lookup, we only check for local domain specifications.  If that
domain is part of your domain ownership, it can't be used by some foriegn IP
address and sender.

These are all lightweight testing with an extremely high rate of
effectiveness.  I don't see how you can get a high false positive here.  If
you do DNS lookups, sure.  That isn't going to work well.

We have over 1.5 year of production statistics that proves all this:

        http://www.winserver.com/spamstats

Another high rejection feature is having a multiline welcome response, this
will knock out atleast 40% of your streaming bulk spammers who are using
broken small footprint mail senders with little logic in it to test for
multiline line response codes.  I've been waiting for these idiots to
adjust - it hasn't happen yet!

If you want to catch the Major Abusers and take them to court for abusing
your system, this is how you do it because these softwares are so stupid
that they continue to re-retry until one way or another they finally learn
they are not getting in.

We got two instances of false positive reports:

1) A poor NAT configuration where the LAN IP was used for the WAN IP.
2) Old legacy software using unbracketed domain literals.

In both cases, they were quickly resolved with configuration corrections on
their end.

In a few cases with customers who still had an "not sure" feeling about this
when they stopped getting large amount of spam, turned it off  but quickly
turned it back on when they saw the high effectiveness of it.  They quickly
became believers.

SMTP Compliancy does indeed works and like I said, we should not be suprise.
Over 60-80% of all malicious transactions fall into a non-SMTP compliancy
category.  Therefore, without a question, applying pressure at SMTP will no
doubt be extremely effective.

> Note that I implemented these tests in SA so that I could encourage people
> to STOP doing hard-rejects tests at the MTA level, due to very high
> false-positive rates. I thought that by assigning a probability (as
> opposed to a simple pass-fail test) that these tests would be more
> effective, but from what I've seen so far, there are just too many
> mis-managed mail servers out there to even establish a probability rate
> that's acceptable.
>
> There are, of course, some very useful pass-fail MTA tests (including
> things like non-existent domain names), but not nearly as many as I
> suspected going into this.
>
> [1] http://www.rulesemporium.com/forums/showthread.php?s=&threadid=105
>

I will check out your information, but I am not into playing "probabilities
and statistics" when it comes to SMTP.  Either the client is SMTP compliant
or its not.  There is no in-between and it is 100% completely independent of
the mail content considerations (the payload).

Our wcSMTP system implements two hooks:

1) RCPT state transaction validation hook  (WCSMTP)
2) DATA state Mail Content Hook (wcSMTPFILTER)

The WCSMTPFILTER is a skeleton. We are not in the business of interpreting
mail content.  Administrator define the mail content filtering policies.
For SMTP, we just want a compliant transaction.    Administrators augment
3rd party AVS solution with wcSMTPFILTER which may include wsSpamGuard,
SpamAssasin, McAfee, Norton, or some Sieve hook, etc   All WCSMTP wants when
it calls the wcSMTPFILTER blackbox is three possible answers using a 32 bit
high/low response value:

- TRUE - accept,  250 response fixed
- FALSE - reject  high word has response (default 451 if I remember)
- KEEP - accept and put aside. 250 response fixed.

We really don't want to add any payload analysis baloney into the SMTP
process. It would mean a new product liability issue.  If it was 100%
perfect, no problem, it would be considered. But if not, we leave this for
the admins to define and setup.  That has been my long time technical
opinion for all mail networks since the dawn of time.  Rejection at the
transaction level is more plausible  Acceptance of the mail for post process
analysis assumes a responsibility for delivery or notification and this has
been exploited by the SORBIG-generation e-viruses.

So the faster you reject before the PAYLOAD is received, the better the
customer is served.

All our old UUCP/SLIP custom who updated to SMTP,  fell into this category.
They didn't want to even bother with all the new SMTP level validation
features, especially RCPT Validation at SMTP. This was normally done at the
gateway.   So an older ISP calls and says "I am getting murdered" I tell
them to simply turn on RCPT validation at SMTP and SORBIG is practically
eliminated from propagating thru his system.  Turn on Sender validation
stuff, and his malicious transactions are practically 100% eliminated.   Of
course, valid transaction spam does make it in, and that is where SA and
others are needed.

Look, all my point is here is that we really have no choice by address
malicious mail by fixing the very issues they exploit at SMTP.   Since it is
unfeasible to believe SMTP will change in the foreseeable future, at a
minimum, promote a stronger enforcement of SMTP client/server compliancy
using the current standard.

I believe once this is done across all major vendors, the industry will
benefit.  Only at this point, we can then invent new ideas that work around
a SMTP compliant transaction expectation.   We can't just continue with
stupid kludged ideas that is based on non-compliant operations.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 9 Mar 2005 13:01:16 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

Hector Santos writes:
> Like I said, there is a reason for everything. Nothing to do with 
> protocols or its reliability. In fact, it was a 100% trusted result.

It's just that the error message did not say so, eh?

Expecting people to report a problem (as you wrote about earlier today) 
is 100% unrealistic when your error message gives them no idea of what 
the problem is. Get real.

Arnt

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 06:30:59 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 08, 2005 6:13 PM
Subject: Re: Site policy vs. HELO

> This particular problem didn't begin a day, a week, or a month ago.
> The first excerpt that I provided today was from 21 Oct 2004, 11:29
> EDT.  And that wasn't the first time.

Well I'll be a MONKEY'S UNCLE!

Oh Bruce,  you were BLACK LISTED!!!  TWITTED from sending into our system
with your erols.com domain.

File: wc:\data\ValidateMailFrom.txt
[1    ] reject erols.com
End of File

Case close!

Lets see whats the file date is.... Aug 27 2003.

I knew there was a reason for all this.  I dont' remember why but I must of
got tired of you harassing me at the time sending duplicate messages to me
and the various IETF mailing list.  Let me see what my MUA logs show for
that time frame... Yup. some stupid thread about MUAs and you were all over
my face sending duplicate messages.  In fact, I found a draft email that I
never bother to post that informs you that you were twritted!

Like I said, there is a reason for everything.  Nothing to do with protocols
or its reliability. In fact, it was a 100% trusted result.

How did I find this?

In reality, what happen here was that the wc:\data\validateMailFrom.txt
filter file was deprecated but still available for backward compatibility.
Filtering is done via a p-code subprocess called wcSAP and the time frame
matches.  The WCSAP project started back in July 2003 time frame as shown by
our stats collection at http://www.winserver.com/spamstats.    I never
bothered to delee the old filter file.

I did a thorough examination of all your transactions since Oct 21 2004 and
I found it odd that the "Return Path Not Verified" response code did not
have a reason in the log.  Well, the source code has it to intentionally
hide the reason from you.

    if ((EnableMailFromFilter == 1) && !IsUserAuthorized) {
        DWORD retcode = 550; // was 552; 03/11/04 08:43 pm
        spfcode = SPF_SKIPPED;       // 450.9b15
        CString sResponse = "";  // 450.9b12b
        if (!ValidateReturnPath("mail",
                                 CallerInfo.szMailFrom,
                                 "",
                                 retcode,
                                 spfcode,
                                 sResponse)) {
            // 450.9b13
            int wcxret = SendWcxResponse();
            if (wcxret == 0) {
                // response not sent: hide reason from sender
                Send("%d Return Path not verifiable.\r\n",retcode);
            }
            //
            CString sLine = "";
            if (!sResponse.IsEmpty()) sLine += " ("+sResponse+")";
            smtplog("MAIL: Return Path not verifiable:
%s%s!",CallerInfo.szMailFrom,sLine);
            ResetState();
            return TRUE;
        }
    }

ValidateReturnPath first check the old validatemailfrom.txt file and then
spawns wcSAP.  wcSAP never ran which is why I never saw a log entry.  So I
had to find out why it didn't run.  The old filter file explained it!

wcSAP is our main AVS system and I don't mind saying it is the BEST out
there. That said, it simply uses strict SMTP compliancy as the basis of the
package.  Here is an example of a wcSAP rejection with a reason recorded on
the server-side.  We still don't send reasons for the sender.

**************************************************************************
Wildcat! SMTP Server v6.0.451.2
SMTP log started at Tue, 19 Oct 2004  23:33:17
Connection Time: 20041019 23:33:17  cid: 0001F991
SSL Enabled: NO
Client IP: 66.219.102.202 (unknown)
23:33:17 S: 220-winserver.com Wildcat! ESMTP Server v6.0.451.2 ready
23:33:17 S: 220-************** WARNING:  FOR AUTHORIZED USE ONLY!
**********************
23:33:17 S: 220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY
COMPUTERS    *
23:33:17 S: 220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE
UNSOLICITED *
23:33:17 S: 220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL
RESTRICT ACCESS *
23:33:17 S: 220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.
*
23:33:17 S: 220
************************************************************************
23:33:18 C: HELO 202box.abscat.com
23:33:18 S: 250 winserver.com, Pleased to meet you.
23:33:18 C: MAIL
FROM:<healthcare-andrea.santos=[email-masked]>
23:33:18 S: 250 <healthcare-andrea.santos=[email-masked]>...
Sender validation pending. Continue.
23:33:18 C: RCPT TO:<[email-masked]>
23:33:45 ** WCX Process: wcsap  ret: 550 (Rejected by WCSAP CBV)
23:33:45 S: 550 Return Path not verifiable.
23:33:45 C: QUIT
23:33:45 S: 221 closing connection

By taking the CID (connnection ID) 0001F991, we can look at the wcSAP log
for the details:

20041019 23:33:18 0001f991 -------------------------------------
20041019 23:33:18 0001f991 version    : 2.01 / 1.60
20041019 23:33:18 0001f991 calltype   : SMTP
20041019 23:33:18 0001f991 state      : rcpt
20041019 23:33:18 0001f991 srvdom     : winserver.com
20041019 23:33:18 0001f991 srvip      : 208.247.131.9
20041019 23:33:18 0001f991 cip        : 66.219.102.202
20041019 23:33:18 0001f991 cdn        : 202box.abscat.com
20041019 23:33:18 0001f991 from       :
<healthcare-andrea.santos=[email-masked]>
20041019 23:33:18 0001f991 rcpt       : <[email-masked]>
20041019 23:33:18 0001f991 testorder  : FLT RBL SPF CEP CBV
20041019 23:33:18 0001f991 sapfilter  : pass (time:62)

You passed the whiite/black list test

20041019 23:33:18 0001f991 saprbl     : testing
202.102.219.66.sbl.spamhaus.org
20041019 23:33:23 0001f991 saprbl     : testing 202.102.219.66.list.dsbl.org
20041019 23:33:26 0001f991 saprbl     : testing
202.102.219.66.bl.spamcop.net
20041019 23:33:27 0001f991 saprbl     : pass (time:9531)

You passed the RBL test.

20041019 23:33:29 0001f991 sapspf     : none (time:1360)

You have no SPF record

20041019 23:33:29 0001f991 sapcep     : test from=healthweb77.com
20041019 23:33:39 0001f991 sapcep     : test cdn=202box.abscat.com
20041019 23:33:40 0001f991 sapcep     : none (time:11297)

Nor do you have a Microsoft CallerID record.  We need to decprecate this
junk. Note the overhead time!

Now it begins the CBV and all we want to know is whether that address is
verifiable and we do a small test for a open relay.

20041019 23:33:41 0001f991 sapcbv     : total mx records: 1
20041019 23:33:41 0001f991 try mx     : mx.healthweb77.com ip:
66.111.217.120
20041019 23:33:41 0001f991 # connecting to 66.111.217.120
20041019 23:33:41 0001f991 S: 220 shadyweasel.com ESMTP
20041019 23:33:41 0001f991 C: NOOP WCSAP v2.01 Wildcat! Sender
Authentication Protocol http://www.santronics.com
20041019 23:33:45 0001f991 S: 250 ok
20041019 23:33:45 0001f991 C: HELO mail.winserver.com
20041019 23:33:45 0001f991 S: 250 shadyweasel.com
20041019 23:33:45 0001f991 C: MAIL FROM: <>
20041019 23:33:45 0001f991 S: 250 ok
20041019 23:33:45 0001f991 C: RCPT TO:
<healthcare-andrea.santos=[email-masked]>
20041019 23:33:45 0001f991 S: 553 sorry, that domain isn't in my list of
allowed rcpthosts (#5.7.1)
20041019 23:33:45 0001f991 C: QUIT
20041019 23:33:45 0001f991 sapcbv     : 553
20041019 23:33:45 0001f991 result     : reject (0)
20041019 23:33:45 0001f991 smtp code  : 550
20041019 23:33:45 0001f991 reason     : Rejected by WCSAP CBV
20041019 23:33:45 0001f991 wcsap finish (27109 msecs)
20041019 23:38:05 0001f9ae -------------------------------------

The CBV in the above case rejected the address. Therefore the final
transaction response is a rejection Period.

If the remote host accepted the address, we do a small test of a random
address to see if this is accepted too.  If so, the transaction is rejected
as a open relay site.

Here is an example of such a test, ironically with your boy Keith:

20041019 16:29:29 0001eb60 -------------------------------------
20041019 16:29:29 0001eb60 version    : 2.01 / 1.60
20041019 16:29:29 0001eb60 calltype   : SMTP
20041019 16:29:29 0001eb60 state      : rcpt
20041019 16:29:29 0001eb60 srvdom     : winserver.com
20041019 16:29:29 0001eb60 srvip      : 208.247.131.9
20041019 16:29:29 0001eb60 cip        : 160.36.56.50
20041019 16:29:29 0001eb60 cdn        : klutz.cs.utk.edu
20041019 16:29:29 0001eb60 from       : <[email-masked]>
20041019 16:29:29 0001eb60 rcpt       : <[email-masked]>
20041019 16:29:29 0001eb60 ruid       : 228761
20041019 16:29:29 0001eb60 testorder  : FLT RBL SPF CEP CBV
20041019 16:29:29 0001eb60 sapfilter  : pass (time:78)
20041019 16:29:29 0001eb60 saprbl     : testing
50.56.36.160.sbl.spamhaus.org
20041019 16:29:34 0001eb60 saprbl     : testing 50.56.36.160.list.dsbl.org
20041019 16:29:36 0001eb60 saprbl     : testing 50.56.36.160.bl.spamcop.net
20041019 16:29:37 0001eb60 saprbl     : pass (time:7781)
20041019 16:29:38 0001eb60 sapspf     : none (time:1078)
20041019 16:29:38 0001eb60 sapcep     : test from=cs.utk.edu
20041019 16:29:39 0001eb60 sapcep     : test cdn=klutz.cs.utk.edu
20041019 16:29:41 0001eb60 sapcep     : none (time:2938)
20041019 16:29:46 0001eb60 sapcbv     : total mx records: 1
20041019 16:29:46 0001eb60 try mx     : smtp.cs.utk.edu ip: 160.36.56.154
20041019 16:29:46 0001eb60 # connecting to 160.36.56.154
20041019 16:29:46 0001eb60 S: 220 klutz.cs.utk.edu ESMTP Postfix
20041019 16:29:46 0001eb60 C: NOOP WCSAP v2.01 Wildcat! Sender
Authentication Protocol http://www.santronics.com
20041019 16:29:46 0001eb60 S: 250 Ok
20041019 16:29:46 0001eb60 C: HELO mail.winserver.com
20041019 16:29:46 0001eb60 S: 250 klutz.cs.utk.edu
20041019 16:29:46 0001eb60 C: MAIL FROM: <>
20041019 16:29:47 0001eb60 S: 250 Ok
20041019 16:29:47 0001eb60 C: RCPT TO: <[email-masked]>
20041019 16:29:47 0001eb60 S: 250 Ok
20041019 16:29:47 0001eb60 C: RCPT TO:
<[email-masked]>
20041019 16:29:47 0001eb60 S: 554
<[email-masked]>: Relay access denied
20041019 16:29:47 0001eb60 C: QUIT
20041019 16:29:47 0001eb60 sapcbv     : 250
20041019 16:29:47 0001eb60 result     : accept (-1)
20041019 16:29:47 0001eb60 wcsap finish (18343 msecs)
20041019 16:35:46 0001eb9d -------------------------------------

In this case, keith's server complied.  Its not an open relay site.

PS: I removed your black list status.

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 9 Mar 2005 04:47:08 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 08, 2005 6:13 PM
Subject: Re: Site policy vs. HELO

>> When there is a legitimate sender, they will report any
>> issue if it mattered to you.

> Pray tell, when you bounce messages from the sender, how exactly do
> you expect the sender to report the problem?

Wasn't it pointed out this is a commercial product?

People will figure it out.  You did!

A) Send email to the sysop,  a requirement for acceptance.
B) You can go to the Support Web Site
C) You can use the Support Mailing list, and
D) If you don't have time for the bullshit, you can pick up the phone!

In all honesty, at this point, you are not a legitimate sender until I can
explain what happen to you specifically.  (see below)

> This particular problem didn't begin a day, a week, or a month ago.
> The first excerpt that I provided today was from 21 Oct 2004, 11:29
> EDT.  And that wasn't the first time.

Simply blabbing it out doesn't cut it.  I don't believe you unless you show
the proof. We have all our logs archived and I intend to analyze all your
transactions.

I will analyze all your transactions for the past two years and see whats
going on.  Something is very strange with your specific transactions.  This
needs to be investigated.

We will do a thorough analysis of your rejections. I should mention, that by
you not reporting this early on, this is NOT the norm.   You didn't report
it so you have a small part of being part of the problem.

Maybe this is pointing out a new unfortunate attitude we need to help put a
halt to - Poor Email Transaction Operations should not be acceptable.  We
have taken it for granted that SMTP compliancy is not a requirement hence we
have the bad transactions/spoofing issues and we have a growing tendency to
just drop mail at will.

Nonetheless, I don't know what your motive is behind all this.

What are you attempting to prove?  What is your point behind all this?
Are you simply saying the
methodology is wrong and unreliable?  Are you saying it is not possible to
apply stronger SMTP transaction management successfully?

I proved otherwise and this mishap with you, whatever is going on, isn't a
measuring stick to throw the baby out with the bath water.

If you want to prove there is a problem with the WCSAP protocol,  I invite
you to use the following URL to test it.  I would love to see false
positives where the SMTP process itself should be aware of.  Again, it is
about SMTP compliancy.  Turn on High Log verbosity to see the complete
details:

        http://www.winserver.com/testwcsap

example input:

        cip        : 207.172.4.61
        cdn        : smtp02.mrf.mail.rcn.net
        from       : [email-masked]

I also invite to see our 1.5 years worth of statistics:

        http://www.winserver.com/spamstats

hmmmmmm,  I see you did look at all this back in Oct 21, 2004.  Thing are
starting a little more sense here with missing WCSMTP -> WCSAP log entries.
You used the TESTWCSAP directly in Oct 21, 2004 successfully but you failed
with a SMTP transaction.

I'll be back with the complete analysis.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Date: Wed, 09 Mar 2005 01:31:10 -0700
To: "Carl S. Gutekunst" <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Cc: [email-masked]
Sender: [email-masked]

At 09:34 PM 3/8/2005 -0800, Carl S. Gutekunst wrote:

>David MacQuigg wrote:
>
>>What I would most like to see here is a standard so simple and 
>>non-controversial that it need not get all the way to final status before 
>>people start following it.  Putting it on the standards track could do this.
>
>The IETF tried this; it was called MARID (MTA Authorization Records In 
>DNS). The original charter was well focused and predicated on the 
>assumption that a simple and field-proven solution was close at hand. But 
>the effort failed, primarily because (IMHO) the problems of MTA 
>authentication and authorization proved difficult to isolate. The "simple" 
>solutions crossed more operational boundaries and reached deeper into the 
>infrastructure than their authors realized, causing controversy among even 
>their most ardent supporters. Solutions that truly were simple were 
>ignored for lacking in breadth of scope and "feel good" impact.
>
>In short, MARID failed because a simple, non-controversial, and 
>all-encompassing solution to Internet Mail Authentication does not seem to 
>exist.

My understanding was that MARID broke up because of Microsoft's surprise 
disclosure of very broad patent claims threatening any method that uses IP 
authentication.
  http://podcast.resource.org/rf-rfc/index.html#item0003 - first hand 
account of the "MARID Fiasco"
  <http://www.taugh.com./weblog/patapp.html>http://www.taugh.com./weblog/patapp.html 
- John Levine's analysis of Microsoft's patent claims

Regardless, IETF should not have given up because the MARID group broke 
up.  Maybe they *were* trying to put too much into a unified proposal, or 
maybe there was deliberate disruption, I don't know, and it doesn't 
matter.  IETF should have continued, perhaps with a different group, 
focusing on what are the common elements of the different proposals.  Every 
IP authentication method needs to communicate the results of its 
authentication.  If the hard-line advocates can't agree on a simple format 
for an authentication header, then a neutral expert should propose the 
format.  I understand that expert advocates can make mind-boggling 
arguments, but that is where IETF needs to exercise its engineering 
judgement.  Give us a simple format that all sides can live with, even if 
none are happy with less than total victory for their side.

>Yet progress is being made. I'm going to be making field trials of several 
>promising technologies in the next week or two, notably Domain Keys and 
>BATV. They don't have big marketing budgets behind them, and they aren't 
>panaceas, but they do provide clean solutions to specific parts of the 
>problem and are simple to pilot.

There does not seem to be much discussion of progress.  The press is very 
negative, and the open-source group seems to have lost its 
direction.  Maybe I'm just not reading the right reports or newsgroups.

I know Microsoft is working full speed behind closed doors.  Maybe we 
should just wait for them to provide us a complete solution.  My worry is 
that there really will be some terrible technical problem, inserted by 
their legal department or whatever, or that the opposition from the open 
source community will be so strong that it takes years before we have a 
widely accepted de-facto standard.

The real issues aren't technical, but social engineering.  There is a great 
benefit for all at the end, but an immediate cost for the players that need 
to make the next move (ISPs, spam filter companies, and spam 
blocklists).  It looks to me like a logjam with the pressure building, 
ready for a nudge to break it loose.  IETF could provide that nudge with a 
simple standard focusing on just the items needed for 
inter-operability.  Give us a format for the authentication headers, so we 
can start implementing domain ratings in our spam filters.

>I would also like to direct your attention to a marvelous paper by Brett 
>Watson of Macquarie University detailing all the work we have to do 
>*after* we have a working authentication system. Authentication is a 
>necessary first step to controlling spam, but by no means sufficient:
>
>    http://www.ceas.cc/papers-2004/140.pdf

Excellent paper, but a little out of date.  Looks like maybe June 2004 from 
the discussion, but I can't tell for sure.  Anyway its great to see someone 
actually giving some serious thought to what the email world will look like 
assuming authentication works.  My own view is a bit more positive.  If 
spam was a publicly traded company, I would be shorting their stock. 
:>)  See my "Spam Scenarios" at 
http://www.ece.arizona.edu/~edatools/etc/  Give me an interesting 
challenge, some way you think spam will survive, and I will think of a 
response and add it to these scenarios.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

Date: Tue, 08 Mar 2005 21:34:02 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

David MacQuigg wrote:

> What I would most like to see here is a standard so simple and 
> non-controversial that it need not get all the way to final status 
> before people start following it.  Putting it on the standards track 
> could do this. 

The IETF tried this; it was called MARID (MTA Authorization Records In 
DNS). The original charter was well focused and predicated on the 
assumption that a simple and field-proven solution was close at hand. 
But the effort failed, primarily because (IMHO) the problems of MTA 
authentication and authorization proved difficult to isolate. The 
"simple" solutions crossed more operational boundaries and reached 
deeper into the infrastructure than their authors realized, causing 
controversy among even their most ardent supporters. Solutions that 
truly were simple were ignored for lacking in breadth of scope and "feel 
good" impact.

In short, MARID failed because a simple, non-controversial, and 
all-encompassing solution to Internet Mail Authentication does not seem 
to exist.

Yet progress is being made. I'm going to be making field trials of 
several promising technologies in the next week or two, notably Domain 
Keys and BATV. They don't have big marketing budgets behind them, and 
they aren't panaceas, but they do provide clean solutions to specific 
parts of the problem and are simple to pilot.

I would also like to direct your attention to a marvelous paper by Brett 
Watson of Macquarie University detailing all the work we have to do 
*after* we have a working authentication system. Authentication is a 
necessary first step to controlling spam, but by no means sufficient:

    http://www.ceas.cc/papers-2004/140.pdf

<csg>

Date: Tue, 08 Mar 2005 19:01:19 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Has the IETF dropped the ball?
Sender: [email-masked]

At 04:34 PM 3/8/2005 -0500, Bruce Lilly wrote:

>On Tue March 8 2005 11:51, David MacQuigg wrote:
>
> > The public is getting mad as hell about spam.  There *will* be a solution
> > to this problem.  If the IETF doesn't provide it
>
>Spam is a social problem. The IETF is an Engineering group. Engineers
>don't solve social problems (as engineers).

The most rewarding efforts in my career have been when I can solve a 
problem that has social benefit.  That is what has me intrigued by this 
spam problem.  I thought of a solution two years ago ( very much like SPF 
), but not knowing anything about TCP/IP, I allowed myself to be talked out 
of it by the experts.

> > some politicians or
> > bureaucrats will.
>
>Unlikely. They seem to create more problems, such as the aptly
>named CAN-SPAM legislation, which confirms that spammers indeed
>can spam.

I didn't say the solution would work. :>)

> > I may not understand all that is going on in the IETF,
> > but it sure looks like they dropped the ball.
>
>Let's try to objectively and calmly consider which ball you
>claim has been dropped.
>
> > They have allowed the
> > standards effort
>
>Which standards effort? IETF standardizes many things. Are
>you complaining about IP? TCP? HTTP? SIP? ACAP? DHCP? What,
>precisely?

Email authentication.

> > to disintegrate into a bunch of programmers squabbling
> > over details that are not needed in a universal standard.
>
>"Universal standard" for what?

A standard allowing MTAs using different authentication methods to 
communicate with each other and with the other components in an email 
system.  For example, how should a spam filter read the headers and 
determine the domain to be rated when a forwarder is involved in the transfer?

> > If we continue
> > the present path, it may be years before there is a clear "victory" in the
> > battle between competing and incompatible standards.
>
>It is unclear precisely which standards you are categorizing as
>"competing and incompatible".  IETF standards become standards
>through one of two processes, as detailed in RFC 2026.  Either
>there is some procedure that is not suitable for phased review
>and has no interoperability considerations, in which case it is
>subject to scrutiny and possibly approved as BCP, which goes
>into immediate effect with full force of a standard.  The other
>process is the "Standards Track", which begins with a Proposed
>Standard, which is itself subject to review (in Internet-Draft
>form), is supposed to contain no known technical omissions, to
>have resolved design choices, and to be well-understood.  If
>there is "a bunch of programmers squabbling over details", that
>is a good indication that design choices have not been resolved,
>and/or that there is something which is not well-understood,
>and/or there are some technical omissions. After remaining at
>Proposed Standard for at least 6 months, and after there are at
>least two independent and interoperable implementations, a
>specification can progress to Draft Standard (after formulated
>as a new Internet-Draft and subjected to the review process).
>After at least another 4 months and when there is demonstrable
>wide support, the specification may finally become a Standard.
>Proposed Standards and Draft Standards are not Standards.
>Sometimes there are competing design choices, and no clear way
>to make an objective decision that would be required to enter
>the Standards Track at Proposed Standard.  In such cases, the
>competing design specifications might be published as Experimental
>RFCs so that a rational choice can be made based on implementation
>experience.  Experimental RFCs are not standards, and cannot
>become standards except by one of the processes mentioned above
>(involving a new Internet-Draft and the review process).
>Experimental specifications might well be "competing and
>incompatible"; their publication (but let's be clear; they are
>in no way standards) provides an opportunity to evaluate the merits
>and shortcomings of those specifications in order to gain
>understanding, resolve design choices, and ensure that there are
>no technical omissions.

What I would most like to see here is a standard so simple and 
non-controversial that it need not get all the way to final status before 
people start following it.  Putting it on the standards track could do this.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 18:13:15 -0500
Sender: [email-masked]

On Tue March 8 2005 16:24, Hector Santos wrote:

> When there us a legitimate sender,  they will report any
> issue if it mattered to you.

Pray tell, when you bounce messages from the sender, how exactly do
you expect the sender to report the problem?

> In this case, we add a backend beta ware issue -  a bug which I need to find
> out so I appreciate the report.  A reverted to a day earlier version for now
> and restarted. I can see your message came in now for which I will respond
> to now.

This particular problem didn't begin a day, a week, or a month ago.
The first excerpt that I provided today was from 21 Oct 2004, 11:29
EDT.  And that wasn't the first time.

To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 08 Mar 2005 14:50:59 -0800
Sender: [email-masked]

David MacQuigg <[email-masked]> writes:

> The public is getting mad as hell about spam.  There *will* be a
> solution to this problem.  If the IETF doesn't provide it, some
> politicians or bureaucrats will.  I may not understand all that is going
> on in the IETF, but it sure looks like they dropped the ball.

Your confusion may be cleared up by realizing that the IETF, like most
standards organizations, does far better at clearly documenting the right
way to do something after that's been hashed out over time than it does at
making up a new way of solving a brand new problem.  The latter is more
properly termed "research and development" and is something that works
most effectively in small teams of smart people rather than in large
committees worried about getting all the details right.

Given that no clear technical solution has yet emerged from the pack on
dealing with spam, it doesn't surprise me at all that the IETF has not
been able to pick a clear winner and say "everyone should do this."

> They have allowed the standards effort to disintegrate into a bunch of
> programmers squabbling over details that are not needed in a universal
> standard.

I don't think all those people are just stupid and squabbling for no
reason.  I don't think we've figured out a great solution and are just
failing to implement it because we're fighting over trivia.  Rather, I
think people have proposed a huge number of solutions that range in
quality from hideous to barely mediocre, without a *good* solution in the
lot, and much of the arguing that you're seeing is over whether it's
worthwhile to try to implement one of the several barely mediocre
solutions and deal with the resulting breakage because nothing better
seems to be appearing, or to wait for more R&D.

> If we continue the present path, it may be years before there is a clear
> "victory" in the battle between competing and incompatible standards.

And how do you think that the IETF should pick among the current bunch?
Drawing lots?  The reason why there's no victory is that none of them are
clearly better, and they all take very different approaches.

The fundamental problem of spam is that we want to use a system
specifically designed to not use authentication to either reject all mail
from "bad people" (where the definition of who's bad varies wildly from
one person to another) or to instead reject all "bad mail" (with a similar
wild variation in definition).  This isn't exactly an easy problem.  This
is certainly not a problem that we understand well enough that all that's
needed now is to get a few experts in a room and write the final draft.
For one thing, it's not even clear that it's *possible* to retrofit
authentication into the e-mail system in such a fashion that doesn't break
so much of e-mail that it would be better to just use a different protocol
entirely, and that's just one of the many issues that arises.

Yes, people are frustrated, but there is little correlation between how
frustrated people are and how easy the problem is to solve.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 17:39:23 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Cc: "Hector Santos" <[email-masked]>
Sent: Tuesday, March 08, 2005 2:51 PM
Subject: Re: Site policy vs. HELO

>
> On Tue March 8 2005 11:36, Hector Santos, a.k.a.
<[email-masked]> wrote:
>
> > Over the pass year and a half, we have proved that by increasing the
level
> > of SMTP compliancy required by senders, you can address an extremely
high
> > rejection rate with a very low to non-existence false positives.   100%
> > based on SMTP compliancy.
>
> Perhaps you're not paying attention to the false positives:

Now why would we do that?

This is a premium cost commerical product installed in thousands of sites
world wide.  Support and quality operations is a requirement, not an option.
Everyone always checking for operational problems in rejections especially
since there is such a high rejection rate,  some can't believe their eyes.
So the early scrunity was expected.   By far, whether you wish to believe it
or not,  both false positive and negatives are statisticaly nil.

If and when there is a problem, there will always be a report to you and it
is alway explainable.

Guess who is not reporting problems?  The spoofers, the spammers, the
malicious transactions which represents 60-80% of all transactions.

You found one false positive against our system. I appreciate the report,
the issue was corrected and.I sincerely hope that you don't use this weight
your views on email authentication effots.

> ==============================================================
> This message was created automatically by mail delivery software (Exim).
>
> A message that you sent could not be delivered to one or more of its
> recipients. This is a permanent error. The following address(es) failed:
>
>   [email-masked]
>     SMTP error from remote mailer after RCPT
TO:<[email-masked]>:
>     host mail.winserver.com [208.247.131.9]: 550 Return Path not
verifiable.
>
> ------ This is a copy of the message, including all the headers. ------
>
> Return-path: <[email-masked]>
> ===============================================================
>
> Now consider for a moment how silly that transaction was:

> o Everything from the client side was 100% SMTP compliant

So you say.   For the transaction we did not have any proof of the sender
authorization take place.  You and I know you are compliant because you sent
mail to us before and you pass the SMTP compliancy test.    In this case,  I
was a fault in our sub-process here. It was a legitmate bug that needs to be
corrected. It didn't even get a chance to run.  First time that happen. So
again I appreciate the input.

> o The return code claiming "Return Path not verifiable" had the
>   predictable result of sending a notice (excerpts above) to
>   precisely that path which is (incorrectly) claimed to be
>   "not verifiable"

And you reported the issue.  Exactly what we would expected if and when we
do get false positives which is a very low occurance.  Trust me, our
customers will report problems.  Got me today saying his user was getting
rejected.  After seeing his logs,  he was running into the SPF forwarding
problem and the problem was solved using a white list.  Simple, Done,
Customer happy!

You have shown nothing thus far.

> > Most modern SMTP systems are compliant and legitimate senders are indeed
> > compliant.
>
> As the case above...

Exactly.  You are legit, thus you will report it as most will in practice,
exactly what I said will happen.

> Somebody needs to pay more attention to unwarranted rejection
> of legitimate mail.

Someone did - YOU!!

> I predict that the courtesy copy of this message will also
> bounce for no good reason.

Bruce, what is your problem?   If you have some true input here, just
provide it and quit the nonsense?

We are not just doing this to be a nuisance.   Using SMTP compliancy works,
plain and simple, and most will see this immediately  when they put it into
practice.   We have the proof.  It is not just talk.

Will they exist false positives?  I wouldn't expect there not be.  But what
makes this incredibly successful is that it indeed works with a very low
false positive rate and nearly always when there is a problem, it is always
reported. There is always a reason for a false positive and a BUG doesn't
count.  And just in case you have more nonsense, this pre-release is only in
the hands of a few field testers.  Whatever happen, I don't know. I have not
checked yet. But it was not an expected situation and it is completely
unrelated to the protocol.  What I will be checking is why there wasn't a
temporary 45x response code instead.

Finally, the SMTP compliancy is simple:

HELO/EHLO

1) Check for syntax compliance
2) Check for Domain Literal IP match
3) Check for local domain spoofs

All optional.  No DNS overhead.  Provides high rejection rates. By far,
little to no false positives.  The false positives that did occur in the
early stages were:

- Indicative of a poor client setup
- Broken legacy software, which were updated or replaced.

MAIL FROM:

1) Check for syntax compliance
2) Per 2821, delay validation until Forwarding path is known.

Currently, only anonymous local final destination transactions are checked.
Remote transactions require a standard SMTP level authentication process,
either with SMTP AUTH, POPB4SMTP or Allow IP tables.  No need for Mail From
Validations if the sender is authenticated.

Our main validation is based on a CBV.  Very few false positives.  I'm sure
there are zombies sites but its rare and are quickly picked up when you
analyze why you got a spam!

The point in all this is that are many systems that do nothing at SMTP and
this is the heart of the spoofing/spam problem.  Using some simple SMTP
compliancy checking will go a long way.  I am sure you don't believe that,
but its true based on real production experience.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

To: [email-masked]
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 08 Mar 2005 14:37:43 -0800
Sender: [email-masked]

David MacQuigg <[email-masked]> writes:

> A Spam Bounce is a message generated in response to a spam, or what is
> suspected of being spam.  This could be a reject message, or a challenge
> message, or a bounce generated by the recipient of the spam.  In the
> past, these have been a problem, because they generally go to a forged
> address, where they become just more spam.

That sure sounds like a Delivery Status Notification to me.  I don't see
why it should be treated differently from any other DSN.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 08 Mar 2005 17:00:08 -0500
From: "Eric A. Hall" <[email-masked]>
Cc: ietf-smtp <[email-masked]>
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

On 3/8/2005 11:36 AM, Hector Santos wrote:

> Over the pass year and a half, we have proved that by increasing the
> level of SMTP compliancy required by senders, you can address an
> extremely high rejection rate with a very low to non-existence false
> positives.   100% based on SMTP compliancy.

I'm seeing just the opposite. I've been doing some expirementation with
some strict SMTP rules in SpamAssassin (looking for things like literal
HELO, mismatched rDNS, etc), and am getting reports of extremely high
error rates from testers. [1]

Note that I implemented these tests in SA so that I could encourage people
to STOP doing hard-rejects tests at the MTA level, due to very high
false-positive rates. I thought that by assigning a probability (as
opposed to a simple pass-fail test) that these tests would be more
effective, but from what I've seen so far, there are just too many
mis-managed mail servers out there to even establish a probability rate
that's acceptable.

There are, of course, some very useful pass-fail MTA tests (including
things like non-existent domain names), but not nearly as many as I
suspected going into this.

[1] http://www.rulesemporium.com/forums/showthread.php?s=&threadid=105

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Has the IETF dropped the ball?
Date: Tue, 8 Mar 2005 16:34:48 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Tue March 8 2005 11:51, David MacQuigg wrote:

> The public is getting mad as hell about spam.  There *will* be a solution 
> to this problem.  If the IETF doesn't provide it

Spam is a social problem. The IETF is an Engineering group. Engineers
don't solve social problems (as engineers).

> some politicians or  
> bureaucrats will.

Unlikely. They seem to create more problems, such as the aptly
named CAN-SPAM legislation, which confirms that spammers indeed
can spam.

> I may not understand all that is going on in the IETF,  
> but it sure looks like they dropped the ball.

Let's try to objectively and calmly consider which ball you
claim has been dropped.

> They have allowed the  
> standards effort

Which standards effort? IETF standardizes many things. Are
you complaining about IP? TCP? HTTP? SIP? ACAP? DHCP? What,
precisely?

> to disintegrate into a bunch of programmers squabbling  
> over details that are not needed in a universal standard.

"Universal standard" for what?

> If we continue  
> the present path, it may be years before there is a clear "victory" in the 
> battle between competing and incompatible standards.

It is unclear precisely which standards you are categorizing as
"competing and incompatible".  IETF standards become standards
through one of two processes, as detailed in RFC 2026.  Either
there is some procedure that is not suitable for phased review
and has no interoperability considerations, in which case it is
subject to scrutiny and possibly approved as BCP, which goes
into immediate effect with full force of a standard.  The other
process is the "Standards Track", which begins with a Proposed
Standard, which is itself subject to review (in Internet-Draft
form), is supposed to contain no known technical omissions, to
have resolved design choices, and to be well-understood.  If
there is "a bunch of programmers squabbling over details", that
is a good indication that design choices have not been resolved,
and/or that there is something which is not well-understood,
and/or there are some technical omissions. After remaining at
Proposed Standard for at least 6 months, and after there are at
least two independent and interoperable implementations, a
specification can progress to Draft Standard (after formulated
as a new Internet-Draft and subjected to the review process).
After at least another 4 months and when there is demonstrable
wide support, the specification may finally become a Standard.
Proposed Standards and Draft Standards are not Standards.
Sometimes there are competing design choices, and no clear way
to make an objective decision that would be required to enter
the Standards Track at Proposed Standard.  In such cases, the
competing design specifications might be published as Experimental
RFCs so that a rational choice can be made based on implementation
experience.  Experimental RFCs are not standards, and cannot
become standards except by one of the processes mentioned above
(involving a new Internet-Draft and the review process).
Experimental specifications might well be "competing and
incompatible"; their publication (but let's be clear; they are
in no way standards) provides an opportunity to evaluate the merits
and shortcomings of those specifications in order to gain
understanding, resolve design choices, and ensure that there are
no technical omissions.

> I understand the details are important, but how much of what is in the 
> current proposals is really necessary for these different methods to work 
> together in the same transfer?

It is a complete mystery exactly which "current proposals" you
are referring to here on the ietf-smtp mailing list, which is
part of no current IETF working group, has no chair, no charter,
and no work items.

> Surely the IETF can get together a few neutral experts who can listen to 
> advocates

Advocates for what?

> from all sides, and come up with a standard on just the few items  
> needed for *interoperability*.

Interoperability of what?

> Then we can break the current logjam,

Which "logjam"?

> and   
> all parties can do their own thing within the standard.

What standard?

> More importantly,  
> we can tell ISPs, companies in the spam fighting business, and the public 
> "We have a standard. Get moving."

It is unclear precisely what standard you think is possible
within the IETF regarding the social problem of spam.  The
IETF is no more able to define whether or not a particular
message is "spam" than it is able to determine whether or
not a particular group of people constitute a "country"
(RFC 1591, bottom of page 6).

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 16:24:15 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Well Bruce,

Please don't get offended. I can assure you we will not be offended if there
was a problem found.

You told us "nothing."   As far as I am concern, you proved my point time
and time again.  When there us a legitimate sender,  they will report any
issue if it mattered to you.  In fact is,  the times you do write to me
your address is always verified without a problem.

In this case, we add a backend beta ware issue -  a bug which I need to find
out so I appreciate the report.  A reverted to a day earlier version for now
and restarted. I can see your message came in now for which I will respond
to now.

But bear in mind, this  is absolutely unrelated to any sender authorization
fault you might be against.  Your points on the matter is completely
unwarranted and by far, the statistics we have over 1.5 years of operations
in thousands of sites completely nullifieds your input on the matter.   In
short, you have no practical experience here and it is 100% completely
"feelings" base.

Meet you in the next message.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 08, 2005 3:14 PM
Subject: Re: Site policy vs. HELO

On Tue March 8 2005 14:51, Bruce Lilly wrote:

> I predict that the courtesy copy of this message will also
> bounce for no good reason.

Wheee! Here we go:
===========================================================
I'm sorry to have to inform you that the message returned
below could not be delivered to one or more destinations.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the message returned below.

The Postfix program

<[email-masked]>: host mail.winserver.com[208.247.131.9]
said:
550 Return Path not verifiable. (in reply to RCPT TO command)
===========================================================

I hate to have to say it, but "I told you so".

Date: Tue, 8 Mar 2005 20:28:47 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--1870869256-510682828-1110313727=:1602
Content-Type: TEXT/PLAIN; charset=iso-8859-1
Content-Transfer-Encoding: QUOTED-PRINTABLE

On Tue, 8 Mar 2005, Bruce Lilly wrote:

> <[email-masked]>: host mail.winserver.com[208.247.131.9]=
 said:
> =A0 =A0 550 Return Path not verifiable. (in reply to RCPT TO command)

I also like the protocol breakage in the response to EHLO (looks like the
same bug affects the banner too).

smtpc: < 220-winserver.com Wildcat! ESMTP Server v6.0.451.3 ready
smtpc: < 220-winserver.com Wildcat! ESMTP Server v6.0.451.3 ready
smtpc: < 220-************** WARNING:  FOR AUTHORIZED USE ONLY! ************=
**********
smtpc: < 220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY COMP=
UTERS    *
smtpc: < 220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE UNS=
OLICITED *
smtpc: < 220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRIC=
T ACCESS *
smtpc: < 220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.             =
         *
smtpc: < 220 **************************************************************=
**********
smtpc: > EHLO cyan.csi.cam.ac.uk
smtpc: < 250-winserver.com, Pleased to meet you.
smtpc: < 250-winserver.com, Pleased to meet you.
smtpc: < 250-SIZE 5120000
smtpc: < 250-ETRN
smtpc: < 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
smtpc: < 250-AUTH=3DLOGIN
smtpc: < 250 HELP

Tony.
--=20
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: EAST OR NORTHEAST 3 OR 4. FAIR. GOOD.
--1870869256-510682828-1110313727=:1602--

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 15:14:36 -0500
Sender: [email-masked]

On Tue March 8 2005 14:51, Bruce Lilly wrote:

> I predict that the courtesy copy of this message will also
> bounce for no good reason.

Wheee! Here we go:
===========================================================
I'm sorry to have to inform you that the message returned
below could not be delivered to one or more destinations.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the message returned below.

                        The Postfix program

<[email-masked]>: host mail.winserver.com[208.247.131.9] said:
    550 Return Path not verifiable. (in reply to RCPT TO command)
===========================================================

I hate to have to say it, but "I told you so".

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 14:51:10 -0500
Cc: "Hector Santos" <[email-masked]>
Sender: [email-masked]

On Tue March 8 2005 11:36, Hector Santos, a.k.a. <[email-masked]> wrote:

> Over the pass year and a half, we have proved that by increasing the level
> of SMTP compliancy required by senders, you can address an extremely high
> rejection rate with a very low to non-existence false positives.   100%
> based on SMTP compliancy.

Perhaps you're not paying attention to the false positives:

==============================================================
This message was created automatically by mail delivery software (Exim).

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  [email-masked]
    SMTP error from remote mailer after RCPT TO:<[email-masked]>:
    host mail.winserver.com [208.247.131.9]: 550 Return Path not verifiable.

------ This is a copy of the message, including all the headers. ------

Return-path: <[email-masked]> 
===============================================================

Now consider for a moment how silly that transaction was:
o Everything from the client side was 100% SMTP compliant
o The return code claiming "Return Path not verifiable" had the
  predictable result of sending a notice (excerpts above) to
  precisely that path which is (incorrectly) claimed to be
  "not verifiable"

> Most modern SMTP systems are compliant and legitimate senders are indeed
> compliant.

As the case above...
 
> So I propose that we take a step back and look at RFC 2821 and discuss what
> areas need to change to provide SMTP compliancy enforcement at the first
> step towards addressing the spam problem.

RFCs 821 (as amended by RFC 1123 and other RFCs) and 2821 *define*
SMTP compliance requirements.  This "SMTP compliancy" is a red
herring.
 
> I hope I am not off based

They say that hope springs eternal...

> Someone needs to get a hold of the ball and begin making changes to RFC
> 2821.

Somebody needs to pay more attention to unwarranted rejection
of legitimate mail.

I predict that the courtesy copy of this message will also
bounce for no good reason.

Date: Tue, 08 Mar 2005 12:11:02 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>
cc: Vince Sabio <[email-masked]>, ietf-smtp <[email-masked]>
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

Hector,

I'm not going to try for a complete response here; much too busy 
this week...

	* Any time I answer a question on this list about 2821
	and what it says, I'm answering in terms of my
	understanding, as editor, of what is in that document
	and what the WG wanted.  There are things in it that I
	don't like, but the document is as close to instructions
	from the WG as I could make it.   If I'm answering in
	terms of personal opinions about that spec, I'll be
	pretty clear about it.
	
	* In principle, it wouldn't be hard to produce an
	applicability statement document that changed 2821's
	conformance rules to ban legacy (821-only)
	implementations.  In practice, if the IETF were to
	consider such a thing, I'd predict a lot of controversy
	and either a "no" decision or a standard that was widely
	ignored: while you might make a rational site decision
	to reject HELO, I can't imagine any sane implementer of
	a commercial implementation removing the backward
	compatibility.

If you want to pursue the above, _write a draft_, then contact 
the applications ADs and discuss a WG charter.  Ranting on this 
mailing list is fairly useless, whether you are right or wrong.

    john

--On Tuesday, March 08, 2005 11:36 AM -0500 Hector Santos 
<[email-masked]> wrote:

>
> ----- Original Message -----
> From: "John C Klensin" <[email-masked]>
> To: "Keith Moore" <[email-masked]>
> Cc: "Vince Sabio" <[email-masked]>; "ietf-smtp"
> <[email-masked]> Sent: Tuesday, March 08, 2005 8:40 AM
> Subject: Re: Site policy vs. HELO
>
>...
> Over the pass year and a half, we have proved that by
> increasing the level of SMTP compliancy required by senders,
> you can address an extremely high rejection rate with a very
> low to non-existence false positives.   100% based on SMTP
> compliancy.
>
> That should not be a surprise to no one:  By industry
> measurements, over 60-80% of all transactions are spoofed, bad
> user or NXDOMAIN.
>
> The questions we need to face are:
>
> How much longer do we accept malicious transactions to use
> non-compliant SMTP transactions?
>
> The argument of legacy issues as a reason not to even consider
> stronger SMTP compliant is not only a red herring but old
> thinking.
>
> Most modern SMTP systems are compliant and legitimate senders
> are indeed compliant.
>
> It is the exact segment of the population of malicious
> transaction who are not SMTP compliant thus exploiting the
> weak provisions by not applying pressure to comply.
>...

Date: Tue, 08 Mar 2005 09:51:15 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Has the IETF dropped the ball?
Sender: [email-masked]

The public is getting mad as hell about spam.  There *will* be a solution 
to this problem.  If the IETF doesn't provide it, some politicians or 
bureaucrats will.  I may not understand all that is going on in the IETF, 
but it sure looks like they dropped the ball.  They have allowed the 
standards effort to disintegrate into a bunch of programmers squabbling 
over details that are not needed in a universal standard.  If we continue 
the present path, it may be years before there is a clear "victory" in the 
battle between competing and incompatible standards.

I understand the details are important, but how much of what is in the 
current proposals is really necessary for these different methods to work 
together in the same transfer?  Not much I suspect.

Surely the IETF can get together a few neutral experts who can listen to 
advocates from all sides, and come up with a standard on just the few items 
needed for *interoperability*.  Then we can break the current logjam, and 
all parties can do their own thing within the standard.  More importantly, 
we can tell ISPs, companies in the spam fighting business, and the public 
"We have a standard. Get moving."

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: "Vince Sabio" <[email-masked]>, "ietf-smtp" <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Tue, 8 Mar 2005 11:36:17 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "John C Klensin" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: "Vince Sabio" <[email-masked]>; "ietf-smtp" <[email-masked]>
Sent: Tuesday, March 08, 2005 8:40 AM
Subject: Re: Site policy vs. HELO

> Yep.  I should have said
>
> "those weak methods increase the volume of spam, cause more
> false positives when they fail to work precisely, and generally
> reduce the transparency and reliability of the email network"

Hi John,

Do you have any emperical statistics, documentation, proof or otherwise or
specific "methods" which matches or comes close to these assertions?

I mean, we am not not seeing any of this at all.  Complete opposite.  There
is a major DNS issue with the DNS based email authentication proposals, but
there are ways around it.

> Does that sound like I'm not impressed with them as a partial
> solution?  :-(

What I do see is an unfortunate trend to focus on a payload "fuzzy" analysis
solution.  These are the ones I suspect will have severe impact on the
network, a increase of mail oblivion and decrease of proper system
notifications.

We need to remove all fuzzy analysis of mail from the SMTP client/server
transaction.

Over the pass year and a half, we have proved that by increasing the level
of SMTP compliancy required by senders, you can address an extremely high
rejection rate with a very low to non-existence false positives.   100%
based on SMTP compliancy.

That should not be a surprise to no one:  By industry measurements, over
60-80% of all transactions are spoofed, bad user or NXDOMAIN.

The questions we need to face are:

How much longer do we accept malicious transactions to use non-compliant
SMTP transactions?

The argument of legacy issues as a reason not to even consider stronger SMTP
compliant is not only a red herring but old thinking.

Most modern SMTP systems are compliant and legitimate senders are indeed
compliant.

It is the exact segment of the population of malicious transaction who are
not SMTP compliant thus exploiting the weak provisions by not applying
pressure to comply.

The argument that spammers will adjust to become SMTP compliant, to which I
said "GREAT!" it is exactly what we should want.   Make the spammer change
in our direction.

Once the spammer is SMTP compliant, they become more "traceable", given
strength to the US Federal law called CANSPAM and new ideas and new sender
authentication inventions will work even better.

But I always felt we were jumping the gun by trying to kludge ideas or
wrappers around an exploitable SMTP system, when in fact we need to get the
SMTP compliancy issue resolved first.  We did for our system and I consider
it a major success for our customers.

What are we going to do to strengthen SMTP transaction?

I propose we begin with complete review of the RFC-2821 document to remove
all ambiguity and to give developers the power to strengthen the
transaction.

We can't continue under the basis that we need not handle or skip the
validation process just because it is possible to be non-compliant.  We need
to get over this hurdle and in my opinion, until we do,  we will continue to
have these "get-no-where" "I'm right, you are wrong" discussions.

So I propose that we take a step back and look at RFC 2821 and discuss what
areas need to change to provide SMTP compliancy enforcement at the first
step towards addressing the spam problem.

I hope I am not off based, but I find much of I read here ridiculous,
spending more time bashing each other than getting any real significant work
done when in fact, its is all right there in front of us.

Someone needs to get a hold of the ball and begin making changes to RFC
2821.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Tue, 8 Mar 2005 09:45:50 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Tue March 8 2005 00:54, David MacQuigg wrote:

> A Spam Bounce is a message generated in response to a spam, or what is 
> suspected of being spam.  This could be a reject message, or a challenge 
> message, or a bounce generated by the recipient of the spam.  In the past, 
> these have been a problem, because they generally go to a forged address, 
> where they become just more spam.

First, "spam" is subjective; as Dave has noted, it is a social
phenomenon, not something that can be objectively differentiated
from an architectural point of view.  Second, "forged" is a loaded
term; there are legitimate reasons for setting a return-path (i.e.
destination for delivery notifications) to point to a particular
mailbox -- there is no objective criterion that one can use to
determine a "forged" return-path from a legitimate one; sending IP
address and inbound mailbox (for notifications) are unrelated,
message header address fields are part of end-to-end user-to-user
communication and play no role in transport, etc.

> Email authentication

No such critter, other than two cases:
1. single-hop authentication between parties having a prior
   arrangement and using SMTP AUTH, SSL/TLS (there are also
   some possible mechanisms for single-hop authentication
   which are not standardized).
2. end-to-end, user-to-user non-repudiation via established
   MIME security mechanisms (e.g. S/MIME and PGP/MIME).

The first case requires a prior arrangement which is inconsistent
with the email architecture as noted in Dave's draft.  Now you
might wish to discuss some proposed new architecture, but that's
a separate issue from documenting the existing architecture.

The second case is a user-level mechanism which therefore is not
applicable to delivery (transport level) notifications.  Moreover,
the second case is ineffective in the all-too-common case of a
machine owner who has installed a trojan for use by spammers,
in which case he is effectively no longer the (sole) owner, and
any authentication keys, etc. stored on the machine are available
to the (remote) spammer. [the first case is also ineffective
under such conditions, but affects a single hop rather than end-
to-end communication]

> will allow these Spam Bounces to be routed properly,  
> back the path they came.

You are assuming that delivery (transport) notifications always
go to the same hosts that are involved in forward transfer, and
that is simply incorrect; it is not unusual for large-volume
message handlers to use separate hosts for inbound and outbound
messages -- a host which has an SMTP client MTA for sending
outbound messages might well have no SMTP receiver and vice versa.
Looking at the email landscape from a higher altitude, a user
might use one ISP for sending messages (one with his "current"
connection; think of mobile users), and a different ISP for
receiving messages (including delivery notifications; his "home"
ISP).

> If the email is *not* spam

That's a subjective appraisal.

> , the Spam Bounce will   
> get back to where it belongs, most likely the same domain as the 
> Return-Path.  If it *is* spam, the Spam Bounce will be lost or ignored at 
> the point where the spam was injected.

You are presuming that categorization as "spam" is an objective
determination, that is is possible for an automaton implementing
an MTA to determine the precise *mailbox* for a non-delivery
notification separate from the architecturally-established source
of that information (i.e. the return-path) etc.  In short, you're
not talking about the current email architecture at all.

> Here are the Actors in a spam scenario:
> 
>     +------------+                         +-----------+
>     | Originator |                         | Recipient |
>     +-----+------+                         +-----------+
[...]
>              +---------+                    +---------+
>              | Spammer +-->   - - - -    -->|Forwarder+
>              +---------+                    +---------+

You're saying that the spammer is not the originator?

Date: Tue, 08 Mar 2005 08:40:22 -0500
From: John C Klensin <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Vince Sabio <[email-masked]>, ietf-smtp <[email-masked]>
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

--On Monday, March 07, 2005 5:41 PM -0500 Keith Moore 
<[email-masked]> wrote:

>
>> Really, applying anti-spam solutions that the spammers can
>> trivially and obviously work around is not a good investment.
>> Over time, arguably contributes to an increase in the total
>> volume of spam without accomplishing much else.
>
> ...other than to decrease transparency and reliability of the
> email network.

Yep.  I should have said

"those weak methods increase the volume of spam, cause more 
false positives when they fail to work precisely, and generally 
reduce the transparency and reliability of the email network"

Does that sound like I'm not impressed with them as a partial 
solution?  :-(

    john

Date: Mon, 07 Mar 2005 22:54:47 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

At 11:12 PM 3/7/2005 -0500, Bruce Lilly wrote:

>On Mon March 7 2005 19:27, David MacQuigg wrote:
>
> > We need to treat a "Spam Bounce" as a separate kind of special message, to
> > be handled differently than either DSNs or MDNs.  We had great 
> confusion in
> > a recent discussion on [spf-discuss], because I was trying to define
> > "Bounce" as something separate from DSN, but most think of the terms as
> > synonymous.  So I'm not sure what to call it, but "Spam Bounce" seems
> > reasonably descriptive and unambiguous.
>
>Not either as far as I'm concerned.  Why don't you try to explain
>what you mean by the term.

A Spam Bounce is a message generated in response to a spam, or what is 
suspected of being spam.  This could be a reject message, or a challenge 
message, or a bounce generated by the recipient of the spam.  In the past, 
these have been a problem, because they generally go to a forged address, 
where they become just more spam.

Email authentication will allow these Spam Bounces to be routed properly, 
back the path they came.  If the email is *not* spam, the Spam Bounce will 
get back to where it belongs, most likely the same domain as the 
Return-Path.  If it *is* spam, the Spam Bounce will be lost or ignored at 
the point where the spam was injected.  All forwarders from the recipient 
back to that point, however, will have an opportunity to do something 
useful with the bounce (improve their blocking/filtering, etc.).

Here are the Actors in a spam scenario:

    +------------+                         +-----------+
    | Originator |                         | Recipient |
    +-----+------+                         +-----------+
          |                                      ^
          |                                      |
          V Sender    Return-Path       Receiver |
      +---------+    +--------+             +----+-----+
      |         |    | Notice |<------------+ Destin-  |
      | Source  +--->| Handler|     DSN     |  ation   |
      |         |    |        |<---+        |          |
      +----+----+    +--------+    |        +----------+
           |                       |             ^
           V                       |             |
      +---------+             +----+----+   +----+----+
      |Forwarder+-->  - -  -->|Forwarder+-->|Forwarder|
      +---------+             +----+----+   +----+----+
                                                 ^
                          <-- Spam Bounce <--    |
             +---------+                    +---------+
             | Spammer +-->   - - - -    -->|Forwarder+
             +---------+                    +---------+

And here are the Agents:

                         +------+
              Originator | oMUA |<------------------------------+
                         +--+---+                               |
                            |    {smtp, submission              |
    Spammer                 V                                   |
        +------+         +------+                               |
        | MTA  |         | MSA  |<--------------------+         |
        +--+---+         +--+---+                     |         |
           |                |    {smtp                |         |
           |         Sender V                         |         |
           |             +------+                /+===+===+\    |
           .             | sMTA |                ||  dsn  ||    |
    Relays .      ^      +--+---+                \+=======+/    |
           .      |         |    {smtp              ^   ^       |
           | /+===+====+\   .                       |   |       |
           | || bounce ||   . Relays                |   |       |
           | \+========+/   .                       |   |       |
           V     <--        V                       |   |       |
        +------+         +------+                   |   |       |
        | MTA  |-------->| MTA  |                   |   |       |
        +------+         +--+---+                   |   |       |
        Relay               | Relay                 |   |       |
                            |                       |   |       |
                            |                       |   |       |
                   Receiver V                       |   |       |
                         +------+                   |   |   /+==+==+
                         | rMTA +-------------------+   |   || mdn ||
                         +--+---+                       |   \+=====+/
                            |    {local, smtp, lmtp     |       |
                            V                           |       |
                         +------+                       |       |
                         |      +-----------------------+       |
                         | MDA  |                               |
                         |      |<--------------------+         |
                         +-+--+-+                     |         |
                   local}  |  |                       |         |
                           V  |                       |         |
                     +------+ |                  /+===+===+\    |
                     | MS-1 | |                  || sieve ||    |
                     +-+--+-+ |                  \+=======+/    |
                       |  |   |  {pop, imap           ^         |
                       |  V   V                       |         |
                       | +------+                     |         |
                       | | MS-2 |                     |         |
                       | +--+---+                     |         |
                       |    |    {pop, imap, local    |         |
                       V    V                         |         |
                      +------+                        |         |
            Recipient | rMUA +------------------------+---------+
                      +------+

If the formatting is messed up on these figures, you can view the original 
at http://purl.oclc.org/net/edatools/email Spam Flows.txt

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date:  Tue, 08 Mar 2005 05:37:34 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 18
Sender: [email-masked]

David MacQuigg wrote:

> Frank made me do it. :>)

He, I only said that I'd prefer "MX" instead of "incoming MTA".

For bounces the text uses always the meaning "bounce message",
that should be clear.  

In 2821 it has also the meaning "reject" (SMTP error reply).
It could make sense to note that any "bounce messages" in that
case are actually created the client MTA, not by the rejecting
server MTA.  Some discussions about I18N in MARID were rather
surreal.  The poor server just can't do this, and if it would
try it anyway the client won't understand it.

                         Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Mon, 7 Mar 2005 23:21:37 -0500
Cc: [email-masked]
Sender: [email-masked]

On Mon March 7 2005 17:13, [email-masked] wrote:

> > I specifically mentioned delivery notification because a
> > sender may receive such a notification from any of the
> > distinct message-handling components, and the significance
> > of such notifications (and the user's reaction) depends
> > on the component.
> > 
> 
> A that makes it much more clear but I still disagree.
> section 2 is a (very) general discription of the mail architecture.
> error messages nitty gritty belongs elsewhere.

Nitty gritty, yes.  But notifications (success as well as
errors) are mentioned to illustrate the point; if I receive
a notification from a nearby MTA shortly after sending a
message (as has recently happened), it has quite a bit more
significance than a random notification from some random,
distant MTA (or anti-virus spamware, etc.).
 
> Maybe it is better to create a seperate section on messages with subsections for 
> - Delivery Status Notification 
> - Message Disposition Notification 
> - Message Filtering (SIEVE)

Filtering is an operation, not a message (while one thing
that SIEVE can do is sending "vacation"-like messages, that's
not its sole (or even primary) purpose).

And don't forget Message Tracking Status Notifications.

> List expansion and forwarding are done by the MDA, (they depend on the localpart of the email address)

OK, I'll buy that.

> the MDA sends the message to an MTA and the MTA sends it to other domains. 

Not if the alias/forwarding destination are local.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Mon, 7 Mar 2005 23:12:42 -0500
Cc: David MacQuigg <[email-masked]>
Sender: [email-masked]

On Mon March 7 2005 19:27, David MacQuigg wrote:

> We need to treat a "Spam Bounce" as a separate kind of special message, to 
> be handled differently than either DSNs or MDNs.  We had great confusion in 
> a recent discussion on [spf-discuss], because I was trying to define 
> "Bounce" as something separate from DSN, but most think of the terms as 
> synonymous.  So I'm not sure what to call it, but "Spam Bounce" seems 
> reasonably descriptive and unambiguous.

Not either as far as I'm concerned.  Why don't you try to explain
what you mean by the term.

Date: Mon, 07 Mar 2005 17:27:19 -0700
To: [email-masked], Bruce Lilly <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

At 10:13 PM 3/7/2005 +0000, [email-masked] wrote:

>Maybe it is better to create a seperate section on messages with 
>subsections for
>- Delivery Status Notification
>- Message Disposition Notification
>- Message Filtering (SIEVE)
>
>I would suggest to break section 4 services in to 2 sections
>one for messages  and one for services (or agents)
>Messages are things that are transfrered.
>Services are the things that allow that things to happen.
>
>the structure becomes then
>
>4 messages
>4.1 message components
>4.1.1 Envelope
>4.1.2 Message Header Fields
>4.1.3 Body
>4.1.4 Identity References in a Message
>  (but this section should use the names used in the section on services)
>
>4.2 Special messages
>4.2.1 Delivery Status Notification
>4.2.2 Message Disposition Notification
>4.2.3 Message Filtering (SIEVE)

We need to treat a "Spam Bounce" as a separate kind of special message, to 
be handled differently than either DSNs or MDNs.  We had great confusion in 
a recent discussion on [spf-discuss], because I was trying to define 
"Bounce" as something separate from DSN, but most think of the terms as 
synonymous.  So I'm not sure what to call it, but "Spam Bounce" seems 
reasonably descriptive and unambiguous.

A Spam Bounce could be generated by the rMUA or the MDA and passed up 
through the same MTAs by which it came.  Or it could be dropped at some 
MTA, depending on local policy and agreement between the two MTAs.  It 
should never go the path of DSNs or MDNs because at that point we have 
reason to believe that the headers are forged, and the "Originator" will 
consider it more spam.  There should be some easy way to identify Spam 
Bounces, like the Return-Path <null> on a DSN, to avoid interfering with 
normal mail and regular DSNs.

Until we have some common language, it is hard to even discuss these ideas.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Mon, 7 Mar 2005 18:06:53 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Frank,

How did you create this message?  The mailing list address was lost.  It
required a manual input.

Anyway, see comments below:

----- Original Message -----
From: <[email-masked]>
To: "Bruce Lilly" <[email-masked]>
Sent: Monday, March 07, 2005 5:13 PM
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative
domains

> - Message Filtering (SIEVE)

SIEVE is a specific implementation. There are many other models for mail
filtering.

> List expansion and forwarding are done by the MDA, (they depend on the
localpart of the email address) the MDA sends the message to an MTA and the
MTA sends it to other domains.

MDA is a final destination agent.  Not all models of a MDA will do list
expansions.  In our model, the list expansion is a completely separate
process altogether outside the SMTP process.

We need to be careful about how specific software you are using works.  I
believed Dave's document is a general model of the mail architecture.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

Cc: Vince Sabio <[email-masked]>, ietf-smtp <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Mon, 7 Mar 2005 17:41:15 -0500
To: John C Klensin <[email-masked]>
Sender: [email-masked]

> Really, applying anti-spam solutions that the spammers can
> trivially and obviously work around is not a good investment.
> Over time, arguably contributes to an increase in the total
> volume of spam without accomplishing much else.

...other than to decrease transparency and reliability of the email 
network.

To: Bruce Lilly <[email-masked]>
From: [email-masked]
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Mon, 7 Mar 2005 22:13:54 -0000
Sender: [email-masked]

Made me think again
Can we think of another name for the MDA?
And i think that a message is not a service. (see further in this email)

> On Sat March 5 2005 09:54, [email-masked] wrote:
> 
> > But i disagree with:
> > 
> > > Section 2.1 refers to the MHS as monolithic; it is not --
> > > it is composed of autonomous agents (MTAs, MSAs, MDAs,
> > > gateways, list expanders) and a user may receive a delivery
> > > notification of some sort from any of those agents.
> > 
> > For users the (image of the) MHS is monolithic.
> > In This section I would keep it that way.
> 
> I specifically mentioned delivery notification because a
> sender may receive such a notification from any of the
> distinct message-handling components, and the significance
> of such notifications (and the user's reaction) depends
> on the component.
> 

A that makes it much more clear but I still disagree.
section 2 is a (very) general discription of the mail architecture.
error messages nitty gritty belongs elsewhere.

Maybe it is better to create a seperate section on messages with subsections for 
- Delivery Status Notification 
- Message Disposition Notification 
- Message Filtering (SIEVE)

I would suggest to break section 4 services in to 2 sections
one for messages  and one for services (or agents)
Messages are things that are transfrered.
Services are the things that allow that things to happen. 

the structure becomes then

4 messages
4.1 message components
4.1.1 Envelope
4.1.2 Message Header Fields
4.1.3 Body 
4.1.4 Identity References in a Message
 (but this section should use the names used in the section on services)

4.2 Special messages
4.2.1 Delivery Status Notification 
4.2.2 Message Disposition Notification 
4.2.3 Message Filtering (SIEVE)

5 services

rest needs renumbering.

> > I like the concept of MDA and want to distinguish it more clearly from the MTA.
> > I would suggest the MTA only works with domain part of the message, the MDA with domain parts and local parts.
> > So the MTA does the transfering, the MDA does all the rest.
> > If an operation needs the localpart of the address it is a MDA function.
> > (auto responders, mailinglist actions, the SMTP VRFY, EXPN, ETRN commands ect.)
> > 
> > Architectonal ideas
> > Eexcept with LMDP, an MDA is always hiding behind an MTA.
> > LMDP uses SMTP-Direct (sending to a fixed ip address, not to a public MX record)
> 
> LMTP?
> 
Sorry slip of the finger.

> > There is no direct contact between MTA outside the administrative domain and the MDA in that administrative domain.
> 
> Architecturally (or topologically), an MDA can only appear
> on the recipient side of all MX hosts for the recipient's
> domain. There may, however, be MTAs in that area (some
> administrative entities configure their systems with a
> border MTA for aliasing, etc.).
> 

> > Within one administrative domain globaly valid MX records are NOT used.
> > MX records are used for transfer between seperated administrative domains.
> > within an Administrative domain SMPT-direct. (sending to only one MTA) is used.
> 
> Except for list expansion and other forwarding operations
> that extend beyond the (DNS) domain.
> 

List expansion and forwarding are done by the MDA, (they depend on the localpart of the email address) the MDA sends the message to an MTA and the MTA sends it to other domains.

> > What is your standpoint about Users in general?
> 
> Users of email (authors, senders, moderators, and recipients)
> are part of the end-to-end communications path. Message
> body and header originator/origination-date/destination/
> information/resent fields (RFC 2822) [and some extension
> fields] are exclusively for use of the users (and as assisted
> by their UAs) and should never be altered by the MHS.

Date: Mon, 07 Mar 2005 06:43:36 -0700
To: <[email-masked]>
From: David MacQuigg <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

To: Dave Crocker <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains

At 12:00 AM 3/7/2005 -0600, you wrote:

>  On Sat, 05 Mar 2005 15:50:41 -0700, David MacQuigg wrote:
> >  We should probably add "Spammer" as a fundamental component of the
> >  architecture.

:>)

>In that case, we should also add physician, sales person, and manager.
>
>spam is a social phenomenon, not a technical one.  It is a type of email 
>use, rather than a component of an architecture.

You missed my :>) at the end of that suggestion.

Frank made me do it. :>)

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

From: Dave Crocker <[email-masked]>
To: David MacQuigg <[email-masked]>, <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Date: Mon, 7 Mar 2005 00:00:08 -0600
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

 On Sat, 05 Mar 2005 15:50:41 -0700, David MacQuigg wrote:
>  We should probably add "Spammer" as a fundamental component of the
>  architecture.

In that case, we should also add physician, sales person, and manager.

spam is a social phenomenon, not a technical one.  It is a type of email use, rather than a component of an architecture.

 d/
 --
 Dave Crocker
 Brandenburg InternetWorking
 +1.408.246.8253
 dcrocker  a t ...
 WE'VE MOVED to:  www.bbiw.net

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Sun, 6 Mar 2005 14:08:35 -0500
Cc: [email-masked]
Sender: [email-masked]

On Sat March 5 2005 09:54, [email-masked] wrote:

> But i disagree with:
> 
> > Section 2.1 refers to the MHS as monolithic; it is not --
> > it is composed of autonomous agents (MTAs, MSAs, MDAs,
> > gateways, list expanders) and a user may receive a delivery
> > notification of some sort from any of those agents.
> 
> For users the (image of the) MHS is monolithic.
> In This section I would keep it that way.

I specifically mentioned delivery notification because a
sender may receive such a notification from any of the
distinct message-handling components, and the significance
of such notifications (and the user's reaction) depends
on the component.

> I like the concept of MDA and want to distinguish it more clearly from the MTA.
> I would suggest the MTA only works with domain part of the message, the MDA with domain parts and local parts.
> So the MTA does the transfering, the MDA does all the rest.
> If an operation needs the localpart of the address it is a MDA function.
> (auto responders, mailinglist actions, the SMTP VRFY, EXPN, ETRN commands ect.)
> 
> Architectonal ideas
> Eexcept with LMDP, an MDA is always hiding behind an MTA.
> LMDP uses SMTP-Direct (sending to a fixed ip address, not to a public MX record)

LMTP?

> There is no direct contact between MTA outside the administrative domain and the MDA in that administrative domain.

Architecturally (or topologically), an MDA can only appear
on the recipient side of all MX hosts for the recipient's
domain.  There may, however, be MTAs in that area (some
administrative entities configure their systems with a
border MTA for aliasing, etc.).
 
> Within one administrative domain globaly valid MX records are NOT used.
> MX records are used for transfer between seperated administrative domains.
> within an Administrative domain SMPT-direct. (sending to only one MTA) is used.

Except for list expansion and other forwarding operations
that extend beyond the (DNS) domain.

> What is your standpoint about Users in general?

Users of email (authors, senders, moderators, and recipients)
are part of the end-to-end communications path.  Message
body and header originator/origination-date/destination/
information/resent fields (RFC 2822) [and some extension
fields] are exclusively for use of the users (and as assisted
by their UAs) and should never be altered by the MHS.

Date: Sat, 05 Mar 2005 15:50:41 -0700
To: [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

At 10:45 PM 3/5/2005 +0100, Frank Ellerman wrote:

>David MacQuigg wrote:
>
> > Domain authentication should done by the incoming MTA
>
>Looking for "MX" in email-arch-03, only one hit, sigh. :-(

We should probably add "Spammer" as a fundamental component of the 
architecture.  I guess that depends on what you believe about the ultimate 
effectiveness of the new authentication schemes.  :>)

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Sat, 05 Mar 2005 22:45:12 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 6
Sender: [email-masked]

David MacQuigg wrote:

> Domain authentication should done by the incoming MTA

Looking for "MX" in email-arch-03, only one hit, sigh. :-(

Date: Sat, 05 Mar 2005 12:38:57 -0700
To: [email-masked], Bruce Lilly <[email-masked]> [email-masked]
From: David MacQuigg <[email-masked]>
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Sender: [email-masked]

At 02:54 PM 3/5/2005 +0000, you wrote:

>I like the concept of MDA and want to distinguish it more clearly from the 
>MTA.
>I would suggest the MTA only works with domain part of the message, the 
>MDA with domain parts and local parts.
>So the MTA does the transfering, the MDA does all the rest.
>If an operation needs the localpart of the address it is a MDA function.
>(auto responders, mailinglist actions, the SMTP VRFY, EXPN, ETRN commands 
>ect.)

I agree, and I think a clear distinction between MTA and MDA is also useful 
in discussions of spam.  Filtering, flagging, and blocking should be done 
by the MDA, which is aware of all its recipients and their 
preferences.  Domain authentication should done by the incoming MTA, which 
can enforce a site-wide policy such as "any sender that doesn't 
authenticate, abort the session, don't download anything".  Its a clean 
interface, with the possible exception that some recipients might want an 
individual policy, like "send me everything, even if it doesn't authenticate".

The concept of administrative domains is also useful in discussing when 
authentication should occur (at the incoming MTA for each domain).  A clear 
standard here will allow a recipient to quickly sort through piles of 
headers, skipping past multiple  Received lines from within his own domain 
and any domains of trusted forwarders, and getting straight to the domain 
that needs to be blocked or filtered.

-- Dave

*************************************************************     *
* David MacQuigg, PhD              * email:  dmq'at'gci-net.com   *  *
* IC Design Engineer               * phone:  USA 520-721-4583  *  *  *
* Analog Design Methodologies                                  *  *  *
*                                  * 9320 East Mikelyn Lane     * * *
* VRS Consulting, P.C.             * Tucson, Arizona 85710        *
*************************************************************     *

To: Bruce Lilly <[email-masked]> [email-masked]
From: [email-masked]
Subject: Re: Re: draft-crocker-email-arch-03 Users, MDA and administrative domains
Date: Sat, 5 Mar 2005 14:54:51 -0000
Sender: [email-masked]

I mostly agree with your comments.
But i disagree with:

> Section 2.1 refers to the MHS as monolithic; it is not --
> it is composed of autonomous agents (MTAs, MSAs, MDAs,
> gateways, list expanders) and a user may receive a delivery
> notification of some sort from any of those agents.

For users the (image of the) MHS is monolithic.
In This section I would keep it that way.

Your respons. made me also think about the MTA - MDA  distinction.

IN RFC2821 there is no MDA.
I think the MDA is a new entity first mentioned in this draft.

The closest RFC reference I could find are the
Local delivery agent and Local gateway agent in RFC 2033 - Local Mail Transfer Protocol.

I like the concept of MDA and want to distinguish it more clearly from the MTA.
I would suggest the MTA only works with domain part of the message, the MDA with domain parts and local parts.
So the MTA does the transfering, the MDA does all the rest.
If an operation needs the localpart of the address it is a MDA function.
(auto responders, mailinglist actions, the SMTP VRFY, EXPN, ETRN commands ect.)

Architectonal ideas
Eexcept with LMDP, an MDA is always hiding behind an MTA.
LMDP uses SMTP-Direct (sending to a fixed ip address, not to a public MX record)
There is no direct contact between MTA outside the administrative domain and the MDA in that administrative domain.

There should be a chapter about the RFC2821 MTA
how in this draft it has architectionaly been split up in

MSA  for submission 
MTA  for transfer
MDA  for message "delivery"
and the 
MS   for storing

Another point the administrative domain.
Administrative domains can be nested / overlapping ed.

Within one administrative domain globaly valid MX records are NOT used.
MX records are used for transfer between seperated administrative domains.
within an Administrative domain SMPT-direct. (sending to only one MTA) is used.

What is your standpoint about Users in general?

Date: Fri, 04 Mar 2005 18:57:46 -0500
From: John C Klensin <[email-masked]>
To: Vince Sabio <[email-masked]>, ietf-smtp <[email-masked]>
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

--On Wednesday, 02 March, 2005 13:03 -0500 Vince Sabio
<[email-masked]> wrote:

> However, it has recently been pointed out that the following
> graf in RFC2821 may permit sites to _not_ be required to fall
> back to HELO -- and issue a 550 response to HELO instead -- if
> local policy dictates:
>
>     [QUOTE RFC2821 SECT 7.7]
> 
>     In recent years, use of the relay function through
> arbitrary sites     has been used as part of hostile efforts
> to hide the actual origins     of mail.  Some sites have
> decided to limit the use of the relay     function to known or
> identifiable sources, and implementations SHOULD     provide
> the capability to perform this type of filtering.  When mail
> is rejected for these or other policy reasons, a 550 code
> SHOULD be used in response to EHLO, MAIL, or RCPT as
> appropriate.
> 
>     [END QUOTE]
> 
> So, the question: Is this a valid interpretation of Sect. 7.7
> -- i.e., may an MTA provide a 550 response to HELO where it
> would otherwise have given a 250 response to an EHLO if the
> site policy for that MTA forbids HELO?

See Keith Moore's note, with which I agree.  You can, by
invoking the "reject anything for any reason" principle, not via
the text above, which was intended to appeal to the general
principle that HELO is the fallback and servers need to accept
it without identifying that requirement every single time.  But
it would pretty much be silly: while your favorite spammer may
use HELO, rejecting messages containing it will trash some
legitimate messages.  More important, if you are the only one
doing it, it might help stop some spam.  If you and a few
thousand of your friends do it, the spammers will figure out how
to spell "EHLO" and toss out the multiline replies.

Really, applying anti-spam solutions that the spammers can
trivially and obviously work around is not a good investment.
Over time, arguably contributes to an increase in the total
volume of spam without accomplishing much else.

     john

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: draft-crocker-email-arch-03
Date: Fri, 04 Mar 2005 10:21:21 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 18
Cc: [email-masked]
Sender: [email-masked]

Bruce Lilly wrote in <ietf-smtp.imc.org>:

> Dave Crocker wrote:
>> Well, this is interesting.  I believe the intent of rfc2822
>> is to characterize the id-right field as either a domain
>> name or an ip address.  But it appears that it does not say
>> that.

> It is a domain name or domain literal. RFC 2822 introduces
> different grammar productions form those used in angle-addr
> to enforce syntax requirements (no line folding in
> identifiers) but does not change the semantics (RFC 2822
> clearly so indicates).

Quote forwarded to the UseFor mailing list for personal reasons.

                       Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked], Dave Crocker <[email-masked]>
Subject: Re: draft-crocker-email-arch-03
Date: Thu, 3 Mar 2005 14:54:57 -0500
Sender: [email-masked]

Some comments:

Description of Gateways and list expanders as users is not
enlightening; it is confusing.  They do not interact with
(human or automaton) users except circumstantially; they
are part of the transport process and act like any other
store-and-forward transport component other than the
expander- or gateway-specific operations.

Section 2.1 refers to the MHS as monolithic; it is not --
it is composed of autonomous agents (MTAs, MSAs, MDAs,
gateways, list expanders) and a user may receive a delivery
notification of some sort from any of those agents.

Section 2.1.1 conflates author with sender; they are
distinct concepts except where they happen to refer to the
same entity by coincidence.  The author is responsible for
creating content; the sender is responsible for initiating
transport.  RFC 2822 has some clear text delineating the
differences.

In Section 2.1.2, "responds" and "responses" would be
preferable to "replies" and "reply"; the latter terms
imply a query, which is applicable to MDNs, but not to
general responses.

Section 2.3 illustrates the confusion caused by conflating
author and sender.  Also, referring to a mailing list as
a Mediator may apply to a moderated mailing list, but a
simple list expander is simply another store-and-forward
MTA with the side-effect of expanding a list alias into
(typically multiple) recipients.  Stating that a (list
expander) Mediator originates messages is confusing; it
does not originate messages -- the author does -- the
list expander merely distributes messages.  By analogy,
the claim that list expanders originate messages is like
a claim that Barnes&Noble writes textbooks.

Section 2.2. states that "it is legal to have only one
[relay]"; in fact it is legal to have none, with the
oMUA connecting to the recipient's MDA as indicated by
an MX record. [or, as noted below, making local delivery]

Section 2.2.2 says that notices are sent to an "address"
specified by the "Source" (described as for an MSA); in
fact Message Disposition Notices are sent to a list of
mailboxes (not an address) specified by the originator.
Delivery status notifications are sent to the return path,
which might be unrelated to the author or (2822) Sender
field (the return bath is elsewhere in the draft called
a "Notices handling 'address'").

Section 2.2.4 refers to gateways as user processes; they
are part of the normal store-and-forward transport
mechanism.

Section 2.3. contains the typo "this distinctions".

Section 3 text contains "mailbox address <addr-spec>"
which conflates 4 different things:
1. an address is a named group or a mailbox
2. a mailbox is an addr-spec, or an addr-spec enclosed
   in angle brackets (known as an angle-addr) with an
   optional display name
3. angle-addr, described above
4. addr-spec, which itself includes no brackets
See RFC 2822 for details.

Section 3.1 claims that domain names are equivalent to
the "Administrative Domains" defined in the document;
that is not necessarily so. A single administrative
entity may be responsible for multiple DNS domains, or
a single DNS domain may have portions (subdomains)
administered by multiple entities.  The "cookie"
analogy is weak and unenlightening; it is confusing
because the history of cookie use in HTTP is that
cookies have been read and interpreted by entities
other than those setting the cookie.

Additional markers referring to RFCs 1034 and 1035
would be appropriate in section 3.2.

The last paragraph of section 4.4 should probably start
with "An identifier", given the reference to multiple
identifiers (examples of which are Received field "id"
components [though RFCs 821 and 1123 botched that]).

The figure in section 4 is botched by a page break.
Gateways, list expanders, and aliasing really belong
in section 4; they fit between the oMUA and rMUA in
the transport stream.

Section 4.1 refers to "a tree" of message components;
technically the structure may be a graph (due to CID
references (RFC 2557) etc.) rather than a tree (graphs
are more general than trees).  The reference to RFC
2298 in section 4.1 is obsolete; all references should
be checked (the RFC Editor provides the file "rfc-ref.txt"
which can be used to obtain correct references as it
indicates which RFCs have been obsoleted by others).

Section 4.2 states than the oMUA always connects to an
MSA, which is incorrect. It may connect to any of several
types of MTA, or to the recipient's MDA.  For that matter,
it might directly make local delivery for local
recipients.  The section states that "[a] Mediator is
[a] special class of MUA" which is simply untrue in the
case of a list expander; there is no UA or UI or user
involved.  The discussion of the Reply-To field is an
oversimplification; the destination of responses has as
much to do with the desires of the respondent as it does
with the suggestion made by the author (which is what
the Reply-To field contains).  The discussion of the
Sender field states that in absence of that field, the
message From field takes its place, and equates 2821
MAIL FROM command arguments to the 2822 header field
Sender.  That is wrong on two counts:
1. The Sender field is independent of the SMTP command
   arguments, except by coincidence of implementation
   (and the draft correctly notes that architecture
   should be described independently of any implementation
   quirks).
2. The Sender field is never used for automatic responses,
   nor automatically presented as a response destination
   by an rMUA (unlike the From field and the SMTP return
   path).
Description of To and Cc fields as recipient addresses
(in this case the fields do in fact specify address lists)
but that is no more true of email than it is of names
listed on the pages of paper mail, concealed inside the
envelope; typically there is some correspondence between
the "recipients" (and author(s)) listed in the header and
those actually used by transport, but architecturally
there is no such requirement nor relationship.

The description of MSAs in section 4.3 should change
"operates" to "may operate" (RFC 2476 permits operation
on port 25).  The discussion of Received fields added
by MSAs should change "may" to MUST (an MSA is an [E]SMTP
receiver, and RFC 2821 sect. 4.4 REQUIRES insertion of
trace fields).

Draft section 4.4 states that MTAs do not make changes
to "addresses" in the "envelope", such changes do happen
in the SMTP envelope, recording/removing path (old
implementations), resolving aliases, expanding lists,
and/or as part of gateway operations.  As noted above,
SMTP MTAS MUST (not "may") insert trace fields.

Section 5, speaking of Mediators (and having categorized
list expanders as Mediators), refers to "other MUAs".
List expanders are not MUAs!

Aliasing as described in section 5.1 is claimed to be
"strictly a Recipient user function".  It is sometimes
an administrative function, mere users not being
permitted to fiddle with aliases.  The draft continues,
claiming that aliasing "resubmits the message, replacing
the 'envelope' 'address'"; aliasing may involve translation
to a local mailbox, in which case there is no resubmission,
no SMTP envelope, and may be no rewriting of "envelope"
information stuffed into the message header.

Resending (section 5.2) omits mention of the RFC 2822
Resent-Date field, which is REQUIRED if any other Resent-
field is used!  Nor does it mention Resent-Message-ID,
which is another example of an additional identifier.

Section 5.3 mentions something called a "ReDirector" which
is undefined and mentioned nowhere else in the draft. The
same section incorrectly describes the Reply-To field as
"Originator or Mediator"; it is defined (in RFC 2822) as
an Originator field.  No transport entity (list expanders
or otherwise) should alter that field; significant problems
have been caused by violation of that transparency rule.
Conversely, "RFC2821.MailFrom" should always be set to
the mailbox of the list administrator by a list expander.

In section 7.2, the customary term is "Informative"
references.  It is unclear why the references are formatted
differently in this section from 7.1 (peculiar line and
page breaks).

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dave Crocker <[email-masked]>
Subject: Re: draft-crocker-email-arch-03
Date: Thu, 3 Mar 2005 12:12:34 -0500
Cc: [email-masked]
Sender: [email-masked]

On Sat February 19 2005 21:33, Dave Crocker wrote:
> 
> On Sat, 19 Feb 2005 15:32:01 -0000, Willemien wrote:

> >  Some motivation for this:
> >
> >  - (practical) They supply the MIME encapsulation of the message.
> 
> MIME is not relevant to the transfer service.

Huh? Content-Transfer-Encoding is not relevant to transfer?!?

> >  3.3 Message Identifiers
> >  Not conform RFC 2822 section 3.6.4
> >  (the right side is not always a domain)
> 
> Well, this is interesting.  I believe the intent of rfc2822 is to characterize the id-right field as either a domain name or an ip address.  But it appears that it does not say that.

It is a domain name or domain literal. RFC 2822 introduces different
grammar productions form those used in angle-addr to enforce syntax
requirements (no line folding in identifiers) but does not change the
semantics (RFC 2822 clearly so indicates).

Cc: ietf-smtp <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 2 Mar 2005 23:34:42 -0500
To: Vince Sabio <[email-masked]>
Sender: [email-masked]

>>    It is a well-established principle that an SMTP server may refuse 
>> to
>>    accept mail for any operational or technical reason that makes 
>> sense
>>    to the site providing the server.  However, cooperation among sites
>>    and installations makes the Internet possible.  If sites take
>>    excessive advantage of the right to reject traffic, the ubiquity of
>>    email availability (one of the strengths of the Internet) will be
>>    threatened; considerable care should be taken and balance 
>> maintained
>>    if a site decides to be selective about the traffic it will accept
>>    and process.
>
> I don't mean to be pedantic, but it is arguably still not clear. Based 
> on that graf, a site could accept EHLO under "proper circumstances" 
> (i.e., well-formed EHLO from a non-blocked site, etc.), yet still 
> reject all use of HELO under any circumstances for "operational or 
> technical reasons that make sense to the site providing the server," 
> where, at that site, "care has been taken and balance maintained [in 
> the decision] to be selective about the traffic it will accept and 
> process."

the reason for not accepting HELO has to "be an operational or 
technical reason that makes sense".

consider three cases:

1. client uses HELO, and because it sends HELO it cannot supply any 
additional information to the server via SMTP extensions
2. client uses EHLO, but doesn't supply any additional information to 
the server via SMTP extensions
3. client uses EHLO but does supply additional information to the 
server.  for example, it might use SMTP AUTH to authenticate itself to 
the server, or it might use a future extension to supply 
originator-supplied credentials to the server.

it makes no operational or technical sense for the server to 
distinguish case 1 from case 2.  in both cases the server has the same 
information _about the message_ available to inform its decision.  the 
single bit of information about whether the client used HELO vs. EHLO 
is meaningless to the server in deciding whether the message is valid 
or whether it merits handling by the server.  (there are occasionally 
good operational reasons for a client to disable EHLO - for instance, 
some firewalls have been known to break EHLO by allowing the server to 
advertise SMTP extensions whose parameters were not acceptable to the 
firewall)

on the other hand, the server might quite reasonably use the additional 
information available in case 3 to change its handling of the message.  
but the additional information isn't in the EHLO vs. HELO bit, it's in 
the additional parameters supplied by the client.

or to put it another way - sure the server can do anything it wants, 
including bouncing random messages, bouncing all messages, delivering 
all mail to a bit bucket, whatever.  whether the standards say you can 
or cannot do it is irrelevant.  the question is whether it helps your 
mail work better. distinguishing between EHLO and HELO is likely to 
harm your mail service rather than help it in any way.  specifically 
it's not a good metric for helping to decide whether traffic is valid.

Keith

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, "Vince Sabio" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Subject: Re: Site policy vs. HELO
Date: Wed, 2 Mar 2005 18:51:17 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Ed,

Unless I am missing something in the question, all clients need to be ready
for a 50x response to a EHLO request.

500 means the command is not understood by the host server
501 or 502 means it is understood but not implemented for what ever reason.

The client should fall back on a 500 and the server MUST be ready for a
HELO.  However, in practice, it is probably best to fallback on a 50x as it
has been shown that 501 or 502 can happen as well with an acceptable fall
back.  This goes against the 2.2.1 section recommendation for EHLO support,
but it is what it is out there. So I recommend your client to be ready for a
50x fallback.

55x on the other hand means that the client SHOULD NOT try again regardless
of the client request, including HELO.  That's it. No questions about it.
55x means don't try again.

In my opinion, section 7.7 inherently implies HELO when it indicated EHLO.
The reason I say that is because the section presumes EHLO ready clients,
but the site policy may also apply to HELO only clients just as well.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

----- Original Message -----
From: <[email-masked]>
To: "Vince Sabio" <[email-masked]>
Cc: "ietf-smtp" <[email-masked]>
Sent: Wednesday, March 02, 2005 2:29 PM
Subject: Re: Site policy vs. HELO

>
> The question posed is: Can a site whose policy is not to do EHLO be in
> compliance with RFC2821?  Or does Sect 7.7 provide an exception to 2.2.1?
>
> <PseudoLegalese to try to be precise>
> Better to ask, How can a site be in compliance with both sections?  Then,
> according to 2.2.1, they MUST support HELO and according to 7.7 they may
> decide to "limit the use of the relay function to known or identifiable
> sources".  Elsewhere, Sect. 3.7, RFC 2821, states, "The relay server may
> accept or reject the task of relaying the mail in the same way it accepts
> or rejects mail for a local user."  That suggests that a server treat a
> HELO that same way it treats an EHLO, applying its policies using the
> (obviously limited) information the HELO provides.
>
> The paragraph in Sect 7.7 that precedes the one quoted makes that clear:
>
>     It is a well-established principle that an SMTP server may refuse to
>     accept mail for any operational or technical reason that makes sense
>     to the site providing the server.  However, cooperation among sites
>     and installations makes the Internet possible.  If sites take
>     excessive advantage of the right to reject traffic, the ubiquity of
>     email availability (one of the strengths of the Internet) will be
>     threatened; considerable care should be taken and balance maintained
>     if a site decides to be selective about the traffic it will accept
>     and process.
>
> Best.../Ed
>
> At 01:03 PM 3/2/05 -0500, Vince Sabio wrote:
> >The following graf in RFC2821 appears to require an MTA (not an
> >RFC2476-type MSA) to fall back to HELO for compatibility with older SMTP
> >clients and servers):
> >
> >    [QUOTE RFC2821 SECT 2.2.1]
> >
> >    Contemporary SMTP implementations MUST support the basic extension
> >    mechanisms.  For instance, servers MUST support the EHLO command even
> >    if they do not implement any specific extensions and clients SHOULD
> >    preferentially utilize EHLO rather than HELO.  (However, for
> >    compatibility with older conforming implementations, SMTP clients and
> >    servers MUST support the original HELO mechanisms as a fallback.)
> >
> >    [END QUOTE]
> >
> >However, it has recently been pointed out that the following graf in
> >RFC2821 may permit sites to _not_ be required to fall back to HELO -- and
> >issue a 550 response to HELO instead -- if local policy dictates:
> >
> >    [QUOTE RFC2821 SECT 7.7]
> >
> >    In recent years, use of the relay function through arbitrary sites
> >    has been used as part of hostile efforts to hide the actual origins
> >    of mail.  Some sites have decided to limit the use of the relay
> >    function to known or identifiable sources, and implementations SHOULD
> >    provide the capability to perform this type of filtering.  When mail
> >    is rejected for these or other policy reasons, a 550 code SHOULD be
> >    used in response to EHLO, MAIL, or RCPT as appropriate.
> >
> >    [END QUOTE]
> >
> >So, the question: Is this a valid interpretation of Sect. 7.7 -- i.e.,
may
> >an MTA provide a 550 response to HELO where it would otherwise have given
> >a 250 response to an EHLO if the site policy for that MTA forbids HELO?
>
> Die hard techie
> Tel: 732.494.1606 Fax: 732.494.4495  Cell: 908.930.9409
> email:[email-masked]
>
>
>

Date: Wed, 2 Mar 2005 14:41:02 -0500
To: ietf-smtp <[email-masked]>
From: Vince Sabio <[email-masked]>
Subject: Re: Site policy vs. HELO
Sender: [email-masked]

** Sometime around 14:29 -0500 03/02/2005, 
[email-masked] said:

>The question posed is: Can a site whose policy is not to do EHLO be 
>in compliance with RFC2821?  Or does Sect 7.7 provide an exception 
>to 2.2.1?

Yes, except "% sed 's/EHLO/HELO/'" in your statement above (which is 
probably what you had intended to write).

><PseudoLegalese to try to be precise>
>Better to ask, How can a site be in compliance with both sections? 
>Then, according to 2.2.1, they MUST support HELO and according to 
>7.7 they may decide to "limit the use of the relay function to known 
>or identifiable sources".  Elsewhere, Sect. 3.7, RFC 2821, states, 
>"The relay server may accept or reject the task of relaying the mail 
>in the same way it accepts or rejects mail for a local user."  That 
>suggests that a server treat a HELO that same way it treats an EHLO, 
>applying its policies using the (obviously limited) information the 
>HELO provides.
>
>The paragraph in Sect 7.7 that precedes the one quoted makes that clear:
>
>    It is a well-established principle that an SMTP server may refuse to
>    accept mail for any operational or technical reason that makes sense
>    to the site providing the server.  However, cooperation among sites
>    and installations makes the Internet possible.  If sites take
>    excessive advantage of the right to reject traffic, the ubiquity of
>    email availability (one of the strengths of the Internet) will be
>    threatened; considerable care should be taken and balance maintained
>    if a site decides to be selective about the traffic it will accept
>    and process.

I don't mean to be pedantic, but it is arguably still not clear. 
Based on that graf, a site could accept EHLO under "proper 
circumstances" (i.e., well-formed EHLO from a non-blocked site, 
etc.), yet still reject all use of HELO under any circumstances for 
"operational or technical reasons that make sense to the site 
providing the server," where, at that site, "care has been taken and 
balance maintained [in the decision] to be selective about the 
traffic it will accept and process."

Thus, a site may arguably _not_ treat a HELO the same way it would 
treat an EHLO.

__________________________________________________________________________
Vince Sabio                                                  [email-masked]

From: [email-masked]
Date: Wed, 02 Mar 2005 14:29:22 -0500
To: Vince Sabio <[email-masked]>
Subject: Re: Site policy vs. HELO
Cc: ietf-smtp <[email-masked]>
Sender: [email-masked]

The question posed is: Can a site whose policy is not to do EHLO be in 
compliance with RFC2821?  Or does Sect 7.7 provide an exception to 2.2.1?

<PseudoLegalese to try to be precise>
Better to ask, How can a site be in compliance with both sections?  Then, 
according to 2.2.1, they MUST support HELO and according to 7.7 they may 
decide to "limit the use of the relay function to known or identifiable 
sources".  Elsewhere, Sect. 3.7, RFC 2821, states, "The relay server may 
accept or reject the task of relaying the mail in the same way it accepts 
or rejects mail for a local user."  That suggests that a server treat a 
HELO that same way it treats an EHLO, applying its policies using the 
(obviously limited) information the HELO provides.

The paragraph in Sect 7.7 that precedes the one quoted makes that clear:

    It is a well-established principle that an SMTP server may refuse to
    accept mail for any operational or technical reason that makes sense
    to the site providing the server.  However, cooperation among sites
    and installations makes the Internet possible.  If sites take
    excessive advantage of the right to reject traffic, the ubiquity of
    email availability (one of the strengths of the Internet) will be
    threatened; considerable care should be taken and balance maintained
    if a site decides to be selective about the traffic it will accept
    and process.

Best.../Ed

At 01:03 PM 3/2/05 -0500, Vince Sabio wrote:
>The following graf in RFC2821 appears to require an MTA (not an 
>RFC2476-type MSA) to fall back to HELO for compatibility with older SMTP 
>clients and servers):
>
>    [QUOTE RFC2821 SECT 2.2.1]
>
>    Contemporary SMTP implementations MUST support the basic extension
>    mechanisms.  For instance, servers MUST support the EHLO command even
>    if they do not implement any specific extensions and clients SHOULD
>    preferentially utilize EHLO rather than HELO.  (However, for
>    compatibility with older conforming implementations, SMTP clients and
>    servers MUST support the original HELO mechanisms as a fallback.)
>
>    [END QUOTE]
>
>However, it has recently been pointed out that the following graf in 
>RFC2821 may permit sites to _not_ be required to fall back to HELO -- and 
>issue a 550 response to HELO instead -- if local policy dictates:
>
>    [QUOTE RFC2821 SECT 7.7]
>
>    In recent years, use of the relay function through arbitrary sites
>    has been used as part of hostile efforts to hide the actual origins
>    of mail.  Some sites have decided to limit the use of the relay
>    function to known or identifiable sources, and implementations SHOULD
>    provide the capability to perform this type of filtering.  When mail
>    is rejected for these or other policy reasons, a 550 code SHOULD be
>    used in response to EHLO, MAIL, or RCPT as appropriate.
>
>    [END QUOTE]
>
>So, the question: Is this a valid interpretation of Sect. 7.7 -- i.e., may 
>an MTA provide a 550 response to HELO where it would otherwise have given 
>a 250 response to an EHLO if the site policy for that MTA forbids HELO?

Die hard techie
Tel: 732.494.1606 Fax: 732.494.4495  Cell: 908.930.9409
email:[email-masked]

Date: Wed, 2 Mar 2005 13:03:27 -0500
To: ietf-smtp <[email-masked]>
From: Vince Sabio <[email-masked]>
Subject: Site policy vs. HELO
Sender: [email-masked]

[I had originally posted this question to the ietf-822 list, where it 
was pointed out that I *should* have posted it here. I apologize if 
this appears to be a cross-post; it's actually a re-post to the 
proper list.]

I'm bringing this question over here from the SPAM-L list....

The following graf in RFC2821 appears to require an MTA (not an 
RFC2476-type MSA) to fall back to HELO for compatibility with older 
SMTP clients and servers):

    [QUOTE RFC2821 SECT 2.2.1]

    Contemporary SMTP implementations MUST support the basic extension
    mechanisms.  For instance, servers MUST support the EHLO command even
    if they do not implement any specific extensions and clients SHOULD
    preferentially utilize EHLO rather than HELO.  (However, for
    compatibility with older conforming implementations, SMTP clients and
    servers MUST support the original HELO mechanisms as a fallback.)

    [END QUOTE]

However, it has recently been pointed out that the following graf in 
RFC2821 may permit sites to _not_ be required to fall back to HELO -- 
and issue a 550 response to HELO instead -- if local policy dictates:

    [QUOTE RFC2821 SECT 7.7]

    In recent years, use of the relay function through arbitrary sites
    has been used as part of hostile efforts to hide the actual origins
    of mail.  Some sites have decided to limit the use of the relay
    function to known or identifiable sources, and implementations SHOULD
    provide the capability to perform this type of filtering.  When mail
    is rejected for these or other policy reasons, a 550 code SHOULD be
    used in response to EHLO, MAIL, or RCPT as appropriate.

    [END QUOTE]

So, the question: Is this a valid interpretation of Sect. 7.7 -- 
i.e., may an MTA provide a 550 response to HELO where it would 
otherwise have given a 250 response to an EHLO if the site policy for 
that MTA forbids HELO?

__________________________________________________________________________
Vince Sabio                                                  [email-masked]