Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID Verification)

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 05:01:04 +0000

On Nov 30, 11:11pm, Eric S. Raymond wrote:
}
} Thus, RFC2822 does not mandate that the address be valid.  We are 
} referred to RFC 2821.  Section 4.1.1.4 reads:
} 
}    When the delivery SMTP server makes the "final delivery" of a
}    message, it inserts a return-path line at the beginning of the mail
}    data.  This use of return-path is required; mail systems MUST support
}    it.  The return-path line preserves the information in the <reverse-
}    path> from the MAIL command.
} 
} I read this to mean that the Return-Path must, if present, be a
} valid mailbox for a responsible person (note the MUST).

I read this to mean that the server is reqired to insert the return-path
field, regardless of its knowledge of the validity of the mailbox.

Suppose I send a message to you.  For whatever reason, it's queued at
my company SMTP server rather than delivered immediately to your MX.
While the message is in the queue, I have an argument with my CEO over
the interpretation of RFC2821, and he fires me.  The sysadmin immediately
disables my email account.

You're saying that its now an _SMTP protocol_ violation for the company
server to complete the process of delivering that queued message to you?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 00:09:09 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "ASRG" <[email-masked]>
>
> Whether the RFCs require a valid return address, or not, in spirit or in
> letter of the law, is something Dave Crocker and Eric Raymond, and
> others, who worked on the original 821 and 2821 RFCs can tell us. But
> the fact today is that no one is expected to provide a valid address,
> and any system relying on this, will fail in some cases unless the
> existing RFCs are changed

Yakov,

The scattered RFC concepts related to the Return Path (RP) are very clear
about this.  In practice, all commercial vendors and legitimate mail sites
expect it to be valid because a high percentage of the mail
client/server/gateway software design is 100% purely based on its validity.

That has NOTHING to do with the fact that it could be invalid.  Yet, the
software is designed to handle this possibility as well.  In this case,  the
mail is dropped, logged, or whatever.    The design assumption is that it is
valid with a no brainer consideration that it it quite possible the
destination is not reachable.

This is just like snail mail.  An invalid return address on the paper
envelope stamped with "Return To sender" with a silly finger pointing to the
return address, will fail.  It happens.  But that does not mean, since it is
possible to be invalid,  it is therefore not required to be valid.  No,  it
is naturally assumed to be valid.

From a mail transport system standpoint, this is vitally important.

With delay validation or mail processing such as as gateway, i.e,  UUCP/SLIP
operations, etc,  in order for to properly support gateway and UUCP
operations,  the SMTP receiver must conform with the RFC by added the
"Return-Path:" line to the TOP of the DATA received for final destination
targets.

      RFC 1123 - Host Requirements

      5.2.8  DATA Command: RFC-821 Section 4.1.1
      ..
         When the receiver-SMTP makes "final delivery" of a message,
         then it MUST pass the MAIL FROM: address from the SMTP envelope
         with the message, for use if an error notification message must
         be sent later (see Section 5.3.3).  There is an analogous
         requirement when gatewaying from the Internet into a different
         mail environment; see Section 5.3.7.

If you don't do this,  your can break gateway operations, including  email
list servers, especially when there is a route involved.

Now the compliant GATEWAY has the support the "Return-Path:" header line in
order to provide a proper bounce path.   The logic in our gateway software
using this priority order to get the return path:

           RETURN PATH:
           SENDER:
           [ERROR-TO:]        (Mostly supported by ListServ, Wildcat)
           Reply-To:
           FROM:

This follows RFC 822:

           4.4.4.  AUTOMATIC USE OF FROM / SENDER / REPLY-TO

However, the legacy RFC 822 does not mention Return-Path.  This is covered
by RFC 1123, 5.2.8.

Please note:  that obtaining the Sender, From, etc, means that a MTA must
reach the DATA stage which again, in my view, is something you wish to AVOID
to address illegal access as a preliminary access check.

Again, the point is, it is EXPECTED to be there, it is expected to be valid.

Unreachable Return Path is a difference subject.  However, it is the subject
that the RFC needs to address in what is to be expected about a unreachable
return path.

In practice, it is handled as a BOUNCE concept and if a system fails to
deliver the BOUNCE, it is automatically blacklisted (at least in our
software it is).

Since a BOUNCE is a natural requirement to a mail server, therefore the
return path is expected to be valid, with logic to handle the failures.

> We understand that changes must be made. However, we need to justify
> these changes before imposing them on the entire Internet. For example,
> it is significantly more lightweight to verify domain/IP association via
> LMAP than do an RCPT TO callback.  Both your proposal and LMAP address
> the same problem - forgery of the MAIL FROM address, except LMAP focuses
> on verifying the domain, while you are verifying the actual address.
> What we need to determine, is why should we go through the burden of
> verifying the actual address, when for the purposes of reducing forgery,
> verifying the domain is sufficient?

OK. thanks for clarification.

But as I indicated above,  current commercial implementations of mail
systems (clients, servers, gateways) in the market place has made it a
pseudo-standard and requirement that a valid return path is provided.
Otherwise, we can't work together.

Unreachable return paths are part of the design, handled differently with
each system.  I can speak for our system that we will retry a system using
our default values of 3 days, once per hour (72 attempts).   I can also tell
you that we have logic that puts a weight on the type of return path
failure. For example:

        Default Retry      =  72
        Failed MX          = subtract 10
        Failed IP            = subtract 5
        Failed Server    = subtract 2

with the idea to accelerate the bounce (when retry <= 0) depending on where
it failed.

So if a MX is bad, using the above numbers:

        hour 1          retry count 72
        hour 2          retry count 62
        hour 3          retry count 52

etc, after 8 hours, the bounce will occur.

However, in practice, this was too tight, so it was relaxed (weight are
zero). It will try 72 times.  The exception to the rule is if a server
receiving the mail responds with a code of 55x, which means don't try again.

I am only telling you this because what you need to do look at is what
"Unreachable" means and put it in writing so that developers will have a
guideline to put constraints on a unrearchable return path.  Look at he
CAN-SPAM bill.  It does require a valid return path. So this should be
enough to put it in the specs.

The specs can say something that reflects the the kind of possible failures
when an attempt is made to connect to a MTA and deliver mail and that is
outlined by the SMTP state machine.

Possible failure points:

        connect
        HELO/EHLO
        MAIL FROM:
        RCPT TO:
        DATA
        QUIT

when you lock into the specs that a DOMAIN and a ADDRESS will be expected to
be valid, then you inherently and immediately begin to address a high
percentage of the spammer problem.  It will solve it 100%, but it will make
the spammers, well, honest spammers and thats a MAJOR benefit in the efforts
to control and regulate them (which I believe is NOT part of the functional
design for SMTP).  Thats a installation desire or an end-user +desire.  The
VENDORs will now be able to offer the features the customers and end users
want.

> Let's say you have verified either the domain or the address, and the
> message in question turns out to be spam. In both cases, you are going
> to complain to the ISP of the domain, not the actual user! So why go
> through the trouble of verifying the actual email address, when a domain
> is sufficient?

Like I said, above, that is none of your business.  You are complicating
what is otherwise a technical problem with a very easy technical solution.
You just need to make it possible.

By attempting to get into "interpretation" of the data, then you are going
into a fuzzy logic and that's not the purpose of SMTP.

In my technical opinion,  you have an fantastic opportunity to address to a
very large degree the anonymous abuse of the servers by simply address the
dearth of controls in the specifications.   Forget about spam. It could
something else together.  Maybe you don't want EXE files or VBS files send
because the customer is placing scrutiny on this type of data.  In all
cases, you need to control and authenticate access that is independent of
the mail content.

With that said, that does not mean the SMTP server should not offer some
level of control here for "data interpretation."     Like I said before, I
see only some sort of CRC32 checker maybe to be added, but that's it.

Compliant clients can say:

            C:  DATA CRC=1BCD233
            S:  354 Begin sending data

or with a fallback:

            C:  DATA CRC=1BCD233
            S:  500 Command not understood
            C:  DATA
            S:  354 Begin sending data

The compliant server can not use the CRC32 to make sure the data integrity
is ok.   We can extend this to address main content controls.

            C:  DATA CRC=1BCD233 HAS=ADV,ATTS
            S:  452 Sorry,  ADV not accepted,  Attachments not accepted

or a positive warning:

            S:  354  Begin sending data.  Warning:  ADV not accepted,
Attachments not accepted

In our design, we offer the HOOK into the DATA stage. That makes a customer
implementation leaving it UP to them to define what is SPAM or not.

Anyway,  not reaching the DATA state is the goal I think the effort should
concentrate on.

The benefit by defining the new controls in SMTP,  is that we can now FEED
traceable information to the customer to allow them to do what they want
with it.

---
Hector Santos, CTO
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

---

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 01 Dec 2003 00:25:04 -0500

Hector Santos wrote:
[..]
> percentage of the spammer problem.
[..]

 From my understanding, what you are basically asking is to make every 
single email on the Internet have a valid return address which will be 
checked by every system. This would require an RFC to be published 
laying out this requirement, which would in my view change the current 
SMTP specification, or in your view would consistute the codification of 
  the "spirit" of the law, as it is inherently present today.

But, why lock down the actual email address instead of the domain? Why 
require every email to have a valid return address and domain? Shouldn't 
requiring a valid domain name suffice?

> into a fuzzy logic and that's not the purpose of SMTP.

The SMTP specification and related RFCs make it very clear that parsing 
the RCPT TO and MAIL FROM addresses is perfectly fine, and that is done 
all the time (DATA is a different story). I do not see any problems with 
"interpretation" of email addresses used in RCPT TO and MAIL FROM 
commands, and having them split into domain/address sections. Why is 
that fuzzy logic if the SMTP specification defines a very specific 
format that addresses must follow?

Why go through the trouble of having to do an RCPT TO callback, when an 
easier and more lighweight mechanism will do fine (like a LMAP cached 
DNS query)?

[..]
> see only some sort of CRC32 checker maybe to be added, but that's it.
[..]
> is ok.
[..]

What the point of having a CRC check? TCP assures an error-free 
connection and SMIME/PGP can assure integrity with a digital signature?

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"And this too shall come to pass"
-------

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 00:31:39 -0500

Exactly,  this is why is critical that before any proposals are worked on,
the RFC specifications must precisely be clear in its functional
specification at each and every interface point.  Once done, the proposals
is then easy to write to fit the new controlled specs.

I've seen this happen before and Yakov only needs to do is speak to Randy
Bush about the demise of the Fidonet Technical Specifications Committee
(FTSC) and the Fidonet Network itself (still present but not in the numbers
it use to be).

The interpretation of the specs, the insistence of FTSC administrators to
comply with legacy specifications,  their fear of losing control to militant
new groups,  help kill it.   The internet was coming and developers were
CRYING to get new internet ready specifications used by practically all the
vendors accepted by the committee.   However, to be compliant (hence get a
product code),  you had to support a legacy protocol called FTSC1 describing
the absolute minimum how a client and server handshakes to identify and
exchange data.   If your software didn't support FTSC1, the net coordinators
(ISP) would remove the node (mail server customer) from the membership list
(DNS).  This is akin to being removed your MX record removed from DNS for
not having a valid SMTP server.    When you joined the network, your host
(ISP) did FTSC1 compliance testing of your installation before issuing an
address (MX record) into the nodelist (DNS).

Now, in no way, am I advocating a strict administrated network like Fidonet
offered, but there were no security or access issues.   We can learn and
borrow quite a few ideas from it.    In a way, we are sort heading in the
same direction with SMTP with a "central authority" controlling your mail
server integration into the backbone.  It works, but do we really want this.
I don't think so.

Fix the SMTP specs people.  Everything else will follow. The specs are
great, flexible. We just need to leave no room for anonymous abuse.

---
Hector Santos, CTO

----- Original Message ----- 
From: "Chris" <[email-masked]>
To: <[email-masked]>; "Yakov Shafranovich" <[email-masked]>
Cc: "ASRG" <[email-masked]>; "Dave Crocker" <[email-masked]>
Sent: Sunday, November 30, 2003 11:35 PM
Subject: RE: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

>
> Doesn't all this discussion prove that the interpretation of a protocol
can
> vary?
>
> Therefore any attempts to use a "feature" that is disputed will probably
end
> up breaking some software
>
> Regards
> Chris
>
>
>
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of Eric
> > S. Raymond
> > Sent: Monday, December 01, 2003 2:42 PM
> > To: Yakov Shafranovich
> > Cc: ASRG; Dave Crocker
> > Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about
> > CallerID Verification)
> >
> >
> > Yakov Shafranovich <[email-masked]>:
> > > existing RFCs are changed.
> >
> > The spirit of the law, at least at the time I was involved with
> > RFC2822, was certainly that the return address was required to be
> > valid.  As for the letter...
> >
> > Section 3.6.7 of RFC2822 says:
> >
> >    The trace fields are a group of header fields consisting of an
> >    optional "Return-Path:" field, and one or more "Received:" fields.
> >    The "Return-Path:" header field contains a pair of angle brackets
> >    that enclose an optional addr-spec. [...]
> >
> > The addr-spec is optional because it maty be omitted, as in daemon
> > bounces.  RFC2822 continues:
> >
> >    A full discussion of the Internet mail use of trace fields is
> >    contained in [RFC2821].  For the purposes of this standard, the trace
> >    fields are strictly informational, and any formal interpretation of
> >    them is outside of the scope of this document.
> >
> > Thus, RFC2822 does not mandate that the address be valid.  We are
> > referred to RFC 2821.  Section 4.1.1.4 reads:
> >
> >    When the delivery SMTP server makes the "final delivery" of a
> >    message, it inserts a return-path line at the beginning of the mail
> >    data.  This use of return-path is required; mail systems MUST support
> >    it.  The return-path line preserves the information in the <reverse-
> >    path> from the MAIL command.  Here, final delivery means the message
> >    has left the SMTP environment.  Normally, this would mean it had been
> >    delivered to the destination user or an associated mail drop, but in
> >    some cases it may be further processed and transmitted by another
> >    mail system.
> >
> >    It is possible for the mailbox in the return path to be different
> >    from the actual sender's mailbox, for example, if error responses are
> >    to be delivered to a special error handling mailbox rather than to
> >    the message sender.  When mailing lists are involved, this
> >    arrangement is common and useful as a means of directing errors to
> >    the list maintainer rather than the message originator.
> >
> > I read this to mean that the Return-Path must, if present, be a
> > valid mailbox for a responsible person (note the MUST).
> > --
> >   <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
> >

---

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 01 Dec 2003 00:42:30 -0500

Hector Santos wrote:
[..]
[..]
[..]

Can we leave your RCPT TO proposal out for a second? Leaving that aside, 
what exactly do you propose?

Yakov
-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"Power tends to corrupt, and absolute power corrupts absolutely" (Lord 
Acton)
-------

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 00:57:50 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Monday, December 01, 2003 12:01 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> field, regardless of its knowledge of the validity of the mailbox.

hmmm??   It is expected to be VALID!   Whether thats true or not, is a
different story which is 100% expected as a possibility and handled
accordingly depending on server implemention.  In our case,  it is
blacklisted after 72 tries (default).   That gives you 3 days to fix your
server can get back up.  If you can't get your server up, then you have no
business being a mail server.   On a new attempt, our software will try once
more.  If successful, it is removed from the blacklist.  Otherwise, it is an
immediate failure.

> server to complete the process of delivering that queued message to you?

Thats not a SMTP problem.  These are ethical and legal issues.

First, if you write/store mail using company resource, they own your email.
They can do whatever they want.  This is legally enforced.

Second, as far as SMTP,  it is doing what it was technical designed done to
do - queue for delivery.  If the ADMIN wants to delete or kill the queue,
thats up to him.  That is not an SMTP decision.

Now, on the other hand, if your system is acting as a passthru system, then
it has TECHNCIAL responsible to move the mail - UNALTERED.   If the ADMIN
kills passthru mail,  he is putting his operations (and business) at risk.
I can't remember but I believe this is legalled enforced as well.   If your
software can not correctly handle the passthru operation of mail, you have
no being in the mail transport business.

Now, what is consided passthru mail?

Basically routed mail, but there are two types of routed mail.

If mail from another server is routed thru your system, you can not delete
it or alter it.   This is considered true passthu mail.

If the mail came from a USER to its HOST (your system) to be routed to
another site (not local), then this may be considered NON-PASSTHRU,
depending on the type of user:

Employee - company owns your email stored on the system.

Commercial -  user using an ISP.  This is questionable.  In my opinion,  in
this case, this is passthru and the ISP has no business in the
killing/altering of a commercial user.   However,  the ISP has to right to
have a disclaimer for user mail (i.e., no porn) passing thru his system as
the original source.  This (disclaimer) is legally enforced.

However, again, if the mail is true passthru, your server has the ultimate
responsibility to move it.  It can't just let it die.

---
Hector Santos, CTO

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 1 Dec 2003 01:34:19 -0500

> field, regardless of its knowledge of the validity of the mailbox.

...and that return-path field must describe the valid submission address
from which the mail entered.

The sentence "This use of return-path is required; mail systems MUST support
it.  The return-path line preserves the information in the <reverse-
path> from the MAIL command." is pretty definitive.

> server to complete the process of delivering that queued message to you?

I am saying that I may refuse mail becayse that path has become invalid
and be in strict conformance with RFC2821.  When Hector Santos talks about
strictly enforcing the SMTP standards in order to close loopholes that
spammers are using, this is what he means -- and he's got a point.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 06:53:12 +0000

On Dec 1, 12:57am, Hector Santos wrote:
} 
} ----- Original Message ----- 
} From: "Bart Schaefer" <[email-masked]>
} 
} > } I read this to mean that the Return-Path must, if present, be a
} > } valid mailbox for a responsible person (note the MUST).
} >
} > I read this to mean that the server is reqired to insert the return-path
} > field, regardless of its knowledge of the validity of the mailbox.
} 
} hmmm??   It is expected to be VALID!

Again, why?  (Where "this" in "I read this" is the quotation from RFC2821,
now elided, and is not the quotation from ESR that is still included.)

} > Suppose I send a message to you.  For whatever reason, it's queued at
} > my company SMTP server rather than delivered immediately to your MX.
} > While the message is in the queue, I have an argument with my CEO over
} > the interpretation of RFC2821, and he fires me.  The sysadmin immediately
} > disables my email account.
} >
} > You're saying that its now an _SMTP protocol_ violation for the company
} > server to complete the process of delivering that queued message to you?
} >
} 
} Thats not a SMTP problem.

It is, according to what you (and ESR) have claimed.  The address from which
the queued message was sent is no longer valid.  If, as you assert, it is a
MUST requirement of 2821 that the return-path be valid, then any SMTP server
that continues delivery of a queued message for which the sender has since
become invalid, is violating the protocol.

Replace the whole scenario with use of a sneakemail.com address which I
disable while the message is in the queue, if that's a better example.

} Second, as far as SMTP, it is doing what it was technical designed
} done to do - queue for delivery.

Exactly my point.  It's technically impossible for a store-and-forward
system to require that the end points are continuously valid.  Ergo, I
don't believe it's reasonable to interpret the aforementioned section of
RFC2821 the way you say it should be interpreted.

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 07:08:57 +0000

On Dec 1,  1:34am, Eric S. Raymond wrote:
}
} I am saying that I may refuse mail becayse that path has become invalid
} and be in strict conformance with RFC2821.

I don't disagree with that.

What I disagree with is the claim that the *sending* system is in violation
of RFC2821 because the return-path has become invalid.  Put another way, I
don't agree that RFC2821 *justifies* your refusal of the message, merely
that it does not *prohibit* such refusal.

Yakov's point (unless I'm completely confused) is that it would constitute
a new requirement for *sending* systems if the return-path "MUST" always be
a valid address.  Hector says that it would not be a new requirement, and I
still haven't been convinced that Hector is correct.

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 1 Dec 2003 02:11:13 -0500

> become invalid, is violating the protocol.

I didn't say that.  There's a nuance here you're missing.

The sending MTA is conformant with RFC2821, as I read it, if it issues 
a Return-Path back to the user that preserves the MAIL FROM information,
e.g. is valid at the time the message was generated.  There is a MUST 
attached to that requirement.

The receiving MTA is conformant if, upon ascertaining that the Return-Path
is invalid, it refuses the mail for that reason.

The fact that said Return-Path may have become invalid after transmission and
before receipt is an interesting edge case which does not affect the
meaning of RFC2821 in any way.  
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 1 Dec 2003 02:17:08 -0500

> that it does not *prohibit* such refusal.

You are correct.  The path may *become* invalid without putting the
sending MTA retroactively in violation.  It must however *be* valid
at time of transmission.  That is the plain meaning of 4.1.1.4.
 
> still haven't been convinced that Hector is correct.

Yakov is correct that that would be a new requiremwnt.  But "MUST
always be a valid address" is not what I am maintaining, nor is it
Hector's position unless I have misunderstood him.

We are simply maintaining that sending MTAs MUST issue a valid return
path at time of transmission.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 03:23:44 -0500

----- Original Message ----- 
From: "Eric S. Raymond" <[email-masked]>
To: "Bart Schaefer" <[email-masked]>
> meaning of RFC2821 in any way.

Exactly.

It really just like regular snail mail.

When someone puts a return address on the upper left corner of the envelope,
it is the expectation of the "SYSTEM" that is is valid in order for it to
work properly.

That doesn't say anything about the "possibility" that it may not be valid
or blank which means no return process needs to be considered.

The consequences of such is that the SYSTEM can no longer return the mail
which is natural part of the system functional specification.   The SYSTEM
can no longer go any further with the invalid return address.

There is a systematic process in place that works with the idea that the
RETURN ADDRESS is valid.

Same is true with SMTP.  All commercial servers work with this design
assumption. It has too. Otherwise it doens't work which the SPAMMERS are
proving for us. :-)

---
Hector Santos, CTO

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 09:13:49 +0000

On Dec 1,  2:17am, Eric S. Raymond wrote:
}
} Bart Schaefer <[email-masked]>:
} > What I disagree with is the claim that the *sending* system is in violation
} > of RFC2821 because the return-path has become invalid.  Put another way, I
} > don't agree that RFC2821 *justifies* your refusal of the message, merely
} > that it does not *prohibit* such refusal.
} 
} You are correct.  The path may *become* invalid without putting the
} sending MTA retroactively in violation.  It must however *be* valid
} at time of transmission.  That is the plain meaning of 4.1.1.4.

You mean 4.1.1.2?  "The reverse-path consists of the sender mailbox."

} We are simply maintaining that sending MTAs MUST issue a valid return
} path at time of transmission.

And I assert that even that is a new requirement.  Valid from what point of
view?

Suppose the (not uncommon case) of a sending MTA that is not the same host
as the MX for the sender's domain.  Call these MTA1 and MX1.  A message is
transmitted via MTA1 to the recipient's MX, which I'll call MX2.  If the
sender mailbox is valid at MTA1 -- that is, MTA1 can successfully deliver
a DSN to that mailbox in the event of a 5xx response from MX2 -- where is
the requirement that the mailbox also be valid when presented as a RCPT TO:
at MX1?

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 1 Dec 2003 04:35:47 -0500

> the requirement that the mailbox also be valid when presented as a RCPT TO:

RFC2821 says the Return-Path has to be the same address that was
presented to the MAIL command.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 1 Dec 2003 10:54:52 +0100

On 2003-11-30 20:12:34 -0500, Yakov Shafranovich wrote:
> >>Allows me to rephrase this - This is a requirement that the receiver is
> >>required to send the delivery failure notice back to the sender. It is
> >>not a requirement that the address for that notice should exist and be
> >>functional, or in your case be able to reach the RCPT TO stage.
> >
> > Sorry, I disagree with your interpretation of the current specification.
> >
> > A Mail From:  must be valid if all other RFC specifications is to fit.
> >
> [..]
> >I am failing to see why so much resistance?  All you are basically proving
> >is how the CURRENT specification needs to be tighten up.
> >
> [..]
> 
> What I am trying to point out that your specific proposal is not 
> compliant with the current architecture.

I do think it is compliant. RFC 2821 mentions here and there that mail
may not be accepted for unspecified policy reasons, for example:

    3.3

    MAIL FROM:<reverse-path> [SP <mail-parameters> ] <CRLF>
    [...]
    If the mailbox specification is not acceptable for some reason, the
    server MUST return a reply indicating whether the failure is
    permanent (i.e., will occur again if the client tries to send the
    same address again) or temporary (i.e., the address might be
    accepted if the client tries again later).

    [...]
    - If the verb is initially accepted and the 354 reply issued, the
      DATA command should fail only [...] or if the server determines
      that the message should be rejected for policy or other reasons.

    3.7

    If it declines to relay mail to a particular address for policy
    reasons, a 550 response SHOULD be returned.

etc.

I believe that these are left intentionally undefined ("some reason",
"policy or other reasons", "policy reasons") to allow the administrator
to impose arbitrary (but hopefully sensible) rules on his systems
without violating the RFC. 

So, I believe that Wildcat or Exim servers which try to validate the
whole return path are just as compliant as sendmail or qmail servers
which try to only validate the domain part of the return path.

> others, who worked on the original 821 and 2821 RFCs can tell us.

Eric told us, but I'm not convinced. I would agree with him that it is
the spirit of the law, but the letters he has shown us to support his
interpretation are rather weak. Anyway, I think this is a red herring.
SMTP doesn't have to require return-path validation, it only has to
allow it.

> But the fact today is that no one is expected to provide a valid
> address, and any system relying on this, will fail in some cases
> unless the existing RFCs are changed.

Yes, but that's the whole point, isn't it? Some people choose to send
out messages with invalid return paths. Many of them are spammers, some
are only incompetent. In any case, they cause problems: DSNs cannot be
delivered and often replies won't work either (if the message headers
contain the same sender address). So it do agree with Hector that it is
a sensible policy decision to not accept such messages in the first
place. 

	hp

-- 
   _  | Peter J. Holzer    | In this vale
|_|_) | Sysadmin WSR       | Of toil and sin
| |   | [email-masked]         | Your head grows bald
__/   | http://www.hjp.at/ | But not your chin.           -- Burma Shave

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 1 Dec 2003 05:09:13 -0500

Bart Schaefer <[email-masked]>:
> } RFC2821 says the Return-Path has to be the same address that was
> } presented to the MAIL command.
> 
> So what?  MX1 isn't any party to the protocol exchange between MTA1 and
> MX2.  The reverse-path is completely valid as far as MTA1 can tell; if
> MX2 responds 5xx, MTA1 will do the right thing.  You still haven't shown
> me how or why any of this affects what happens if MX1 becomes party to
> a separate SMTP transaction initiated by MX2, whether involving the same
> or any other mailbox.

I don't see how the presence of MXes matters.  Are you maintaining that 
what's in the MAIL FROM need not be a valid reply address?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:10:48 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Monday, December 01, 2003 4:13 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> the requirement that the mailbox also be valid when presented as a RCPT
TO:

C: connect to MX2
S: 250 welcome
C:  HELO MTA1
S: 250 HELLO there!   (ignore any validation here for the moment)
C: MAIL FROM: MTA1@MX1
--- BEGIN WCSAP VALIDATION ---
C: connect to MX1
S: 250 welcome
C:  HELO MX2
S:  250  Hello!
C:  RCPT TO: MTA1@MX1
S:  552 - Sorry doesn't exist
C: QUIT
S: 250
--- END WCSAP VALIDATION return 4xx or 5xx depending on policy ---
S: 552,  Local user not found!
C: QUIT

I don't see the problem.

Also, keep in mind that a  return path is persistent cross all routes.

---
Hector Santos, CTO

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 1 Dec 2003 11:13:56 +0100

On 2003-12-01 09:50:53 +0000, Bart Schaefer wrote:
> On Dec 1,  4:35am, Eric S. Raymond wrote:
> }
> } > Suppose the (not uncommon case) of a sending MTA that is not the
> } > same host as the MX for the sender's domain. Call these MTA1 and
> } > MX1. A message is transmitted via MTA1 to the recipient's MX, which
> } > I'll call MX2. If the sender mailbox is valid at MTA1 -- that is,
> } > MTA1 can successfully deliver a DSN to that mailbox in the event of
> } > a 5xx response from MX2 -- where is the requirement that the mailbox
> } > also be valid when presented as a RCPT TO: at MX1?
> }
> MX2 responds 5xx, MTA1 will do the right thing.

Yes, but if MX2 accepts the message (e.g., to pass it on to MX3) and
then discovers a problem, it won't be able to send a DSN. Since MTA1
cannot know whether MX2 is the final delivery agent or just an
intermediate relay it has to send a globally useful reverse path, not one
which is useful only at MTA1.

The open question is: Do the requirements, which the RFC puts on the use
of the reverse path (DSN's MUST be sent to the reverse path, the
reverse-path MUST be copied to the Return-Path header on final delivery,
etc.) also constitute a requirement on the sender to provide a valid
return-path? Or is it allowed to insert an invalid path if it isn't
interested in DSNs, Replies, etc.? I can see no clear requirement in RFC
2821 that the mailbox of the sender MUST be globally reachable and
accept mails. The expectation is clearly there (otherwise the
requirement that DSNs are sent there, or the care in preserving it at
final delivery wouldn't make much sense), but the explicit requirement
is IMHO missing.

	hp

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:28:58 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: <[email-masked]>
Sent: Monday, December 01, 2003 4:50 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> a separate SMTP transaction initiated by MX2, whether involving the same

Doesn't apply.

Your example is akin to a user with an address maintained by an ISP.   In
general, the user will use his ISP server to send mail.

But you said, he will do peer to peer and send directly using his OWN send
software.

The same applies since he is using his ISP ADDRESS, the validation still
takes place at his ISP site.

---
Hector Santos, CTO

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 10:30:28 +0000

On Dec 1,  5:09am, Eric S. Raymond wrote:
}
} Bart Schaefer <[email-masked]>:
} > } RFC2821 says the Return-Path has to be the same address that was
} > } presented to the MAIL command.
} > 
} > So what?  MX1 isn't any party to the protocol exchange between MTA1 and
} > MX2.  The reverse-path is completely valid as far as MTA1 can tell; if
} > MX2 responds 5xx, MTA1 will do the right thing.  You still haven't shown
} > me how or why any of this affects what happens if MX1 becomes party to
} > a separate SMTP transaction initiated by MX2, whether involving the same
} > or any other mailbox.
} 
} I don't see how the presence of MXes matters.

Hector's caller-ID proposal determines the validity of an address coming
from MTA1 by initiating a new SMTP transaction with the corresponding MX,
in this case MX1.  We wouldn't be having this discussion at all if the
presence of MX1 were not required.

} Are you maintaining that what's in the MAIL FROM need not be a valid
} reply address?

I am maintaining that a separate SMTP transaction among MX2 and MX1 is
not a reliable test of whether what's in MTA1's MAIL FROM is a valid
reply address, because the existing protocol definition cannot require
that it be a reliable test.

If that means the MAIL FROM is not a valid reply address, then yes, I
am asserting that nothing I've yet seen quoted from RFC2821 requires
that what's in the MAIL FROM need be a valid address to which MX2 can
reply.  I further assert that if 2821 did require that, it would then
prohibit those edge cases that you've previously dismissed; which edge
cases it manifestly does not prohibit.

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:32:31 -0500

----- Original Message ----- 
From: "Eric S. Raymond" <[email-masked]>
To: "Bart Schaefer" <[email-masked]>
> what's in the MAIL FROM need not be a valid reply address?

Just wish to point out that the MAIL FROM does not need to be a reply
address.  It is usually the case though in non-server MTA clients (Outlook,
Eudora, etc).

It is typically not the same for MAILING LIST systems.  The MAIL FROM is the
list admin or error path.

But in all cases, it is presumed to be valid.

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:42:42 -0500

----- Original Message ----- 
> From: "Peter J. Holzer" <[email-masked]>
> To: <[email-masked]>
> Sent: Monday, December 01, 2003 5:13 AM
CallerID Verification)

> interested in DSNs, Replies, etc.?

In which case, a null address is used.    But using a NULL imposes other
restrictions on spammers, that is why it is not used by SPAMMERS.  Namely, a
NULL address triggers a single RCPT TO: allowance and only a LOCAL user
allowance, among other things.

> I can see no clear requirement in RFC

That is basically all I am recommending that needs to be done, clarify the
specs and the solutions will come like no tomorrow.  It doesn't have to be
an expensive Caller ID method, but it is the only current way to valid it.
The research will immediately go into getting a better, optimized, no
overhead caller id system!

80% of the spammer base will disappear or begin to make them legit, in which
case, makes them traceable.  A remarkable improvement will be made to the
industry. The IETF will become sore from all the pats on the back they will
receive.  :-)

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 06:15:02 -0500

----- Original Message ----- 
> presence of MX1 were not required.

But is is only used because the user provided his address as MX1.

> that it be a reliable test.

The existing procotol must certainly defines that it is a reliable test -
WHEN USED by COMPLIANT systems.  When it is used by non-compliant systems,
then you will be blocked with ESMTP VERSION 9.0!

> cases it manifestly does not prohibit.

Your assertion is based on validity of non-compliant usage which is EXACTLY
what we are trying to block!

My assertion is that the 80% of the spamming problem is SOLVED by conforming
to the specs.

The RFCs clearly indicates that the Return-Path: is a REQUIREMENT which
comes from a required MAIL-FROM:  with the presumption is it a RELIABLE
return path.

But that also doesn't mean that it may not be VALID.

In COMPLIANT systems, it will be.

In SPOOF systems, it will not.

What is so hard to understand about that?

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 06:17:32 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Monday, December 01, 2003 5:32 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> On Dec 1,  5:10am, Hector Santos wrote:
> }
> } I don't see the problem.
>
> If I explain any further, I'll just be repeating myself; so I won't.
>

Well, I'm sorry. I am very opened minded.  You have failed to show me why a
CALLER-ID system based on the requirement of a valid RETURN-PATH will not
work.

---
Hector Santos, CTO

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 10:32:50 +0000

---

From: Dave Crocker <[email-masked]>
Date: Mon, 1 Dec 2003 22:29:45 -0800

Chris,

C> Therefore any attempts to use a "feature" that is disputed will probably end
C> up breaking some software

 good observation.

 we can probably make it work as a variant of 'be liberal in what you
 accept and conservative in what you send'.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>