Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID Verification)

From: Brett Watson <[email-masked]>
Date: Mon, 1 Dec 2003 18:54:43 +1100

Let me see if I can summarise the crux of the matter, here.

Regardless of what the RFCs say about return address validity, it seems like a 
reasonable local policy to reject mail that has an invalid return address. 
Arguments over what the RFCs say on the matter are largely red herrings: the 
sender need not be in violation of the RFCs to justify a "refuse" policy.

The problem arises in actually determining the validity of any address. The 
only explicit mechanism that SMTP ever had for verifying addresses was the 
VRFY command, but it has fallen out of favour. Attempting to verify an 
address via SMTP using any other technique is a configuration and 
implementation dependent hack. Nothing in the RFCs requires that the "MAIL 
From" and "RCPT To" dialogue actually divulge the validity of any recipient 
address, although it's arguably "good behaviour" to do so.

So do we suggest that VRFY ought to be revived as a return-path validity 
checker? Does this benefit outweigh the disadvantages for which VRFY has been 
depreciated in the first place?

Or am I misunderstanding this argument?

Regards,
TFBW

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 03:51:40 -0500

----- Original Message ----- 
From: "Brett Watson" <[email-masked]>
To: <[email-masked]>
Sent: Monday, December 01, 2003 2:54 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> Or am I misunderstanding this argument?

No, you got it.   And you raise a very excellent possible solution which I
like because it doesn't technically change the SMTP protocol.  It now
becomes a functional requirement.

Lets put a version number of the NEXT SMTP system.  V2.0

For ESMTP V2.0 Servers:

Compliant servers must support VRFY as a way to validate return address.

With modern systems,  RCPT TO: is basically doing the same thing. The
difference VRFY is not required to be issued in any specific order and it
doesn't occur any internal server logic to remember a RCPT TO: list.  So
VRFY is the prefer method as a low overhead verification.

For ESMTP V2.0 Clients:

It can use a RCPT TO: only as a fallback if VRFY fails as a "command not
understood"

The beauty of this is that most systems already support VRFY but have it OFF
by default.  So it is just a simple matter of the sysop turning it on.

We can certainly extend this in many ways.  But by using this method, the
SMTP protocol itself does not need to change.  Only the specifications need
to be make it clear in writing that the return path must be valid. If it
isn't valid, the is up to the server implementation to decide what to do
with it.   But we can't just leave it in a "limbo" that suggests in any way
that it is may be unreachable, therefore no action should be taken that
prohibit this possibility.

I vote to "re-institute" the VRFY command as recommended requirement for
ESMTP V2.0 Protocols designs.

The argument about harvesting is a non-issue.

- Most systems offer dynamic validation with RCPT TO:

- Some might delay it to the DATA stage, but it still done (YAHOO)

- Systems that delay with gateway software will try to BOUNCE the message.
If no bounce is reached by the spammers,  then he has learned that the
address is "ACCEPTABLE."

- Systems that delay with gateway software bouncing the message to a
unreachable site, well, this only means this type of spammer didn't care in
the FIRST place if the address was acceptable or not. He was just throwing
darts on the wall.  If some stuck,  GREAT.  If not, still GREAT.  From his
stand point, his SPAM SOFTWARE completed it job (throwing darts) of sending
mail to anyone who would accept it.  Who cares if it was actually by a user
or not.  He will find out when the user reads his mail using html
references.

So in ALL cases,  it is a non-issue.

I vote for VRFY to be part of the next specs!

Thanks Brett!

---
Hector Santos, CTO
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 09:26:27 +0000

On Dec 1,  3:51am, Hector Santos wrote:
} 
} For ESMTP V2.0 Servers:
} 
} Compliant servers must support VRFY as a way to validate return address.

That doesn't address the issue (pun?).  See my example of MTA1 sending to
MX2, which tries to verify the address using MX1.  Unless you impose the
additional requirement that MX1 be able to verify all addresses that may
originate from MTA1, it doesn't matter whether you use VRFY or RCPT.

If, on the other hand, and as Yakov keeps suggesting, you verify only that
MTA1 is a valid source of mail for the domain shared by MTA1 and MX1, you
have placed no new requirements on the SMTP protocol -- you've added an
external authentication step, but not changed SMTP itself.

The most obvious workaround for anyone with an "old" SMTP server will be
to blindly respond success to VRFY of any localpart in their domain, at
which point you've got no more information (in fact, you've got less)
than if you had simply verified MTA1 in the first place.

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:19:04 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: <[email-masked]>
Sent: Monday, December 01, 2003 4:26 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> MX2, which tries to verify the address using MX1.

I did.  I saw nothing there.  See my reply which outlines the logic you
described.

> Unless you impose the additional requirement that MX1 be able to verify
all addresses that may
> originate from MTA1, it doesn't matter whether you use VRFY or RCPT.

Nothing to impose, its already part of the specification.  The RETURN PATH
is a REQUIREMENT for proper SMTP operations. period.  You don't have it, it
presents improper operations and spammers have exploited this hole!

All you have basically shown is that there is a scalability issue.

That is true.   But more importantly it will be become a redundancy issue
more than anything else, because once it works,  there will be a lot of
redundant checks against valid addressing which then says that maybe it
could fall back to a LMAP method where you just validate the domain.

---
Hector Santos, CTO

---

From: Brett Watson <[email-masked]>
Date: Mon, 1 Dec 2003 21:25:43 +1100

On Mon, 1 Dec 2003 19:51, Hector Santos wrote:
> ...

I don't see how this, on its own, overlaps with a proposal like LMAP. If host 
X sends us mail with "MAIL From:<user@Y>", then we ask an MX for Y to "VRFY 
user@Y", all we have determined is whether <user@Y> is a "valid" address, for 
some loose definition of "valid". (One of my domains has a catch-all address 
in force at the moment anyhow, so *every* syntactically valid address is a 
"valid mailbox".) Critically, what we have *not* determined is whether host X 
is *authorised* to represent that address. That is what LMAP attempts to 
ascertain.

The reinstatement of VRFY would potentially give spammers a good way of 
evaluating their address lists. A spammer could also probe around for a 
verifiable address to use in the "MAIL From:" part of the dialogue. In the 
absence of an LMAP-type authorisation mechanism, this strikes me as a rather 
bad thing.

Is your response to this a challenge/response type scenario, where the 
(known-to-be-valid) return address is sent a message asking for confirmation? 
This is a nasty burden on the victim of fraud. We need the LMAP-type 
authorisation mechanism to act as a first-pass filter to reject blatant 
frauds.

Or perhaps I've missed your point. I haven't had the time to fully analyse all 
the messages on this list in the last couple of days.

Regards,
TFBW

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 05:25:52 -0500

> I dispute that the existing protocol supports Hector's position.  As I
> said in another message, I also dispute that VRFY can be adapted for this
> purpose in any useful way.

I already proved that it works.  You have not shown that it does not work.

You all said talked about is scalability issues and FALSE positives.

I already said that the method can not do anything against false positives.

But as my statistics have shown over 80% of the abuse are from GHOST
systems!

That is a DRASTIC reduction in illegal access.

You now also question whether a return path is not required to be VALID???

Is that a question view of your thinking?

A Return Path MUST be valid in order to get a MAIL SYSTEM to bounce
mechanism work.  If you are too assume that a RETURN PATH is not to be
relied on, than we must as well throw out all the RFC dealing with DSN and
bouncing.

Again, that has nothing to do with the "real" possibility that it is NOT
valid.  The spammers have proven that.

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 06:06:10 -0500

> There are various legitimate situations in which VRFY or RCPT TO
> verification wouldn't presently work.  These would have to be solved.
> For example:
>
> - UUCP (mentioned elsewhere in this long thread.)  The host system
>   doesn't have any chance of verifying users behind the UUCP connection.
>   You'd get false positives here.

I already addressed this. This is a SERVER implementation issue.

We had the same problem in our UUCP software awhile back.   The solution was
to support  RFC 1128 5.2.8 and make sure that delay validation performed by
the UUCP/gateway software supported it by expecting the specific header
line:

             "Return-Path: <original-mail from:>

If the UUCP software does not support it, then you are correct.

The Return-Path is persistent across the route until final delivery is
reached.

> - As with UUCP, other sorts of disconnected delivery systems where the
>   receiving MTA doesn't know anything about the usernames attached to
>   a domain.  e.g. the user who has a catchall box for all
>   [email-masked] addresses, who periodically accesses that box and
>   applies local filtering rules.  The user who just has a single
>   maildrop from which mail is retrieved via something like fetchmail
>   (or a number of equivalents).  Again, false positives.

Well,  its a compatibility issue.  Absolutely.

But in the case of the SMTP system,  at this level UUCP doesn't apply yet.
The validation proposed is at the SMTP level.

In our case, where the hosting sysop may have UUCP downlinks, he may not be
a candidate to use a SMTP based validation.  So he would have to turn it
off.

I should point out that our SMTP customers with UUCP downloads are becoming
obsolete. <g>

But we still needed to support it because in our design,  a design since
1984 before SMTP was added in 1996,   still used a UUCP mail/news format and
interface between each part of the systems:

       SMTP <--> UUCP SPOOLS<---> LOCAL MAIL/GATEWAY <--> SERVER DATABASE

UUCP customers then use uucico to move their mail to their downlinks.

> - The recipient address that is only valid under certain circumstances
>   (e.g. graylisted, or firewalled such that it only accepts mail from
>   certain sources, or has a filter installed such that it only
>   accepts mail from certain senders).  False negatives here.

Ok, but I think what with a callerid standard a place, it will help make
some of this unnecessary.

> - The system that accepts all mail so that it can update its
>   'corpus' of spam mail.  False positives.

True, which was something we though YAHOO was doing but they are validating
in the DATA state.

> Maybe they can be solved by extending the capabilities of VRFY,
> but I doubt they can be solved by changing usage behaviour.
> What would be another way?

There is never going to be a 100% solution.

The proposal simply highlights the dearth of the SMTP controls which is what
80% of the spammers are exploiting.

My contention is that you can address the 80% of the problem by having
strict definitions of the specs.  If by change there are made strict,  then
there a big part of the problem is solve when the software begins to comply.

Sure, there could be zombies.  My WCSAP implementation is using a second
fake address test and I was able to pick up many of these as an OPEN RELAY
site.

But you will not get them all. In this case, when the controls are in place,
the spammers will be reduced to traceable systems.

---
Hector Santos, CTO

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 11:07:24 +0000

On Dec 1,  5:25am, Hector Santos wrote:
}
} > I dispute that the existing protocol supports Hector's position.  As I
} > said in another message, I also dispute that VRFY can be adapted for this
} > purpose in any useful way.
} 
} I already proved that it works.

You've proved that spammers use garbage envelope sender addresses.  We
already knew that.

} You have not shown that it does not work.

I'm not trying to show that it does not work.

I'm trying convince YOU to show two things, which you have not yet shown:

(1) WSCAP works _better_ than a system such as LMAP that uses something
    lighter weight than SMTP for the callbacks.

(2) WSCAP does not impose an undue burden by requiring third-party
    SMTP servers to disavow every forged address that appears to be
    in their domain.

Your response to (1) seems always to be to change the subject, by saying
"but WSCAP works now, and needs no changes to SMTP!"  The thread on which
this message is a reply is actually an argument about just the "needs no
changes" clause, which has become a rathole.

You keep dimissing (2) as a scalability issue, but please note that it
is not a scalability issue for the servers that deploy WSCAP -- it is a
scalability issue for servers that _ARE PROBED BY_ WSCAP.  And it is a
larger scalability issue even than bogus DSNs.  You've waved your hands
a lot, but you haven't convinced me that this isn't cost-shifting on a
much larger scale than you imagine.

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 06:47:49 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: <[email-masked]>

> You've proved that spammers use garbage envelope sender addresses.  We

Then why aren't you doing anything about it?   Do you have a better method
that works within the spirit of Mr. Crocker wishes, namely no change the
PROTOCOL!?

Or are you waiting for developers to solve it for you?

Well, I have done something and have provided AMPLE envidence that it helps
solve 80% of the spammer problem.

If you want to help solve the weaknesss, great.  Yakov gave a few and so did
you.  But that does NOT mean it will not work.

> I'm not trying to show that it does not work.

Than what are saying?

> I'm trying convince YOU to show two things, which you have not yet shown:

I sure have!

>     lighter weight than SMTP for the callbacks.

I sure did!  Check out the statistics I provided.

http://www.winserver.com/public/antispam/stats/stats-2003-nov.htm

LMAP is similar to a RBL lookup at the connection level.  RBL provides an
average of 76% success rate and WCSAP provided a near 90% success rate.

Which is EXPECTED because you are goint a STEP further than just checking
the domain.

Now I can 100% verify this by analyzing the WCSAP.LOG to see how any of the
rejects were MX and CONNECT based.  But this is obvious.

>     SMTP servers to disavow every forged address that appears to be

I didn't disclaim it was not an expensive process.  What I said it is a
different issue altogether. It is a scalaribility and redundancy issue.  It
is a DESIGN IMPLEMENTATION issue.  It does not negate that fact that you can
SPAM PROOF to a large degree your system by simply validating the return
path.

> Your response to (1) seems always to be to change the subject,

oh hogwash.

> by saying "but WSCAP works now, and needs no changes to SMTP!"

Baloney, I did not say such a thing.  What needs to CHANGE is the RFC
SPECIFICATION so that we have the ammunition to put a MAIL FROM: validation
into action and be able to do so LEGALLY!   Right now, MAIL FROM is only
required for BOUNCING.   If its required for bouncing then it should be
required for possible dynamic and instant validating is SO desired by a
system.

The last thing I want is be sued by SPAMCO being paid millions by Microsoft
or whoever to spam servers and the transaction is blocked due to an MAIL
FROM: conformity check.

We need to make SPAMCO responsible at the TECHNICAL level.   Once it is
compliant, it is allowed thru, at which case, I dont' give flying $%# what
the mail is.  That is NOT my problem as a SMTP mail server vendor.

> The thread on which this message is a reply is actually an argument about
just the "needs no
> scalability issue for servers that _ARE PROBED BY_ WSCAP.

It still a SCALABILITY issue! The system that is PROBED can restrict
connection access, which is PAR for the course.  In addition, those attacks
are trackable!

> And it is a larger scalability issue even than bogus DSNs.  You've waved
your hands
> much larger scale than you imagine.

Are you are paper hat or what?  Didn't I say that this will be come a
redundancy problem if it was widely deployed?

For the umpteen time,  WCSAP is a BLACK BOX VALIDATION into the SMTP
process.

The REAL goal here is making the MAIL FROM: mandatory IN WRITING.

Once done, OPTIMIZED SOLUTIONS will come.

I don't know much clear I can get.

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 07:00:33 -0500

----- Original Message ----- 
From: "Brett Watson" <[email-masked]>
To: "Hector Santos" <[email-masked]>; <[email-masked]>
Sent: Monday, December 01, 2003 5:25 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

>  We need the LMAP-type

> the messages on this list in the last couple of days.

and I certaintly spent more than I expected this weekend (I hope it is not
all in vain)

A Callerid Verification, inherently includes a DOMAIN lookup.    It has 5
checks:

- DNS MX LOOKUP
- MX IP CONNECTION
- NULL ADDRESS REQUIREMENT  (MAIL FROM: <>)
- RCPT TO:  VALIDATION
[OPTIONAL]
- RCPT TO:  FAKE ADDRESS FOR AN OPEN RELAY TEST

Where does LMAP fit in the above?  Does LMAP required an additional Lookup
(TXT)?

---
Hector Santos, CTO

---

From: "Mark E. Mallett" <[email-masked]>
Date: Mon, 1 Dec 2003 07:16:04 -0500

> The Return-Path is persistent across the route until final delivery is

OK, but what does that have to do with the paragraph that this is
allegedly a response to?  And did you really mean RFC 1128?

> UUCP customers then use uucico to move their mail to their downlinks.

OK, but what does this have to do with the paragraph that it is
supposedly responding to?  (a) that paragraph was specifically
about things *other* than UUCP, and (b) even if it was about UUCP,
how does address the VRFY issue?

> some of this unnecessary.

I doubt it.  I imagine that with or without spam, more sophisticated
filters that apply at SMTP time will be more and more common.  RCPT TO
validation will become more and more fuzzy as that will depend more
on connection-time filter logic.  Spam may be the prime factor promoting
the development of interfaces supporting them, but once they are there,
they will be used.

Yours,
mm

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 1 Dec 2003 08:12:35 -0500

If am I still able to follow this thread, I believe I said:

In the case of the SMTP system,  at this level UUCP doesn't apply yet.
The validation proposed is at the SMTP level.

----- Original Message ----- 
From: "Mark E. Mallett" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Mark E. Mallett" <[email-masked]>; "Brett Watson"
<[email-masked]>; <[email-masked]>
Sent: Monday, December 01, 2003 7:16 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)