Subject: Analysis of SPF benefits for reduced filtering

From: Dean Anderson <[email-masked]>
Date: Tue, 10 Aug 2004 19:55:57 -0400 (EDT)

It has been reported that AOL is already using SPF to give reduced 
filtering to SPF-using domains. Is this a good idea?

IF you use SPF to provide less stringent anti-spam processing, then you
are MORE vulnerable than you were before. You have shot yourself in the
foot.  Suppose for example that AOL subjects MSN users to less stringent
anti-spam filtering because MSN uses SPF.  MSN is still vulnerable to
viruses as it was before it used SPF, and it is just as vulnerable to
disposable account creation as it was before.  Using SPF will __attract__
abusers to MSN, because they can get more spam through to AOL, because it
is subject to less processing.  Since AOL is doing less processing on the
same spam, AOL users get more spam. SPF is bad for both companies.

And of course, anyone who sets up a disposable domain can also get spam
through to AOL by creating an SPF record for the domain. Disposable
domains along with disposable or stolen accounts is a major problem now,
and it remains a major problem under SPF.

Anything that reduces spam filtering without reducing the number of
abusers will be harmful.

Basically, SPF gives abusers the opportunity to whitelist themselves, or
the opportunity to identify ISPs that may be whitelisted. Any kind of
whitelist that is under the control of the sender, rather than the
recipient is also going to be ineffective and harmful.

Dean Anderson
Av8 Internet, Inc

---

From: Rand Wacker <[email-masked]>
Date: Tue, 10 Aug 2004 20:01:30 -0700 (PDT)

> filtering to SPF-using domains. Is this a good idea?

Incorrect, see http://postmaster.aol.com/spf/:

---
AOL will begin using SPF records to maintain our whitelist in the near
future. If you want to remain on AOL's whitelist, you will need to
establish an SPF record for your domain as AOL will begin to query
whitelisted IP addresses from a domain's SPF record shortly. Without an
SPF record, your whitelist entries may expire.
---

And you are right, giving *preferred* treatment for successful
authentication (without other tools like a whitelist) is a bad thing to
do.  Subjecting /un-authenticated/ messages to increased scrutiny might be
justified to do though (as Microsoft has announced they will start doing).

-Rand

---

From: Dean Anderson <[email-masked]>
Date: Wed, 11 Aug 2004 21:03:25 -0400 (EDT)

> SPF record, your whitelist entries may expire.

What do you mean by "establish an SPF record"? This just means putting an 
SPF record in for your domain right?  Is there something AOL-specific that 
must go in this record?

> do.  

Ok, but if you already have a whitelist, what's the point of having SPF?

One can easily use DJB's rbldns for a DNS whitelist, with practically the
same Sendmail/etc rule as a DNS blacklist. There's only a trivial config
difference between DNS whitelisting and DNS blacklisting.

If you use SPF to maintain your whitelist, then anyone who creates an SPF
record will get whitelisted. Do you mean that only certain domains will be
whitelisted? If so, then it seems to me that SPF is just a proprietary
protocol for AOL to maintain its whitelist, and a way for AOL to
discourage outsourcing.  Neither of these things seem to be in the
interests of the IETF or the internet community.

That said, you need more than just a whitelist. If you subject any email
to less filtering, and it is known from which domains you do that with,
you will encourage abusers to conduct abuse from those domains, and your
users will get more junk.  Whitelisting has to be done carefully, and its
more of a response to abusive blacklists by selectively rejecting the
blacklist than a rational response to spam.

We also whitelists, but only around particular filter flaws: One of things
we do is count each occurance of an IP address in headers. Too bulky, and
messages with that IP address get queued instead of delivered.  This test
has to be disabled for common addresses (192.168/16, 172.16/14, and 10/8)
as well as our own mailservers and those of large ISPs. This only modifies
_this_ particular test, and only because _this_ test is invalid if there
is genuinely a lot of email from a particular IP address.  But I would not
want to whitelist just any domain that has an SPF record. That would be
bad.

But I have some doubts about the overall validity of that bulkiness test.  
If the test were valid, it might in fact be helpful if Large ISPs did
identify their outbound servers. However, finding out this information is
not hard, if only you monitor your mail queues. It hardly seems worth the
trouble of an IETF protocol.

> Subjecting /un-authenticated/ messages to increased scrutiny might be

I'd like to draw your attention to relativity: 
Subjecting /un-authenticated/ messages to _more_scrutiny_ is the same as 
subjecting authenticated messages to _less_scrutiny_

	--

---

From: Philip Miller <[email-masked]>
Date: Sat, 14 Aug 2004 12:04:49 -0400

> SPF record in for your domain right?  Is there something AOL-specific that 

It means exactly what it says: publishing a TXT record with the magic number 'v=spf1', 
that complies with the draft SPF specs on spf.pobox.com. There are no AOL-specific 
modifiers or mechanisms in SPF.

> Ok, but if you already have a whitelist, what's the point of having SPF?

AOL maintains a list of domains that have indicated to AOL that they are bulk mailers, and 
are willing to comply with AOL's rules for bulk mailers. One of those rules is that if AOL 
gets too many 'this is spam' complaints about messages from a domain on the list, that 
domain is removed from the list.
Currently, AOL maintains a private list of IPs of these senders, entered by hand by the 
senders, which they call their 'whitelist'. AOL wants to replace a private, proprietary 
list with a distributed list accessible to anyone. It seems to me that this would be 
beneficial to other ISPs competing with AOL, because they would gain access to a large 
portion of the data AOL has collected.

> interests of the IETF or the internet community.

Only domains that ask to be whitelisted by AOL get whitelisted. They must currently enter 
their outgoing MTA data into AOL's database by hand. AOL is moving from a proprietary 
database accessible only to them to a standardized, public, distributed database of 
senders. This puts the rest of the email-receiving world closer to even footing with AOL.

SPF publication appears to be a new ADDITIONAL requirement to enter and stay on the 
whitelist. If anything, it replaces the manual entry of MTA IP addresses, which means less 
work for legitimate senders of all stripes.

As for discouraging outsourcing, SPF makes it pretty easy to list outsourcers, via the 
include mechanism.

> blacklist than a rational response to spam.

If those 'known whitelisted' domains all use SPF, then how will abusers conduct abuse from 
those domains?

> want to whitelist just any domain that has an SPF record. That would be

You can do whatever you like on your mail servers. AOL is telling the world of email 
administrators that if they want their mail to continue to be delivered to AOL, they must 
publish SPF records. They are not lowering any barrier, they are raising another one.

> trouble of an IETF protocol.

And if an ISP suddenly brings a new MTA infrastructure online, they should wait until 
individual administrators see it being rejected in their logs and enter it by hand? 
Senders should bear the burden of identifying themselves. You seem to indicate that every 
domain that runs an MX should take responsibility for identifying legitimate sources 
independently, without the benefit of information that those senders can publish, and that 
other ISPs have already gathered.

> subjecting authenticated messages to _less_scrutiny_

How about authenticated messages are treated exactly as all messages currently are, while 
unauthenticated messages receive more scrutiny than all messages receive at present?

Regards,
Philip Miller

---

From: Dean Anderson <[email-masked]>
Date: Sat, 14 Aug 2004 22:27:50 -0400 (EDT)

> modifiers or mechanisms in SPF.

Ok, that's what I thought.

> AOL maintains a list of domains that have indicated to AOL that they are
> bulk mailers, and are willing to comply with AOL's rules for bulk
> mailers. One of those rules is that if AOL gets too many 'this is spam'
> complaints about messages from a domain on the list, that domain is
> removed from the list. Currently, AOL maintains a private list of IPs of
> these senders, entered by hand by the senders, which they call their
> 'whitelist'. AOL wants to replace a private, proprietary list with a
> distributed list accessible to anyone. 

Others also already do this, but don't need a special protocol to
implement it.  Further, it is still a proprietary, AOL-specific whitelist.

> It seems to me that this would be beneficial to other ISPs competing
> with AOL, because they would gain access to a large portion of the data
> AOL has collected.

I don't see how this puts anyone else on an equal footing. Its rather like
the "standards" published by Microsoft. Microsoft makes the same claims
about those standards, too. But we know that in fact they benefit
Microsoft.

> senders. This puts the rest of the email-receiving world closer to even footing with AOL.

I think its not public (the whitelist).  Only _some_ domains that use SPF 
will have privileges. This is just a whitelist.

> As for discouraging outsourcing, SPF makes it pretty easy to list outsourcers, via the 

Its also pretty easy to deny outsourcers, via refusing to use the include
mechanism.  This is my main objection. Outsourcing isn't something that
AOL or anyone else can be allowed to have any control over. Giving such
control to AOL or others is an actual harm.

The rest of my objections about failure to work, failure to stop spam, etc
aren't actual harms, they just mean the SPF effort is a waste of time.  
Your waste of time doesn't ordinarilly harm anyone, except in this case,
as above.

> If those 'known whitelisted' domains all use SPF, then how will abusers
> conduct abuse from those domains?

That's easy.  They conduct abuse the same as they do now:   They can 
either forge other users at that whitelisted domain, or they can use 
disposable accounts at the whitelisted domain, or they can steal accounts 
from the whitelisted domain.

Your misconception here is well-known and unfortunately common.  The is
misconception that somehow the abusers can't choose to use the relays of
the ISP they are using. But of course they can, and of course the ISP's
relays offer the same benefits as open relays.  But in fact, most abuse
doesn't use the ISP's relays, nor does it use open relays. At present,
most abuse connects directly. But that's just a rather arbitrary choice
made by the abuser. They can always choose to use the ISP's relays, and
they always have 3 basic choices of how to abuse the ISP's domain.  They
connect directly at present merely because its easier than figuring out
the correct ISP relays.

What your misconception boils down to is the assumption that abusers 
don't make some arbitrary choices about how to send email and can't adapt. 
This has been common to every failed anti-spam technique.

> publish SPF records. They are not lowering any barrier, they are raising another one.

First, they aren't actually doing that. They are just 'maintaining a
whitelist'.  Second, they aren't ever likely to do that, for exactly the
same reason they don't require the foolish in-addr check. Not everyone
will have the records, and AOL won't be able to refuse email because of
that.  All they can do is choose to lessen the filtering applied to those
with SPF records.  That's a lowering of the barrier.

Oh, and they will also be able to interfere with outsourcing.  
Interference could come in the form of outright refusal or extortion of
fees, service interference through "unfortunately" unreliable records, 
etc.

> Senders should bear the burden of identifying themselves. 

Not senders: recipients. The recipients monitor their logs and when they
see a bulky sender IP, they check it out.  The reason that this might not 
be helpful is that abusers can adapt their sending pattern so as not to 
appear bulky. Thus, such a check might not be valid.

> You seem to indicate that every domain that runs an MX should take
> responsibility for identifying legitimate sources independently, without
> the benefit of information that those senders can publish, and that

Yep. The reciever can't just trust the sender isn't sending spam.  
Non-abuser can publish information.  Abusers can publish the same
information.  Imagine a Bank that asked its customers for their credit
rating.

> unauthenticated messages receive more scrutiny than all messages receive at present?

Same thing. Abusers can "authenticate" themselves, and get less scrutiny.

There is no reason to conclude that MSN will have less abusers with SPF
than without SPF.  In fact, once the word is out that MSN is subjected to
less scrutiny, abusers will seek out MSN accounts.  This means that AOL
gets more spam and MSN gets more abusers.  Good plan.  Then maybe MSN and
AOL will then halt services, and abusers won't be smart enough to go to
other providers....

		--Dean