Subject: RE: [Asrg] 2. Improving Blacklists and Reputation Services

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 9 Feb 2004 13:09:38 -0800

> 1. Is it feasible to develop a standard format and protocols 
> for storing 
> and quering data from reputation services?

Yes, see the attached.

> 2. Is it feasible for such format to be feature rich 
> providing more data 
> than a simple yes/no. Meng Wong of SPF proposed on his list a 
> while back 
> something like how many messages sent by MTA, how many were 
> spam, etc.,  akin to what SenderBase does.

You could do that, but you have to take account of the fact that the filter
is going to have lots of factors to apply in any case. For example how
reliable is the information source, does it talk about every email sender or
just some? Is it whitelist or blacklist?

If you go into too much detail you impose a model on the solution that is
probably unnecessary and is going to limit what people can do.

I think it is better to simply use the standard A pointer blacklist type
hack and then have a metadata record that tells the filter how it thinks the
information should be interpreted.

> 3. Can this be supplemented by accrediation formats and protocols?

Yes, but the volumes of mail are large, you probably don't want to do more
than DNS lookup in the first pass.

> 4. Would any of this improve blacklists?

Yes, the big problem at the moment is that blacklists are not accountable to
any party. They tend to operate by attempting to ram the arbitrary policies
they choose down every sender's throat.

I think it is better to have the sender say what they their sending policy
is. If they state outright 'I send unsolicited mail to anyone I choose' then
recipients get to say no. If they say 'I send only mail to people who ask
for it by quintuple opt-in' and someone catches them spamming, well they
chose the empirical test that they failled.

> 5. Would all of this reduce spam?

Yes.

I think that it will happen gradually. In the first phases people will use
SPF to stop their domain being joe jobbed. Given that there is so much joe
job spam that gains credits in spam assasin or whatever.

In the next phase you add in an accreditation element. This could be a paid
service, voluntary, whatever the sender chooses. There will be a lot of
different programs for different purposes. I suspect we will have the
standard honeypot approach and blacklists that simply report spam sent to
honeypots from certain domains. These work pretty well provided people do
not imagine that the honeypots are infallible. ISPs and others do not
control their customers as absolutely as some anti-spam zealots claim they
should.

There will also be paid schemes similar to that used for authenticating SSL
cert customers. SSL is not an infalible anti-CC fraud protocol, merchants
can commit abuse after the data has arrived. But it reduces fraud to a very
major degree.

Then you will have your Gates endorsed, TrustE style bonded sender ideas.
Spammers pay.

		Phill

---

From: David Maxwell <[email-masked]>
Date: Mon, 9 Feb 2004 16:21:06 -0500

> > spam, etc.,  akin to what SenderBase does.
[...]
> than DNS lookup in the first pass.

A trivial implementation of this would be one where the blacklist
returns 127.0.0.[0-100] as their percentage of certainty that the source
is a spammer.

The local tools can then multiply that percentage by their 'how much I
trust blacklist provider X' factor.

-- 
David Maxwell, [email-masked][email-masked] -->
All this stuff in twice the space would only look half as bad!
					      - me

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 09 Feb 2004 16:26:48 -0500

>>and quering data from reputation services?

There is a big difference between accreditation and reputation. Lumping 
them together like your draft does not seem to be a good idea.

Additionally, why resort to DNS if other possible ways are possible for 
retrieving such data. Some ISPs might even want to use bulk methods. The 
DNS packet size might also be an issue if the reputation and 
accreditation data is rather large. There is also no indication of what 
the values measure. Multiple values are possible. If an ISP subscribes 
or gets data from a specific reputation service like MAPS, then it is 
very likely that other methods aside from DNS can be used.

Additionally, if you have both, than new MTAs on the Internet do not 
need to pay money to someone in order to get accreditated, rather they 
can use build their reputation over time. Or they can start with 
accrediation and then switch to reputation services once their 
reputation has been established.

> probably unnecessary and is going to limit what people can do.

My question is exactly what should such model have. If it is extensible 
enough that shouldn't be a problem.

> information should be interpreted.

I do not think that using DNS for this purpose is necessary, we should 
look into multiple possibilities.

> than DNS lookup in the first pass.

Unless you use other methods. DDOS attacks on blacklists have shown than 
reliance on DNS might not be such good idea. Other possibilities like 
P2P services might be feasible.

> chose the empirical test that they failled.

I still think that reputation is separate from accreditation and is 
still valuable. All of the blacklist services around today are not going 
to disappear - we will need a way to gauge whether a specific 
accreditation authority is doing its job, and taking third party spam 
data into account can help with that.

> job spam that gains credits in spam assasin or whatever.

There is still a crucial point that SPF/LMAP does NOT reduce joe jobs in 
the mail content, only the SMTP transaction. We must make this 
distinction very clear. I still do not understand how domain identity 
for MTAs is any better than IP identity for MTAs. Sender identity is 
different.

> Then you will have your Gates endorsed, TrustE style bonded sender ideas.

Why can't we apply these ideas to today's IP blacklists? Also, "innocent 
before proven guilty" will turn into "guilty until proven innocent" 
under these schemes.

Yakov
-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"I want to know G-d's thoughts... the rest are details" (Albert Einstein)
-------

---

From: "Walter Dnes" <[email-masked]>
Date: Tue, 10 Feb 2004 01:06:43 -0500

On Mon, Feb 09, 2004 at 01:09:38PM -0800, Hallam-Baker, Phillip wrote

> You could do that, but you have to take account of the fact that the
> filter is going to have lots of factors to apply in any case. For
> example how reliable is the information source, does it talk about
> every email sender or just some? Is it whitelist or blacklist?
  [...deletia...]
> Yes, the big problem at the moment is that blacklists are not
> accountable to any party. They tend to operate by attempting to ram
> the arbitrary policies they choose down every sender's throat.

  Those two sections above are self-contradictory.  Blacklists are
accountable to MTA operators.  If MTA operators choose not to use a
specific lists, it won't be capable of forcing itself down their
throats.  I am not aware of any DNSbl that blacklists ISPs for the
simple reason that they don't use the DNSbl in question.  DNSbls *DO*
have reputations, which affect how many people are willing to use them.

> I think it is better to have the sender say what they their sending
> policy is. If they state outright 'I send unsolicited mail to anyone I
> choose' then recipients get to say no. If they say 'I send only mail
> to people who ask for it by quintuple opt-in' and someone catches
> them spamming, well they chose the empirical test that they failled.

  The problem with that is that almost *EVERY* spammer^H^H^H^H^H^H^H
"legitimate marketer" makes that claim, so it's useless.  This would
allow them to take make one spam run, morph domain name, make another
spam run, repaet, rinse, lather.

-- 
Walter Dnes <[email-masked]>
Email users are divided into two classes;
1) Those who have effective spam-blocking
2) Those who wish they did