Subject: RE: [Asrg] Trust relationships etc.

From: "Brian Azzopardi" <[email-masked]>
Date: Thu, 21 Jul 2005 10:34:53 +0200

>>You just described statistical filtering, a well-known variant being
>>Bayesian filtering. Later generation Bayesian filters can be very
>>effective and if Aunt Mildred gets zombied the filter will allow her
>>email through while still keeping the spam sent from her machine out. 

>No, not really.

>Those approaches basically look at the words used, etc., and that's not
really 
>what I am talking about (not for MY purposes, anyway).

>My approach involves primarily things like the presence or absence of
HTML (and, 
>more finely, what TYPES of HTML tags are present), the presence or
absence of 
>attachments (and, more specifically, what TYPE of files are attached),
message 
>size, and so forth.

I think you misunderstand what statistical filters can do. Statistical
filters don't, strictly speaking, look at words, they look at events or
tokens. You can decide whatever these events mean. Most statistical
filters simple equate an event with a single word. More sophisticated
implementations can feed the filter other events such as IPs, dollar
amounts, appropriately processed time, etc. For example, in my
implementation I consider very long words as a type of event. I don't
pass the filter the word itself, just an event telling the filter
"very-long-word-here". You can easily extend a filter to know about
attachments.

Bayesian filters can not only detect the absence/presence of html but it
can also tell you which tags or even attributes are relevant
automatically. I would never have thought that #fffff or #ff0000 were
very good indicators of spam mail until I went through the statistical
data of my filter. A human can't be as thorough or as accurate as a
machine.

> ... Those would be checked subject to a fine-grained 
>"permissions list" established by each recipient, based upon who the
stated 
>sender of the message was.

This is fundamentally a whitelist being based on heuristics instead of
email addresses. Non-technical users will be stumped trying to configure
it. 

BTW, heuristics, in general, are dodgy - they are static, need manual
maitenance, and are usually easy to evade anyway.

>You're right in that a suitable Bayesian filter MIGHT recognize the
difference 
>between Aunt Mildred's vocabulary and that of a spammer or other
abuser, but 
>spammers for the last year or more have been targeting Bayesian filters
by 
>adding large amounts of gobbledygook to their spams to confuse their
signature 
>vocabulary.

Yes, and as in any arms race, there is a response. An approach is to not
pass a single word as an event but n-grams. Lookup sparse polynomial
binary hashing (SPBH) for an example of this. And it works pretty fine
too, especially with some fine-tuning :)

>>Keeping the statistical data for
>>each recipient is expensive. In organisations it might be possible to
>>have keep statistical data on a per-department basis with a possible
>>loss of accuracy, but for ISPs this can't be done.

>Straw man.  I don't think we need to limit our discussion to only just 
>techniques which are suitable for ISP-level implementation

Indeed. I was just pointing out that ISPs have significant costs in
network bandwith, administrative hassles etc. I prefer to do filtering
on my machine then let an ISP filter it for me. I was just pointing out
that huge ISPs like AOL see *lots* of spam and they too are trying to
reduce the spam problem. 

Brian Azzopardi

This mail was checked for viruses by GFI MailSecurity. 
GFI also develops anti-spam software (GFI MailEssentials), a fax server (GFI FAXmaker), and network security and management software (GFI LANguard) - www.gfi.com 

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Jon Kyme" <[email-masked]>
Date: Thu, 21 Jul 2005 10:23:19 +0100

Brian Azzopardi wrote:
> More sophisticated
> amounts, appropriately processed time, etc. 

And of course it's not hard to arange for reputation and authentication
data to be input to statistical filtering - this is easily done by adding
appropriate headers. This is a general mechanism for upstream entities to
provide input to downstream filtering.