Subject: RE: [Asrg] Trust relationships etc.

From: "Brian Azzopardi" <[email-masked]>
Date: Wed, 20 Jul 2005 14:28:10 +0200

> I simply believe it makes a LOT more sense to identify most spam by
observing its variance from accepted and agreed form.  
> E-mail coming from a given  correspondent which DOES NOT LOOK LIKE the
mail you expect to get from that correspondent 
> can, and probably should, be quarantined or even t-canned until a
different treatment is indicated.

> If I get a 170K-byte PIF file attachment from my dear old Aunt
Mildred, it's a pretty safe bet that 
> it's a virus or worm... she would simply never legitimately send me
anything like that (nor, in fact, 
> would probably anybody else).

You just described statistical filtering, a well-known variant being
Bayesian filtering. Later generation Bayesian filters can be very
effective and if Aunt Mildred gets zombied the filter will allow her
email through while still keeping the spam sent from her machine out. 

SPF, reputation, et al can't do that.

The "problem" with bayesian filtering and other content checking methods
is that you need to work on at least a non-trivial part, say 4k, of the
message body. This is fine for most organisations and individuals, but
maybe too resource intensive for ISPs. Keeping the statistical data for
each recipient is expensive. In organisations it might be possible to
have keep statistical data on a per-department basis with a possible
loss of accuracy, but for ISPs this can't be done.

Has anyone looked at SURBL? My experience is that it works fine.
Ofcourse, SURBL too requires having the whole message but it does not
require per-user data.

Brian Azzopardi

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Tuesday, July 19, 2005 8:26 PM
To: [email-masked]
Subject: [Asrg] Trust relationships etc.

> [HIV analogy snipped]

>> These "house of cards" trust relationships in the E-mail sphere are
> exactly
> comparable... 

> That's not exactly true of course - getting that first spam DOES NOT 
> mean
that you now have spam for the rest of your life. If only HIV infection
could be eliminated by revoking trust after the fact.

Fine, but the same thing holds true in that you don't know how much of
the "trust" you inherited from someone else was inherited from someone
else which was inherited from someone else, and somewhere down the line
there was someone whose "trust" was expressed prematurely or unwisely.
And usually there's no ready way for you to figure out what to revoke,
such that you don't throw out the baby with the bath water.

The other point is that people very simply make mistakes.  Even sites
which are normally well-administered and trustworthy still can be
infected, or still compromised.

I simply believe it makes a LOT more sense to identify most spam by
observing its variance from accepted and agreed form.  E-mail coming
from a given correspondent which DOES NOT LOOK LIKE the mail you expect
to get from that correspondent can, and probably should, be quarantined
or even t-canned until a different treatment is indicated.

If I get a 170K-byte PIF file attachment from my dear old Aunt Mildred,
it's a pretty safe bet that it's a virus or worm... she would simply
never legitimately send me anything like that (nor, in fact, would
probably anybody else).

If she sends me an ActiveX-based attachment or a Java-based decryption
script, that also is pretty clearly outside of the technical
capabilities I'd expect to see coming from her.

Meanwhile, the fact that I get illicit stuff CLAIMING to have been sent
by her should not prevent the delivery of the stuff Aunt Mildred sent me
that DOES look like the stuff she sends me.  It shouldn't be terribly
difficult for software to differentiate (at least for such extreme
cases, and probably for more nuanced cases as well) between the two.

>> Meanwhile, the stupidity of trusting SPF and such
> "reputation/authentication"
> approaches to control spam is evidenced by the following report that 
> came out within the last week... (and please forgive me if this has 
> already been >
reported here)...

> Neither *reputation* nor *authentication* schemes can have any real 
> value
ON THEIR OWN. 

I'm not EVEN convinced that they have any really compelling value
OTHERWISE, either.  The fact is that people move and travel, and
sometimes legitimately send mail from unusual places (airport E-mail
kiosks, cruise ship Internet cafes, etc etc) and while they still need
to sign and return-address their mail as usual, they may not have ANY
control whatsoever about the servers used to process the outgoing mail.

> Only by using them _together_ might one hope to gain any real
benefit. Do not make the mistake of damning reputation schemes because
of the ineffectiveness of 'authentication'.

The fact remains that authentication and reputation schemes are broken
as soon as the trusted system gets compromised by viruses or worms that
turn them into zombie spambots.  

> That said - I'm inclined to think that *reputation* established by a 
> wider
audience with a good overview of subject behaviour is more likely to be
useful (and harder to break) than some transitive *trust* thing.

Perhaps, but IMHO it's still better to have the RECIPIENT control it,
based on who THEY trust and what THEY expect to legitimately receive
from each familiar and recognized sender.

I think it makes perfect sense to put a suitably restrictive set of
acceptability rules on E-mails coming from previously unknown senders...
loose enough to allow for initial contacts, but tight enough to trip up
most spam (and, at a minimum, tight enough to ban the tricks that are
commonly used to evade antispam content filtering).  And, of course, to
virtually eliminate worms and viruses (the genesis of so many spambot
zombies) arriving in E-mails.

Gordon Peterson                  http://personal.terabites.com/
1977-2002  Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections!
http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they
"represent".
12/09/00: the date the Republican Party took down democracy in America.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

This mail was checked for viruses by GFI MailSecurity. 
GFI also develops anti-spam software (GFI MailEssentials), a fax server (GFI FAXmaker), and network security and management software (GFI LANguard) - www.gfi.com 

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Wed, 20 Jul 2005 18:52:15 +0000 (GMT)

>loose
>enough to allow for initial contacts, but tight enough to trip up most
>spam
>(and, at a minimum, tight enough to ban the tricks that are commonly used
>to
>evade antispam content filtering). And, of course, to virtually eliminate
>worms
>and viruses (the genesis of so many spambot zombies) arriving in E-mails.

> What is the leanest, cleanest message that a spammer would be happy
sending? 

Well, judging by what I see *here*, there are messages which contain little or 
no plain text at all... (or at most, a little gibberish which seems to have no 
commercial value).  I presume that they had spam content in the HTML-burdened 
portion (which got stripped by my incoming mail filtering system).

But I get a fair number of little E-mails which contain less than 2K (most of 
which is in the header).  Usually they contain a Web site URL somewhere (which 
obviously nobody ought to visit... but at least, without HTML, the link isn't 
misrepresented.)

> How is it distinguished from a 'good' message from an previously
unknown sender?

In the general case, it cannot be... at least not reliably, in every case.  And 
that's in part because different recipients might not be able to universally 
agree on what "good" means.

The important thing is that eliminating HTML and attachments from unknown 
senders *hugely* reduces the number of tricks and subterfuges available to the 
spammer for confusing, evading, and bypassing content filters.  This maximizes 
the capability of content filters for further differentiating desirable from 
undesirable E-mails.

As to the specific techniques used by the content filter (and Spam Assassin 
seems a good example of the genre), that's really a separate question;  I'm not 
even convinced that we want there to be one universal acceptance ruleset (since 
spammers already seem to tweak their messages so they pass the widely used 
default content filter rulesets).  The more such rulesets there are, and the 
more widely divergent the rules used, the narrower and twistier the gauntlet 
that spammers must try to negotiate to try to see their mail delivered.

Meanwhile, just establishing HTML and attachments as the virtual "kiss of death" 
for spam E-mail can hardly help but reduce total spam volume (at least in terms 
of aggregate byte count) so that achieves part of the cost reduction goal.

And of course, virtually eliminating E-mail as a transmission vector for 
recruiting spambot zombies strikes ANOTHER major blow in the fight against spam 
and E-mail abuse.

Gordon Peterson                  http://personal.terabites.com/
1977-2002  Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections!  http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.

---

From: [email-masked]
Date: Thu, 21 Jul 2005 06:46:42 +0000 (GMT)

>email through while still keeping the spam sent from her machine out. 

No, not really.

Those approaches basically look at the words used, etc., and that's not really 
what I am talking about (not for MY purposes, anyway).

My approach involves primarily things like the presence or absence of HTML (and, 
more finely, what TYPES of HTML tags are present), the presence or absence of 
attachments (and, more specifically, what TYPE of files are attached), message 
size, and so forth.  Those would be checked subject to a fine-grained 
"permissions list" established by each recipient, based upon who the stated 
sender of the message was.  This could either be managed directly, or indirectly 
via some sort of "allow this sender to send this type of material in the future" 
dialog which would open the restrictions for that sender to allow the specific 
(perhaps even unidentified) features which caused the mail to be questioned.

The DEFAULT (for unrecognized/prevously unknown senders) would be NO HTML, NO 
attachments of any kind, and limited message size (25K, 50K, 100K, or whatever, 
but probably on that order).  NOTE SPECIFICALLY that these defaults would block 
essentially all worms and viruses and other e-mail-borne malware exploits coming 
from unrecognized senders... and the narrow established permissions would 
probably block most or all such stuff coming from RECOGNIZED senders too.

A nice extension to my approach would be to add additional content expected to 
be found in a message from the given sender.  For example, a message body from a 
familiar newsletter would be expected to contain the masthead or copyright 
notice found in every legitimate copy of that newsletter.  A message coming from 
a particular correspondent might be expected to contain their characteristic 
signature file.  A message from a Yahoogroups mailing list might be expected to 
contain a Yahoogroups-type ad, or perhaps the group name in square brackets as 
part of the subject.  One might even include the characteristic mailer software 
(based on the message header tag) that the sender is known to use.  A message 
claiming to be from that sender and NOT containing their characteristic content 
and style would be immediately treated as suspect.

You're right in that a suitable Bayesian filter MIGHT recognize the difference 
between Aunt Mildred's vocabulary and that of a spammer or other abuser, but 
spammers for the last year or more have been targeting Bayesian filters by 
adding large amounts of gobbledygook to their spams to confuse their signature 
vocabulary.

>SPF, reputation, et al can't do that.

Of course.  And I consider it a (perhaps-)fatal and fundamental flaw in those 
approaches that they are based more on HOW the message was sent than WHAT was 
sent.

It's much harder for a spam or virus or worm to spoof (in a general and 
universal way) the CONTENT STYLE of the owner of the machine they've 
commandeered than it is to simply spoof a sender ID.  Even more, a typical 
recipient will likely have **NOBODY AT ALL** authorized to send them executable 
content... which would essentially make it an impossibility to spoof ANYBODY and 
get viruses, worms, or other zombie-spambot software into their machine via an 
E-mail vector.

>maybe too resource intensive for ISPs. 

I'm not proposing that this filtering be done by ISPs, at least not at the final 
levels.  There is simply (in aggregate) far more computing power available (and 
more FREELY available) at the recipient machines than there is anywhere else 
enroute.

At least as a first-cut, I think it's fine to do all the filtering at the 
recipient level (and especially if the downloading and filtering can be done in 
a nonblocking way, hopefully mostly transparently, which is of particular 
concern for dialup users).  

>Keeping the statistical data for

Straw man.  I don't think we need to limit our discussion to only just 
techniques which are suitable for ISP-level implementation, especially if that 
seems to eliminate consideration of the most promising and useful approaches.

---

From: [email-masked]
Date: Fri, 22 Jul 2005 19:45:59 +0000 (GMT)

>>Brian Azzopardi replied up here to something quoted way down there:
> 
> Yes of course. But what's the point? 
> 

> Well, if you're interested...

>> Reputation schemes will only be effective if enough people implement
> them, 

> Well no. For instance - we can consider DNSBLs as sources of reputation
reports pertaining to (usually) the source IP "identity". This identity is
not typically subject to any "authentication" since we normally think of it
as "practically unforgeable". Or a locally maintained blacklist, even. We
only need one or two participants in these simple reputation schemes.
Generally of course, we'd expect reputation to become more useful as the
observation is wider (and deeper).

Sure, but it also becomes subject to corruption and misuse, too... it's easier 
for someone to maliciously "joe job" someone, the way Comcast was recently 
tricked into blocking all incoming mail referencing "afterdowningstreet.com".

>> and it will still *not* solve a spamming zombied machine.

> Depending on what identities we're collecting reputation scores for, this
need not be the case. A zombied machine will tend rapidly to loose what
good reputation it had.

Right, but again, that's locking the door after the horse is gone.

>> Authentication is not an answer - we must assume that all data sent from
> a zombied machine can be falsified and that authentication details can
> be stolen.
 
> I think you're missing the value of authentication. We'd like to "know"
that a message does indeed "belong" to the identity asserted (e.g. as the
"sender"). Then we can apply the reputation associated with that identity
when assessing the message. Or to put it another way; we'd expect that the
reputation for an *authenticated* identity would be a *better* predictor
(of future behaviour associated with that identity) than would be the
reputation of an unauthenticated identity.

The problem is that it encourages one to make go-nogo decisions based on 
PREVIOUS behavior.

Part of that means allowing messages from a recently-compromised machine because 
it used to be reputable.  Another (equally or even more nasty) component is now 
not accepting mail from a system because it had briefly been zombied, after it's 
now again clean.

> If an authenticated identity is associated with a spam stream, I don't care
whether it's a zombie or a "real person". That credentials can be stolen,
or misappropriated tokens presented, isn't relevant. Messages claiming that
identity will be associated with the poor reputation.

Exactly, and that's not really fair, nor is it even justified.

If one can reliably differentiate "good stuff" they send from "bad stuff" the 
spambot is sending, then it's crazy NOT to do that.

>> Spam filtering has to be done on a per-message basis.
 
> It certainly can be - but can also be done on other bases.
Messages are not simply isolated blocks of text. They're parts of a stream
(or body) of mail that has properties which may be worth considering.

> But back to your original question, the "point" would be that reputation
scores pertaining to authenticated identities might be useful input to
statistical (and heuristic) filters. You may feel that these filters are as
good as they need to be - I suspect that this won't always be true.

In the end analysis, I'm not sure that reputation and authentication are EITHER 
necessary OR sufficient... or, for that matter, that they even add enough to the 
reliability of the differentiation function to justify the effort of deriving 
them.

---

From: "Jon Kyme" <[email-masked]>
Date: Fri, 22 Jul 2005 21:08:36 +0100

Brian Azzopardi:> 
> > we can consider DNSBLs as sources of reputation reports
> If you consider a "table" a "chicken" you can claim a table lays eggs
> but the benefits for doing this escape me...
> 
> DNSBLs are blacklists and usually have strict criteia for being
> blacklisted. Trust is fudgy, it's not the black and white answer that a
> DNSBL gives you.
> 

You seem confused.
Reputation != Trust. Whereas black/[grey/]white lists are indeed usefully
considered as reputation 'services' (albeit binary - as a rule).

There's a pretty good overview article (with links / references) writen by
one Robert Haskins, here:
http://linux.sys-con.com/read/48128.htm

---

From: "Jon Kyme" <[email-masked]>
Date: Fri, 22 Jul 2005 21:25:38 +0100

>reliability of the differentiation function to justify the effort of

That remains to be seen. You're entitled to your *opinion* of course. I'll
guarantee that reputation will be pretty valueless as long as it's not
used.