Subject: [Asrg] Summary of junk button discussion

From: John Levine <[email-masked]>
Date: 14 Feb 2010 00:53:20 -0000

I've put a summary of the proposals and discussion so far here:

     http://wiki.asrg.sp.am/wiki/Adding_a_junk_button_to_MUAs

If I missed anything, let me know.  Or even better, log in and
fix it yourself.  If you don't have a password for the wiki and
want one, just ask.

R's,
John

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Mon, 15 Feb 2010 22:22:47 +0100

Hello,

> fix it yourself.  If you don't have a password for the wiki and

Well, sorry, I didn't participated too much in the discussion but
I have a point. I prefered not touch the wiki (although I have a
login/password) but better let you know, as it's probably not a
detail.

I don't like the name "Add a junk button", as this mechanism can
be used to signal a junk message when someone want to complain,
but it may also be used to do feedback about messages : both
false positives and false negatives.

Regards

José-Marcio

--

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 23 Feb 2010 11:30:44 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Levine
> Sent: Saturday, February 13, 2010 4:53 PM
> To: [email-masked]
> Subject: [Asrg] Summary of junk button discussion
> 

It's probably worth mentioning at this point that the IETF's MARF working group has as a charter item the issue of publishing somehow the address to which abuse reports should be sent.  We won't be getting there for a while yet, but if you're interested in this issue you might want to start following the working group list.

---

From: John Leslie <[email-masked]>
Date: Tue, 23 Feb 2010 17:49:38 -0500

Murray S. Kucherawy <[email-masked]> wrote:
> 
> It's probably worth mentioning at this point that the IETF's MARF
> working group has as a charter item the issue of publishing somehow
> the address to which abuse reports should be sent.

   Frankly, anything but in the email itself will be abused. abuse@ is
already useless.

   And within the email itself, only within the trace headers can ensure
that it's what the "most responsive party" wants.

   (What an irresponsible spammer that initiated it wants really doesn't
matter; what the ISP who maintains your mailbox wants matters a lot.)

> We won't be getting there for a while yet, but if you're interested in
> this issue you might want to start following the working group list.

   I'll pass for now... It'd be nice to notify this RG again when you
do get to it.

--
John Leslie <[email-masked]>

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 24 Feb 2010 10:14:20 +0100

On 23/Feb/10 20:30, Murray S. Kucherawy wrote:
>>  a summary of the proposals and discussion
>>
> It's probably worth mentioning at this point that the IETF's MARF working group has as a charter item the issue of publishing somehow the address to which abuse reports should be sent.

I've added a link to the marf's charter in the "See also" section of 
that page.

> We won't be getting there for a while yet, but if you're interested in this issue you might want to start following the working group list.

I'd hope that, by the time the marf WG gets there, there will be a raw 
understanding of /generalized FBLs/. Otherwise, it would be a hard 
time trying to standardize something that doesn't exist at all.

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 24 Feb 2010 11:36:25 +0000

> false positives and false negatives.

Yes, we've already discussed that problem, and several ways to mitigate the 
problem, quite extensively. I've added this text to the wiki: "Also out of 
scope are other ways in which administrators respond to such reports, 
however it's worth noting that any response mechanism should be robust to 
false reports, whether they're malicious or accidental."

> Regards
>
> José-Marcio
>
> -- _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg

-- 
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Wed, 24 Feb 2010 12:54:34 +0100

Ian Eiloart wrote:

> Yes, we've already discussed that problem, and several ways to mitigate 
> the problem, quite extensively. I've added this text to the wiki: "Also 
> out of scope are other ways in which administrators respond to such 
> reports, however it's worth noting that any response mechanism should be 
> robust to false reports, whether they're malicious or accidental."

This mechanism being described and discussed may be used more largely. Let me give an example.

You and me, we both work in an university. So, some thing useful in environnements like the ours is 
a way to have users sending feedback to us (us = mail admins) : both false negatives AND false 
positives. These feedbacks may be useful both to tune our filters, to feed statistical learning filters.

The underlying mechanism may be the same as the "junk button", although I agree with you that there 
are weaknesses related to false reports, and noise. But this is a trust problem which may probably 
be handled in a higher level, or adding some more precise information.

I find it too restrictive to call this a "junk button" as this mechanism can be used more widely 
than just this way.

-- 
  ---------------------------------------------------------------

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 24 Feb 2010 14:20:07 +0000

--On 24 February 2010 12:54:34 +0100 Jose-Marcio Martins da Cruz 
<[email-masked]> wrote:

>
> I find it too restrictive to call this a "junk button" as this mechanism
> can be used more widely than just this way.

Ah, well that's a different question. This particular discussion is about 
how to get reports from the user to the administrator of the system. How 
the user triggers the report is an exercise for the MUA programmers, but 
we've been thinking that it's likely to be a button labelled "junk".

Of course, there could be more than one button, with different labels, or 
it could be simply one or more (virtual) mailboxes into which the user 
drags emails, or it could be that some smart process monitors the behaviour 
of the user (perhaps it would report emails that the user always deletes 
without reading).

Whatever triggers the reporting process, it's possible that the client may 
ask the user for more details about the report by asking questions like 
"this email came from a mailing list, do you wish to unsubscribe?", or 
"that's the third time you've reported email from this sender, do you want 
me to add the sender to a block list", or something.

But, these are all possibilities that become easier to implement if we can 
answer a few questions to standardise reports: like what messages can the 
report carry, and what's the mechanism that will carry the report.

---

From: der Mouse <[email-masked]>
Date: Wed, 24 Feb 2010 10:25:32 -0500 (EST)

> Frankly, anything but in the email itself will be abused.  abuse@ is

One of my jobs is at an ISP.  We maintain an abuse@, and I'm one of the
people who gets mail from it.

The ham/spam ratio I see in that mail stream is substantially higher
than it is for the other mail I get through them (and there are a
handful of other role addresses at that same ISP that forward to
me-among-others, so I do have something to compare with).  Indeed, the
abuse@ mail stream is practically nothing but real abuse reports -
presumably of real abuse, though I don't really know because
investigating them is not one of my jobs there.

Of course, others' experience may - probably will - differ.  But
"abuse@ is useless" disagrees with my experience.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: John Levine <[email-masked]>
Date: 25 Feb 2010 05:45:46 -0000

>we've been thinking that it's likely to be a button labelled "junk".

Having talked to a fair number of ISPs who provide junk buttons, none of
them thought that more buttons would help.  Users don't distinguish among
all the reasons they might not want a message, and asking them to do so
is more likely to get random answers or no answers than good ones.

The only think other than a junk button that appears useful is a
not-junk button to display when looking at stuff in a junk folder.  I
suppose we could do that, but then we'd have to define what a junk
folder is.  Or if we do this with a header applied by the MDA, it could
have a flag hinting which way the junk flag is set now.

R's,
John

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 25 Feb 2010 01:22:37 -0500

On 2/25/2010 12:45 AM, John Levine wrote:

> folder is.

Not really.

Thunderbird has a junk button that toggles a "junk" flag on or off per 
email, without tying it to folders.  You can define additional actions 
as to what happens when you hit junk (move to junk folder or delete). 
And undo it.  While it's specifically tied to Bayesian training in the 
client, it doesn't really have to be.  And by fairly trivial extension, 
you could add a "FBL to <x@y>" action, default perhaps supplied by MDA 
header.

[The bayesian filtering will set the junk flag, and the actions aren't 
necessarily identical to when you explicitly mark something junk.]

Later versions of Tbird have yet another "state" - "I dunno if this is 
junk" (I think.  Probably an intermediate Bayesian score).  Which makes 
things a bit more awkward - marking something not-junk from that state 
can be tricky if you have "move marked junk to junk folder" set because 
it toggles thru "junk" first.  But that's a Bayesian implementation 
artifact/feature I think.

With more tweaking, such an implementation would be a bit more 
user-friendly, and can stand as a guide as to how you might spec out 
what the junk/unjunk button "means", and leave actions to the 
implementation.

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 25 Feb 2010 10:59:41 +0000

--On 25 February 2010 05:45:46 +0000 John Levine <[email-masked]> wrote:

>> Ah, well that's a different question. This particular discussion is
>> about  how to get reports from the user to the administrator of the
>> system. How  the user triggers the report is an exercise for the MUA
>> programmers, but  we've been thinking that it's likely to be a button
>> labelled "junk".
>

Well, maybe that's true. But have you talked to Twitter, or anyone else who 
has actually tried it. I haven't, but Twitter do provide two buttons: one 
to block and one to report. It's not exactly the same, but it's there.

Apart from that, we're not actually creating these buttons, are we? We're 
defining a reporting mechanism. My view is that it's better to have a 
richer vocabulary defined, and allow implementers to chose which parts to 
use.

In fact, we probably wouldn't be having this conversation (well, I 
wouldn't!) if a few more IMAP flags had been provided for these purposes.

> not-junk button to display when looking at stuff in a junk folder.

>  I suppose we could do that, but then we'd have to define what a junk
> _______________________________________________
> Asrg mailing list
> [email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 25 Feb 2010 18:04:19 +0100

>> suppose we could do that, but then we'd have to define what a junk

I don't think John meant a "general" definition here... :-/

> [...]

I cannot help distinguishing between IMAP and POP3 here. For IMAP, 
synchronization of Bayesian data among several servers and clients may 
be viewed as a generic distributed database problem, possibly 
complicated by an amount of fuzziness. It is possible to send an abuse 
report as a consequence of particular user's actions; it is just 
similar to "move marked junk to junk folder".

For POP3, there are no folders. That's what makes that definition 
difficult. Users can look for X-Spam-* headers only after they've 
already downloaded the message. In case servers maintain _per-user_ 
Bayesian data --as they should-- the whole idea of filtering on the 
servers seems rather pointless.

To recap, junk buttons can be embedded within a more sophisticated 
architecture (as for IMAP). But not the other way around: anti-spam 
filter training cannot (in general) be based upon junk buttons and 
abuse reporting.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 25 Feb 2010 15:28:48 -0500

> I don't think John meant a "general" definition here... :-/

John seemed to be implying you can't have a "non-junk" button without a 
junk folder (eg: in an IMAP sense).  I was just pointing out 
Thunderbird's "not junk" implementation, which is functional independent 
of the existence of any kind of foldering mechanism, IMAP or otherwise.

> similar to "move marked junk to junk folder".

It's an implementation _convenience_ (for IMAP).  Nothing more.

There is nothing preventing you sending your Bayesian data out-of-band 
to the server, nor, keeping it local for client-based filtering (as in 
Thunderbird).

Heck, SpamAssassin even manages to tune Bayesian without having any 
end-user feedback at all.

> For POP3, there are no folders.

Perhaps no server-visible folders, but there are folders none the less 
in most (if not all) POP3 reader implementations.

> That's what makes that definition
> already downloaded the message.

This is moot.  Most filters these days don't bother making "clearly 
spam" visible at all to the end user or mail reader, and the X-Spam- 
headers (as in the SpamAssassin sense) don't exist.  It's only in 
marginal cases where the user has to see the email to decide whether an 
email is spam or not that the X-Spam headers will be worth while. 
Which, in the case of POP3, means they have to download the message 
anyway to make that determination.

> In case servers maintain _per-user_

Of course you can train spam filters based on abuse reports.  We've been 
doing precisely that for 13 years in several different incarnations.

It may well make sense to include an "tickle IMAP" server as part of a 
spec, but, also having an abuse reporting mechanism makes sure that you 
have just about all implementations covered, IMAP or otherwise.

We could spec both, and leave it up to an installation or user to decide 
which (or both) to use in any particular instance.

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Thu, 25 Feb 2010 22:03:29 +0100

> On 2/25/2010 12:04 PM, Alessandro Vesely wrote:

...

> end-user feedback at all.

...

> which (or both) to use in any particular instance.

Hmmmm... IMHO, it isn't a good idea to restrict this work based on what one think "--as they 
should--" or "most ISP think"...

Bayesian filters aren't the only kind of statistical filters. There are many ways to think about 
spam. There are a lot of research being done in universities or research departements of private 
companies (sometimes presented at CEAS).

So, we should consider that the mechanism which is being specified here could be used in ways we 
don't know today with current MUAs and current filter technologies.

my 2 cents.

JM

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 26 Feb 2010 13:22:31 +0100

On 25/Feb/10 21:28, Chris Lewis wrote:
> John seemed to be implying you can't have a "non-junk" button without a junk folder (eg: in an IMAP sense).

That seems reasonable...

> I was just pointing out Thunderbird's "not junk" implementation, which is functional independent of the existence of any kind of foldering mechanism, IMAP or otherwise.

Yup. The JunQuilla extension makes that even more manageable. It is an 
interactive tool running on the end-user's box, though.

> Heck, SpamAssassin even manages to tune Bayesian without having any end-user feedback at all.

I never adventured into such esoteric settings. Are there howtos or 
any docs about it?

> doing precisely that for 13 years in several different incarnations.

Hm... I've been tinkering with my server's settings based on users' 
reports as well, but not automatically. There are various mechanisms, 
e.g. Vipul's Razor, that allow users to share their verdicts about the 
spamminess of a given message. However, in order to attach to junk 
buttons a meaning of "filter messages /like/ this" we would need to 
define what that means in rather unambiguous terms.

> which (or both) to use in any particular instance.

I'd lean toward specifying just how to deliver abuse reports. Neither 
junk buttons nor their color should be mandated.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 27 Feb 2010 01:22:01 -0500

> interactive tool running on the end-user's box, though.

It doesn't have to be.

> I never adventured into such esoteric settings. Are there howtos or

I think it's called "Autolearn".  I think it works by treating SA scores 
 > <threshold> as "spam", and scores < <possibly a different threshold> 
as "ham", and tunes Bayesian from that.  IOW: the existing SA rules 
refine Bayesian, and in the long term this allows Bayesian to 
cross-correlate across individual emails, and Bayesian score stuff that 
the SA rules don't necessarily even see.

> reports as well, but not automatically.

I've been doing it for years.

> There are various mechanisms,

No, you don't.  That's up to the implementer of the report handler what 
it does.

Just as it is with Bayes.

Why are you treating this any different than spam/ham training in Bayes? 
  It's no different.

> junk buttons nor their color should be mandated.

Who is trying to specify buttons or their color?

I'm aiming for a specification that permits a single <user action> to 
communicate upstream for _both_ filtering and reporting purposes, where 
whether it's used for filtering or reporting or both in any given 
instance is up to the site admin and/or end-user.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 27 Feb 2010 12:49:27 +0100

On 27/Feb/10 07:22, Chris Lewis wrote:
>> I never adventured into such esoteric settings. Are there howtos or any docs about it?
>
> I think it's called "Autolearn". I think it works by treating SA scores > <threshold> as "spam", and scores < <possibly a different threshold> as "ham", and tunes Bayesian from that. IOW: the existing SA rules refine Bayesian, and in the long term this allows Bayesian to cross-correlate across individual emails, and Bayesian score stuff that the SA rules don't necessarily even see.

I've found some explanation in 
http://spamassassinbook.packtpub.com/chapter9_preview.htm . My 
understanding is that auto-learn works against that gray area of 
uncertain cases. The book's author recommends not to rely on that 
feature alone. He notes that "Once a false positive occurs the 
Bayesian database will begin to lose effectiveness, and future 
Bayesian results will be compromised."

If we consider machine-learning for what it is, we must agree that 
current technology does not allow 'puters to understand human speech 
better that we do. Even though SA is able to classify a bunch of 
messages much better that an unmotivated human, it does not really 
/understand/ their contents. Therefore, it has to be trained by a 
(motivated) human, which implies interaction with users.

>> [...] in order to attach to junk buttons a meaning of "filter messages /like/ this" we would need to define what that means in rather unambiguous terms.
>
> No, you don't. That's up to the implementer of the report handler what it does.

I agree that it's up to abuse report consumers to state what they do.

A discussion about "generalized FBLs" will probably involve concerns 
about who is entitled to consume ARs, and may also consider whether 
consuming ARs implies any duty. Letting users know about any outcome, 
or letting them further modify any such outcome, are examples of 
possible duties. The less of them, the better.

For example, consider the last MX sending an AR to the 1st MX, as in 
the picture in http://wiki.asrg.sp.am/wiki/Abuse_Reporting . If the 
1st MX feeds the reported message to its Bayesian engine, then it 
should also allow forwarded users to check uncertain messages and 
correct any false positives. If it does not grant such interactivity, 
users may want to omit forwarding to it, in order to avoid losing mail.

Would it make sense to require that interactive filtering activity is 
limited to the last MX? Bandwidth-wise, it is counter-intuitive. It 
may be better to avoid this argument entirely.

> Why are you treating this any different than spam/ham training in Bayes? It's no different.

Potentially, it is orders of magnitude better than Bayes.

>> I'd lean toward specifying just how to deliver abuse reports. Neither junk buttons nor their color should be mandated.
>

I hope we won't. But it seems difficult to specify MUA's AR addresses 
without creating false expectations.

> I'm aiming for a specification that permits a single <user action> to communicate upstream for _both_ filtering and reporting purposes, where whether it's used for filtering or reporting or both in any given instance is up to the site admin and/or end-user.

+1, and I would welcome an efficient IMAP implementation in that 
sense. However, the spec should also allow to just send complaints.

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Sat, 27 Feb 2010 14:08:37 +0100

>> communicate upstream for _both_ filtering and reporting purposes, 
>> where whether it's used for filtering or reporting or both in any 
>> given instance is up to the site admin and/or end-user.
> 
> +1, and I would welcome an efficient IMAP implementation in that sense. 
> However, the spec should also allow to just send complaints.

+1. "upstream communication is the idea". It's up to the (email address|device|...) to interpret how 
to interpret it and what to do with.

--