Subject: Resending David Harris' Keyword Comments
From: Lyndon Nerenberg - VE6BBM/VE7TFX <[email-masked]>
To: [email-masked], [email-masked]
Date: Thu, 27 Aug 2009 16:52:10 -0600
List-Id: <ietf-imapext.imc.org>

[ I think the first part was truncated in the original message. ]

> First off, I assume we're talking about "keywords" as defined in 
> RFC3501 section 2.3.2 - or at least, I use the word "defined" loosely, 
> because the word "keyword" just sort of appears out of nowhere in that 
> section without any definition at all, and the BNF doesn't amplify the 
> situation much.
> 
> My client is Pegasus Mail - it has had IMAP support for about 14 years 
> now, but that support never been very advanced or ambitious, mostly 
> because in the very early days I ran into so many inconsistencies with 
> server implementations that I never felt it was safe to use anything 
> other than the most basic functions of the protocol (and even then I've 
> had my share of problems). Keywords are one of the things I've always 
> avoided, simply because in the early stages, I found that some servers 
> either did not support client-defined keywords, or defined them in 
> variable ways that made them difficult to use. As a general rule, you 
> can't hang an entire piece of functionality on a feature that might or 
> might not be implemented in the server, so it was best simply to avoid 
> keywords altogether. Instead, I check for specific message headers 
> and allow the user to define others. My client code is very heavily 
> cached, so the overhead of examining the headers is only a one-time 
> hit in most cases, and a hit that has to be done for other reasons in 
> any event.
> 
> [ As an aside, I've just re-read the section of RFC3501 on keywords 
> and find that it's still server-optional whether they can be client-defined, 
> which effectively renders them all but useless, in my view. ]
> 
> Unfortunately, one of the problems with a very long legacy is that 
> decisions made early in the life cycle can be hard to reverse at a later 
> stage. I'm currently engaged in a total rewrite of my message store 
> code: some of the code there is nearly 20 years old, and it's *really* 
> showing its age, so however painful, a total reappraisal and 
> redevelopment using more modern techniques and tools has become 
> unavoidable. As part of the rewrite, my IMAP code is getting rewritten 
> and much more compartmentalized allowing it to be much freer in how 
> it goes about doing a wide range of tasks. I expect this to allow me to 
> use facilities like keywords if it seems appropriate, and to make it 
> much easier to add support for such facilities if it becomes appropriate 
> in future.
> 
>> > I'd also really like to know if the client has "this is spam" or "this 
>> > is not spam" buttons in the UI, and if it does how those buttons 
>> > interact with the use of keywords.
> 
> Pegasus Mail has various spam-fighting facilities, including a regex-
> based content processing engine and a full Bayesian spam filter. 
> Identifying a message as spam is simply a matter of moving it into the 
> spam folder. Similarly, indicating that a message that is in the spam 
> folder is not spam is simply a matter of moving it out of the folder to 
> any other location. There is also a clickable status button that can be 
> used for in-place evaluation and reclassification, although our 
> experience is that most users are more at home with the "move-
> in/move-out" paradigm. None of the built-in anti-spam features has any 
> interaction with any IMAP keyword at any level, although it would be 
> very convenient if there were one I could use reliably.
> 
>> > I ask this because users of any client that has such buttons are clearly 
>> > going to use those buttons preferentially over any other labelling 
>> > mechanism. And if a significant number of clients have such buttons that 
>> > do a particular thing - regardless of what that thing is - that's pretty 
>> > much going to be determinative in finding the most easily deployed 
>> > approach.
> 
> Speaking purely as a client developer (since I develop both client and 
> server IMAP code), if there were a formally-defined keyword that I 
> could reliably expect to find on all server implementations, then I'd 
> definitely use it, but otherwise I won't go near it. It's really a pretty 
> simple equation for me - IMAP is only a part of my message store 
> strategy, and I can't shape the entire way I work on the basis of what it 
> does and does not support (I suspect this may be a fairly common 
> conundrum for client developers, but have no hard evidence of that).

To: [email-masked], [email-masked]
subject: [Redux] Spam-related keyword usage survey
From: Lyndon Nerenberg - VE6BBM/VE7TFX  <[email-masked]>
Organization: The Frobozz Magic Homing Pigeon Company
Date: Thu, 27 Aug 2009 16:44:24 -0600
Sender: [email-masked]

Oops -- I spoke too soon.  There were valid replies, but my summary
missed them due to a corrupted mail folder screwing up my results
search.  My apologies to the two Davids!  Their responses are included
below (along with Mark's server commentary). 

* Client: Pegasus
* From: [email-masked]

   Unfortunately, one of the problems with a very long legacy is that
   decisions made early in the life cycle can be hard to reverse at a later
   stage.  I'm currently engaged in a total rewrite of my message store code:
   some of the code there is nearly 20 years old, and it's *really* showing
   its age, so however painful, a total reappraisal and redevelopment using
   more modern techniques and tools has become unavoidable.  As part of the
   rewrite, my IMAP code is getting rewritten and much more compartmentalized
   allowing it to be much freer in how it goes about doing a wide range of
   tasks.  I expect this to allow me to use facilities like keywords if it
   seems appropriate, and to make it much easier to add support for such
   facilities if it becomes appropriate in future.

   > > I'd also really like to know if the client has "this is spam" or "this
   > > is not spam" buttons in the UI, and if it does how those buttons > >
   interact with the use of keywords.

   Pegasus Mail has various spam-fighting facilities, including a regex- based
   content processing engine and a full Bayesian spam filter.  Identifying a
   message as spam is simply a matter of moving it into the spam folder.
   Similarly, indicating that a message that is in the spam folder is not spam
   is simply a matter of moving it out of the folder to any other location.
   There is also a clickable status button that can be used for in-place
   evaluation and reclassification, although our experience is that most users
   are more at home with the "move- in/move-out" paradigm.  None of the
   built-in anti-spam features has any interaction with any IMAP keyword at
   any level, although it would be very convenient if there were one I could
   use reliably.

   > > I ask this because users of any client that has such buttons are
   clearly > > going to use those buttons preferentially over any other
   labelling > > mechanism.  And if a significant number of clients have such
   buttons that > > do a particular thing - regardless of what that thing is -
   that's pretty > > much going to be determinative in finding the most easily
   deployed > > approach.

   Speaking purely as a client developer (since I develop both client and
   server IMAP code), if there were a formally-defined keyword that I could
   reliably expect to find on all server implementations, then I'd definitely
   use it, but otherwise I won't go near it.  It's really a pretty simple
   equation for me - IMAP is only a part of my message store strategy, and I
   can't shape the entire way I work on the basis of what it does and does not
   support (I suspect this may be a fairly common conundrum for client
   developers, but have no hard evidence of that).

* Client: Thunderbird
From: [email-masked]

   We store Junk/NonJunk keywords when either the user or the bayesian junk
   filter have determined a message to be spam/not spam.  We treat any message
   with a keyword containing the strings "NonJunk" or "NotJunk" (for mail.app)
   as a spam message, and if we don't find those strings in the keywords, if
   we find "Junk" in any keyword, we treat the message as spam.  So we'll
   treat $Junk as junk, for example.  We do a case-insensitive find on the
   sub-strings.

* Server: M*Guardian
* From: [email-masked]

   This isn't done by an IMAP client, but it is exposed to IMAP clients by our
   product.

   We use the following keywords in M+Guardian.  We quarantine both incoming
   and outgoing.  The content and attachment quarantines are based upon site
   configurable policy.

   $MPlusSpam	quarantined due to spam

   $MPlusVirus	quarantined due to virus

   $MPlusContent quarantined due to text content

   $MPlusAttach	quarantined due to attachment

   $MPlusOutbound	outbound vs.  inbound message (e.g., you sent
			    something that you aren't allowed to send)

Date: Mon, 24 Aug 2009 21:12:25 -0600 (MDT)
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked], [email-masked]
Subject: Survey Results for Spam-related keyword usage by IMAP clients.
Organization: The Frobozz Magic Homing Pigeon Company
Sender: [email-masked]

I realize this is a bit later than promised, but I was holding out hope 
for (more) responses.

The results were underwhelming. Two unofficial reports of client 
behaviour, and a note from Mark describing how their server products look 
at spam-related keywords.

Apparently SPAM isn't an issue after all.

I'll let Mark elaborate about his stuff if he thinks it's relevant. The 
rest of the info was anecdotal and shouldn't go into the record.

--lyndon

To: [email-masked], [email-masked]
Subject: Resending David Harris' Keyword Comments
From: Lyndon Nerenberg - VE6BBM/VE7TFX  <[email-masked]>
Organization: The Frobozz Magic Homing Pigeon Company
Date: Thu, 27 Aug 2009 16:52:10 -0600
Sender: [email-masked]

[ I think the first part was truncated in the original message. ]

> First off, I assume we're talking about "keywords" as defined in 
> RFC3501 section 2.3.2 - or at least, I use the word "defined" loosely, 
> because the word "keyword" just sort of appears out of nowhere in that 
> section without any definition at all, and the BNF doesn't amplify the 
> situation much.
> 
> My client is Pegasus Mail - it has had IMAP support for about 14 years 
> now, but that support never been very advanced or ambitious, mostly 
> because in the very early days I ran into so many inconsistencies with 
> server implementations that I never felt it was safe to use anything 
> other than the most basic functions of the protocol (and even then I've 
> had my share of problems). Keywords are one of the things I've always 
> avoided, simply because in the early stages, I found that some servers 
> either did not support client-defined keywords, or defined them in 
> variable ways that made them difficult to use. As a general rule, you 
> can't hang an entire piece of functionality on a feature that might or 
> might not be implemented in the server, so it was best simply to avoid 
> keywords altogether. Instead, I check for specific message headers 
> and allow the user to define others. My client code is very heavily 
> cached, so the overhead of examining the headers is only a one-time 
> hit in most cases, and a hit that has to be done for other reasons in 
> any event.
> 
> [ As an aside, I've just re-read the section of RFC3501 on keywords 
> and find that it's still server-optional whether they can be client-defined, 
> which effectively renders them all but useless, in my view. ]
> 
> Unfortunately, one of the problems with a very long legacy is that 
> decisions made early in the life cycle can be hard to reverse at a later 
> stage. I'm currently engaged in a total rewrite of my message store 
> code: some of the code there is nearly 20 years old, and it's *really* 
> showing its age, so however painful, a total reappraisal and 
> redevelopment using more modern techniques and tools has become 
> unavoidable. As part of the rewrite, my IMAP code is getting rewritten 
> and much more compartmentalized allowing it to be much freer in how 
> it goes about doing a wide range of tasks. I expect this to allow me to 
> use facilities like keywords if it seems appropriate, and to make it 
> much easier to add support for such facilities if it becomes appropriate 
> in future.
> 
>> > I'd also really like to know if the client has "this is spam" or "this 
>> > is not spam" buttons in the UI, and if it does how those buttons 
>> > interact with the use of keywords.
> 
> Pegasus Mail has various spam-fighting facilities, including a regex-
> based content processing engine and a full Bayesian spam filter. 
> Identifying a message as spam is simply a matter of moving it into the 
> spam folder. Similarly, indicating that a message that is in the spam 
> folder is not spam is simply a matter of moving it out of the folder to 
> any other location. There is also a clickable status button that can be 
> used for in-place evaluation and reclassification, although our 
> experience is that most users are more at home with the "move-
> in/move-out" paradigm. None of the built-in anti-spam features has any 
> interaction with any IMAP keyword at any level, although it would be 
> very convenient if there were one I could use reliably.
> 
>> > I ask this because users of any client that has such buttons are clearly 
>> > going to use those buttons preferentially over any other labelling 
>> > mechanism. And if a significant number of clients have such buttons that 
>> > do a particular thing - regardless of what that thing is - that's pretty 
>> > much going to be determinative in finding the most easily deployed 
>> > approach.
> 
> Speaking purely as a client developer (since I develop both client and 
> server IMAP code), if there were a formally-defined keyword that I 
> could reliably expect to find on all server implementations, then I'd 
> definitely use it, but otherwise I won't go near it. It's really a pretty 
> simple equation for me - IMAP is only a part of my message store 
> strategy, and I can't shape the entire way I work on the basis of what it 
> does and does not support (I suspect this may be a fairly common 
> conundrum for client developers, but have no hard evidence of that).

To: [email-masked], [email-masked]
subject: [Redux] Spam-related keyword usage survey
From: Lyndon Nerenberg - VE6BBM/VE7TFX  <[email-masked]>
Organization: The Frobozz Magic Homing Pigeon Company
Date: Thu, 27 Aug 2009 16:44:24 -0600
Sender: [email-masked]

Oops -- I spoke too soon.  There were valid replies, but my summary
missed them due to a corrupted mail folder screwing up my results
search.  My apologies to the two Davids!  Their responses are included
below (along with Mark's server commentary). 

* Client: Pegasus
* From: [email-masked]

   Unfortunately, one of the problems with a very long legacy is that
   decisions made early in the life cycle can be hard to reverse at a later
   stage.  I'm currently engaged in a total rewrite of my message store code:
   some of the code there is nearly 20 years old, and it's *really* showing
   its age, so however painful, a total reappraisal and redevelopment using
   more modern techniques and tools has become unavoidable.  As part of the
   rewrite, my IMAP code is getting rewritten and much more compartmentalized
   allowing it to be much freer in how it goes about doing a wide range of
   tasks.  I expect this to allow me to use facilities like keywords if it
   seems appropriate, and to make it much easier to add support for such
   facilities if it becomes appropriate in future.

   > > I'd also really like to know if the client has "this is spam" or "this
   > > is not spam" buttons in the UI, and if it does how those buttons > >
   interact with the use of keywords.

   Pegasus Mail has various spam-fighting facilities, including a regex- based
   content processing engine and a full Bayesian spam filter.  Identifying a
   message as spam is simply a matter of moving it into the spam folder.
   Similarly, indicating that a message that is in the spam folder is not spam
   is simply a matter of moving it out of the folder to any other location.
   There is also a clickable status button that can be used for in-place
   evaluation and reclassification, although our experience is that most users
   are more at home with the "move- in/move-out" paradigm.  None of the
   built-in anti-spam features has any interaction with any IMAP keyword at
   any level, although it would be very convenient if there were one I could
   use reliably.

   > > I ask this because users of any client that has such buttons are
   clearly > > going to use those buttons preferentially over any other
   labelling > > mechanism.  And if a significant number of clients have such
   buttons that > > do a particular thing - regardless of what that thing is -
   that's pretty > > much going to be determinative in finding the most easily
   deployed > > approach.

   Speaking purely as a client developer (since I develop both client and
   server IMAP code), if there were a formally-defined keyword that I could
   reliably expect to find on all server implementations, then I'd definitely
   use it, but otherwise I won't go near it.  It's really a pretty simple
   equation for me - IMAP is only a part of my message store strategy, and I
   can't shape the entire way I work on the basis of what it does and does not
   support (I suspect this may be a fairly common conundrum for client
   developers, but have no hard evidence of that).

* Client: Thunderbird
From: [email-masked]

   We store Junk/NonJunk keywords when either the user or the bayesian junk
   filter have determined a message to be spam/not spam.  We treat any message
   with a keyword containing the strings "NonJunk" or "NotJunk" (for mail.app)
   as a spam message, and if we don't find those strings in the keywords, if
   we find "Junk" in any keyword, we treat the message as spam.  So we'll
   treat $Junk as junk, for example.  We do a case-insensitive find on the
   sub-strings.

* Server: M*Guardian
* From: [email-masked]

   This isn't done by an IMAP client, but it is exposed to IMAP clients by our
   product.

   We use the following keywords in M+Guardian.  We quarantine both incoming
   and outgoing.  The content and attachment quarantines are based upon site
   configurable policy.

   $MPlusSpam	quarantined due to spam

   $MPlusVirus	quarantined due to virus

   $MPlusContent quarantined due to text content

   $MPlusAttach	quarantined due to attachment

   $MPlusOutbound	outbound vs.  inbound message (e.g., you sent
			    something that you aren't allowed to send)

Date: Mon, 24 Aug 2009 21:12:25 -0600 (MDT)
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked], [email-masked]
Subject: Survey Results for Spam-related keyword usage by IMAP clients.
Organization: The Frobozz Magic Homing Pigeon Company
Sender: [email-masked]

I realize this is a bit later than promised, but I was holding out hope 
for (more) responses.

The results were underwhelming. Two unofficial reports of client 
behaviour, and a note from Mark describing how their server products look 
at spam-related keywords.

Apparently SPAM isn't an issue after all.

I'll let Mark elaborate about his stuff if he thinks it's relevant. The 
rest of the info was anecdotal and shouldn't go into the record.

--lyndon