Subject: [lemonade] draft-ietf-lemonade-imap-sieve-00

From: Barry Leiba <[email-masked]>
Date: Mon, 19 Dec 2005 00:13:53 -0500

I have just sent the attached draft to be published as an I-D.  It's a
first pass at describing an extension to IMAP and Sieve that could be
used to drive server-to-client notifications with a Sieve script.

Please review and comment here, and especially note the "open issues"
section.

Barry

--
Barry Leiba, Pervasive Computing Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba
http://www.research.ibm.com/spam
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Mon, 19 Dec 2005 15:42:55 +0000

> Please review and comment here, and especially note the "open 

There's quite a few mentions of atomic moves - these can be quite 
slow for some mailstores, and in general they're tricky, and they 
almost invariably involve maintaining locking on multiple mailboxes.

Moreover, I really do fear what a naïve client - or indeed any client 
- will do when it casually flips a flag, and discovers the message 
has just been expunged. Even if it's not naïve, but is the most 
intelligent and aware client in the world, I don't see how it's going 
to detect the difference between "this flag change caused the message 
to be atomically moved" and "this flag change didn't happen because 
the message was already expunged".

I also have a nasty feeling that some clients will take it upon 
themselves to make setting the \Deleted flag instantly expunge the 
message.

I also think it's likely to break UIDPLUS - a client will lose track 
of apparently witnessed moves, because it has no idea of what the new 
UID is, as well as not knowing if the message still exists somewhere 
else on the server. If even one message is moved in a MULTIAPPEND, 
then the APPENDUID response cannot be sent.

Section 3.8 suggests that even if the message isn't shifted about 
atomically behind the client's back, and UIDPLUS actually works, then 
because the headers may be edited, a client will potentially have the 
wrong data in the cache.

Oddly enough, I do think we could use reject, though. Reject could 
generate a protocol level error, hence an APPEND or COPY to a mailbox 
could be forced to fail for a fairly arbitrary reason, for instance. 
Of course, with APPEND and LITERAL+, that's going to make client 
implementors a little upset.

Really, I think this ought to limit itself to two actions:

1) Notifications, so that message appends, copies, and flag changes 
can be sent to disconnected devices.

2) Flag changes, primarily for interoperability between clients using 
locally defined keywords.

Otherwise, I'm really not sure how we're going to get all those worms 
back into the can.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Barry Leiba <[email-masked]>
Date: Tue, 20 Dec 2005 22:44:32 -0500

Dave says...

> almost invariably involve maintaining locking on multiple mailboxes.

I don't see that what I'm describing as an "atomic move" is any more of
a burden for servers than
   COPY 5 Newmailbox
   STORE 5 +Flags.Silent (\Deleted)
   EXPUNGE

What the spec means is that one of these "moves" must implement that
sequence as one operation, and what that mostly means is that if
something fails, none of it is done... and that the clients just see
the EXPUNGE responses, and not the intermediate FETCH FLAGS(\Deleted)
updates.

Do we prefer to just let servers implement these moves as those three
separate operations, and allow us to have intermediate states?  Or do
some agree with what Dave says later, that the moves aren't appropriate
at all (see my comments below)?

> Moreover, I really do fear what a naïve client - or indeed any
> client - will do when it casually flips a flag, and discovers the
> message has just been expunged.

The same thing it does when it sees any message expunged.  I remind you
that every client, nowever naïve or worldly, MUST deal with EXPUNGE
responses, and the underlying expungement (is that a word?) of the
messages.  Why should you think this should be a problem?  This can
happen in other situations, such as this:

Client 1                          Client 2
                                  C: X1 STORE 5 +Flags (\Deleted)
                                  S: * FETCH 5 Flags (\Deleted)
C: T1 STORE 5 +Flags (\Answered)
                                  S: X1 OK done
                                  C: X2 EXPUNGE
                                  S: * 5 EXPUNGE
S: T1 NO failed                   S: X2 OK done
C: T2 NOOP
S: * 5 EXPUNGE
S: T2 OK done

> most intelligent and aware client in the world, I don't see how it's
> going to detect the difference between "this flag change caused the
> message to be atomically moved" and "this flag change didn't happen
> because the message was already expunged".

I don't see why it has to detect the difference.  Can you explain?

> themselves to make setting the \Deleted flag instantly expunge the

This certainly doesn't give them license to do that.  If someone wants
to have a Sieve script do that, they certainly can, of course:

  if allof (${IMAPSieve.cause} :is "FLAG", hasflag "\\Deleted") {
      discard;
  }

Whether this is a good idea or not is something the implementer of the
script has to decide.

> then the APPENDUID response cannot be sent.

Huh?  This makes no sense.  Even if the server responds with a UID, the
message can disappear before the client can do anything with it anyway.
This automates things, perhaps making such situations more apt to
happen, but it doesn't change the fact that they can happen.

> because the headers may be edited, a client will potentially have the

That IS a good point, and I'd originally started to write it to ban
EditHeader, but then changed my mind.  What do other client-writers
think about this?

> could be forced to fail for a fairly arbitrary reason, for instance.

I hadn't thought about that.  Yes, we could allow/define the reject
action.  Other comments?

> implementors a little upset.

Using APPEND with LITERAL+ is a bad idea anyway, since the server can
always say NO to APPEND, without the client's being able to determine
that in advance.  Again, this doesn't change that.

> locally defined keywords.

Noted.  Do others agree?

Barry

---

From: Mark Crispin <[email-masked]>
Date: Tue, 20 Dec 2005 20:26:50 -0800

>   STORE 5 +Flags.Silent (\Deleted)

The operations that you describe are not a burden on servers.

Mandating that these three operations become one atomic operation is a 
burden.

An even greater burden is the implication that the "move" can take place 
without requiring additional storage equivalent to the sum of the 
message(s) being moved.  This implication has been in requests for a 
"move" operation since day 1.

This is a can of worms that I am loathe to see opened.

>> Moreover, I really do fear what a naive client - or indeed any
> that every client, nowever naive or worldly, MUST deal with EXPUNGE
> responses

The current IMAP specification guarantees that expunge events will not 
happen except at certain well-defined points.

In particular, a client is guaranteed that no EXPUNGE events will happen 
during or between a FETCH or STORE.

> S: T1 NO failed                   S: X2 OK done

This is a broken scenario.  As far as I am concerned, this is forbidden by 
the IMAP base specification, and that text to the contrary in RFC 2180 
sections 4.1.2 and 4.1.3 is wrong.  [Anyway, RFC 2180 is informational, 
not normative.]

The correct response to command T1 is:

* 5 FETCH (Flags (\Deleted \Answered))
T1 OK STORE completed (but this message is expunged, so it's gonna vanish)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 20 Dec 2005 23:39:57 -0500

> Mandating that these three operations become one atomic operation is
> a burden.

OK.  Well, as I said, the draft was a first pass.  We can change it in
one of two ways, so let's get opinions here.  What we're discussing
right now is what happens with existing messages (cause=FLAG and
cause=ANNOTATE) if the implicit KEEP is cancelled.

1. We can change the text to say that "fileinto" behaves as though the
Sieve script were a second client that did COPY/STORE/EXPUNGE, as
above, and that "discard" behaves as though it did "STORE/EXPUNGE".

2. We can change it to ban "discard" and NEVER cancel the implicit keep
(in which case "fileinto" just does a COPY).

...or we can hear other suggestions.  Yes?

> The current IMAP specification guarantees that expunge events will
> not happen except at certain well-defined points.

The current IMAP specification guarantees that EXPUNGE responses will
not happen except at certain well-defined points.

> This is a broken scenario.  As far as I am concerned, this is
> forbidden by the IMAP base specification

Well, I think you know that we disagree strongly here.  But I don't
think it matters, so let's not get into it again.  We should fix this
by deciding how to write it into the imap-sieve spec so that we're all
happy.  My "atomic" wording was something to start with; I'm happy to
change that to something else.

Barry

---

From: Dave Cridland <[email-masked]>
Date: Wed, 21 Dec 2005 14:47:13 +0000

On Wed Dec 21 03:44:32 2005, Barry Leiba wrote:
> something fails, none of it is done...

That's the burdensome bit.

> responses, and the underlying expungement (is that a word?) of the

I'm not sure that expungement is a word, but to a client, the phrase 
you're looking for is "irrevocable destruction without trace".

> S: T1 NO failed                   S: X2 OK done
Right. But that's not what's happening. It's what *looks* like what's 
happening, but it's not.

> I don't see why it has to detect the difference.  Can you explain?
The above case is a rare race condition, which isn't a big deal. 
(That is, it's a circumstance that arises by a combination of events 
happening in a precise timing, not a security flaw).

When hit send on this message, your message (or rather, the copy of 
it in my mailbox), will get flagged as answered by Polymer. If it 
happens to be deleted by another session, that's not a real worry - 
it's also pretty unlikely.

If that happens to every message, Polymer doesn't know what's 
happening, it just assumes that by fluke, some other session happens 
to have vapourized the message.

Polymer won't get paranoid, although the user might.

If Polymer wanted to both answer the message and forward it, however, 
it now can't - the act of answering the message has destroyed it 
forever. Of course, in fact, it might not have been destroyed, but 
Polymer cannot know that.

If some other client had cheerfully decided to automatically sweep 
read messages to an archival folder, that will instantly break things 
badly for Polymer, and people will think that Polymer is broken, and 
doesn't support attachments, since Polymer happens to be an online 
client relying on the implicit Seen.

You *could* add in protocol messages to indicate third-party COPY 
operations, but Polymer won't understand those, so that doesn't 
actually change the situation at all.

Neither of these are problems with Polymer, incidentally. You can 
subsitute any other client name you like.

> This certainly doesn't give them license to do that.

That's exactly what it does do. This grants full license to anyone 
and everyone to provide their very own favourite semantics for 
mailstores.

>   If someone wants
No, it's not. It's up to the one client author who doesn't like a two 
phase delete, and sticks an innocuous checkbox marked "Delete 
Immediately" somewhere.

Moreover, there are already server implementors who use precisely the 
arguments you do here to do immediate expunges of messages. This 
makes the argument altogether easier - "Well, *if* I supported X, and 
*if* there was a script Y, then... So it doesn't make a difference."

> happen, but it doesn't change the fact that they can happen.
Okay, so it can lie about the UIDs all the time, breaking the 
client's cache. That helps things in a Lemonade setting how, exactly? 
Clients will waste storage space until they happen to look at the 
mailbox and realize that UID doesn't exist anymore, never knowing 
that attempting to preserve the cache was a waste of time and 
resource.

In any case, I really dislike the notion of a server saying "OK, I 
did what you asked, here is the result", and the client later finding 
out that not only did the server lie about the result, but it did 
something completely different, which the client cannot find out 
about.

All this is not to say that the likes of fileinto and discard have no 
purpose for IMAP-level Sieve, just that they really shouldn't be 
implicit actions that the client might be unaware of. Running a Sieve 
script explicitly over a set of messages would be nice, and would 
allow for all sorts of useful behaviour, commonly handled by clients 
locally:

- It could be used to flag a set of messages matching complex 
criteria currently unexpressable in IMAP's SEARCH.
- It could be used to sweep old or irrelevent messages from the INBOX.
- etc.

The key thing here is "explicit" - the client wouldn't be taken by 
surprise here, and existing semantics aren't broken.

> EditHeader, but then changed my mind.  What do other client-writers
Without meaning to sound arrogant, it doesn't matter a jot what 
anyone thinks. Message content in IMAP is immutable. The only way to 
allow EditHeader would be to allocate a new UID for the message, 
which in turn destroys the cache.

These message moves and edits *do* destroy the cache. You're 
increasing the chances of a client downloading the same thing twice, 
wasting both a round trip *and* a hefty chunk of bandwidth. I don't 
see why this can possibly be considered a good thing.

> that in advance.  Again, this doesn't change that.
No, it most certainly does change this.

If a client understands ACL sufficient to handle MYRIGHTS, then a 
failed APPEND becomes very much an exceptional condition, and it can 
eliminate one round trip by using LITERAL+.

Moreover, failed APPENDs *are* exceptional anyway, given that they're 
almost always used in the message send scenario, and, in general, if 
an APPEND works once, it'll work again. Failed COPY is relatively 
common, but a failed APPEND is usually a sign of misconfiguration in 
real-world MUAs.

Of course, theoretically, all things are possible, but theory is only 
right in theory.

Dave.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 21 Dec 2005 11:05:52 -0500

Dave Cridland wrote:
> If Polymer wanted to both answer the message and forward it, however, it 
> now can't - the act of answering the message has destroyed it forever. 
> Of course, in fact, it might not have been destroyed, but Polymer cannot 
> know that.

Sieve scripts are meant to be user-defined things; if the user wants to
file \Answered messages away, and remove them from her inbox, I think the
user should be able to do that.

I suppose what it comes down to is that you're arguing that clients should
provide that function, and users who want it should choose clients that do.
And that the function shouldn't be available on the server, through Sieve
scripts.

That argument is fine, but for "diverse service environments", where we
might WANT some of this stuff done on the server.

When we get down to it, though, this document was written because of the
need to do notifications, so my thoughts about what else one might do with
these functions are speculative.  See below.

> That's exactly what it does do. This grants full license to anyone and 
> everyone to provide their very own favourite semantics for mailstores.

It does not, and I can't see how you can claim that.  But let's not get
into an "Is so! / Is not!" match.

> phase delete, and sticks an innocuous checkbox marked "Delete 

Oh, be real: a client author who wants to do that already can, by just
putting in the checkbox and putting code in that always sends EXPUNGE
whenever it sets the \Deleted flag.  The IMAP spec is quite clear about
that being a bad idea, and clients insist on implementing "Trash"
mailboxes anyway.  This doesn't change anything with respect to THAT.

> Okay, so it can lie about the UIDs all the time, breaking

I had thought that UIDPLUS allowed an "I don't know the UID" response;
as I look again, I see that it doesn't.  OK.

So we get back to the choices in my response to Mark, extended to
the "new message" case.  Again, see below.

> Running a Sieve script explicitly over a set of messages would be nice,
 > and would allow for all sorts of useful behaviour, commonly handled by
 > clients locally:

That's a good idea, which might be good to add here: an IMAP RUNSIEVE
command, with a corresponding cause=RUNSIEVE.  I like it.  Do others?
Shall I add it?

> Without meaning to sound arrogant, it doesn't matter a jot what anyone 
> thinks. Message content in IMAP is immutable. The only way to allow 
> EditHeader would be to allocate a new UID for the message, which in turn 
> destroys the cache.

Uh, we ARE assigning new UIDs, and we're NOT changing headers on existing
messages.  My draft explicitly says that the EditHeader actions are only
available for NEW MESSAGES, and so only for cause=APPEND and cause=COPY.

That said, you've brought up a point that I hadn't considered: that
clients expect the messages created by APPEND and COPY to be unchanged,
and that this will hurt their message cache.  It might be best, then,
to go with my original thought of banning EditHeader in all cases.
Any objections?

It's "below" now:
In my response to Mark, I suggested that we choose to change what happens
for cause=FLAG and cause=ANNOTATE in one of these ways (and we have to
decide):

1. Have the Sieve script actually behave as a client that has issued
the appropriate COPY/STORE/EXPUNGE commands (and, actually, we can
discuss whether to eliminate the EXPUNGE from there).

2. ALWAYS retain the implicit keep.

Let's expand that to the cases of cause=APPEND and cause=COPY, and say
that for the new messages created like that, we actually DO store the
message first (which gives the UIDPLUS response code), and THEN behave
as we choose, above, for existing messages.

I think this will take care of most of the concerns that Dave has raised,
though, of course, there's still the issue of wasted effort and resource
in the client if we choose option 1.  I'm more and more in favour of
saying that for all of the cases described in this document, we retain
the implicit keep... EXCEPT that we allow the "reject" action, which
gives a protocol-level NO to APPEND/MULTIAPPEND/COPY, and stores nothing.

Barry

---

From: Anil SRIVASTAVA <[email-masked]>
Date: Wed, 21 Dec 2005 09:10:39 -0800 (Pacific Standard Time)

On 2005-12-21/11:05 [-0500], [email-masked] [Barry Leiba] wrote:

> 
> So we get back to the choices in my response to Mark, extended to the 
> "new message" case.  Again, see below.
> 
> > Running a Sieve script explicitly over a set of messages would be 
> > nice, and would allow for all sorts of useful behaviour, commonly 
> > handled by clients locally:
> 

That is something we have had customers ask for quite often.  This is a 
function most clients support and there are legitimate use cases for 
such a capability.  

+1

> Uh, we ARE assigning new UIDs, and we're NOT changing headers on 
> existing messages.  My draft explicitly says that the EditHeader 
> actions are only available for NEW MESSAGES, and so only for 
> cause=APPEND and cause=COPY.
> 
> clients expect the messages created by APPEND and COPY to be 
> unchanged, and that this will hurt their message cache.  It might be 
> best, then, to go with my original thought of banning EditHeader in 
> all cases. Any objections?
> 

I have to agree with Dave's original assertion that allowing for headers 
to be edited is a bad thing so banning EditHeader sounds like the right 
thing to do.

> 
> It's "below" now:
> In my response to Mark, I suggested that we choose to change what 
> happens for cause=FLAG and cause=ANNOTATE in one of these ways (and we 
> have to decide):
> 

retain the implicit keep.

Anil

> I think this will take care of most of the concerns that Dave has 
> raised, though, of course, there's still the issue of wasted effort 
> and resource in the client if we choose option 1.  I'm more and more 
> in favour of saying that for all of the cases described in this 
> document, we retain the implicit keep... EXCEPT that we allow the 
> "reject" action, which gives a protocol-level NO to 
> APPEND/MULTIAPPEND/COPY, and stores nothing.
> 
> Barry
> 
> --

_______________
Anil SRIVASTAVA
[email-masked]

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 21 Dec 2005 12:27:49 -0500

Hi Barry,

> command, with a corresponding cause=RUNSIEVE.  I like it.  Do others?

Actually I have to say that despite my initial enthusiasm for the idea of 
incorporating sieve into IMAP, given the complexity we already have and all 
the arguments about simplifying things and not doing ACAP in IMAP etc, I am 
now leaning towards not using sieve.

Instead, what I would propose for clients being able to 'run scripts', is a 
combination of the following:

1) Enhance the SEARCH command with better comparators (globbing, regex etc)

2) Introduce a 'pipe' operation that will allow the result set of the last 
(or named) search command to be fed into a subsequent fetch/store. I think 
there was already a proposal for that.

3) Introduce a 'macro' facility that would allow clients to define commonly 
used commands via a keyword and store those on the server, and then execute 
the commands via the keyword. This would allow for 'saved searches' - 
frequently used search commands stored on the server that can be quickly 
executed without having to uplaod the search each time. Anyone who has 
written a client that attempts to do traditional mua filtering using IMAP 
search will appreciate the benefit of something like that. These 'macros' 
could execute more than one command at a time with some limitations. e.g. a 
UID SEARCH followed by a UID COPY, followed by a UID STORE, followed by a 
UID expunge would emulate an mua filter action that could move messages 
matching certain criteria to another mailbox.

These three things combined together can do the majority of what is needed, 
and won't require clients/servers to have sieve script handling capability 
- just a few (pretty basic) extensions to the IMAP protocol.

That then leaves the question of whether it makes sense to use sieve for 
notifications. I would argue not - if its just for notifications. Lets 
define another IMAP extension for setting notification information. Maybe 
it results in something that is 'script-like' and if so perhaps SIEVE 
syntax could be borrowed for that, or maybe we stick with parenthesised 
lists as being more IMAP-like.

-- 
Cyrus Daboo

---

From: Dave Cridland <[email-masked]>
Date: Wed, 21 Dec 2005 18:10:29 +0000

On Wed Dec 21 16:05:52 2005, Barry Leiba wrote:
> Dave Cridland wrote:
>> If Polymer wanted to both answer the message and forward it, 
>> however, it now can't - the act of answering the message has 
>> destroyed it forever. Of course, in fact, it might not have been 
>> destroyed, but Polymer cannot know that.
> 
Yes. Out of curiosity, do you think all IMAP clients send the message 
first, then flag as \Answered? Because if they don't, and they happen 
to reference the message after flagging, perhaps via BURL, perhaps 
not, for body parts, things might get a little confused if setting 
\Answered causes the message to be destroyed.

> And that the function shouldn't be available on the server, through 
Yes, but not because "that function shouldn't be available on the 
server", but because IMAP servers should remain IMAP servers, and I 
honestly think this breaks behaviour, because it allows different 
clients to effectively define arbitrary semantics for flags which the 
server then manages, in potentially confusing ways. I think allowing 
message destruction as a automatic byproduct retrofitted onto FLAG 
changes is opening a container full of worms, let alone a can. I 
really cannot stress this enough.

The APPEND/COPY automatic message destruction is more palatable to 
me, but the only use for it I can actually see is the notion of 
retrofitting complex filtering and searching[1], which in my mind 
would be better handled explicitly, and I think there's at least a 
economy sized can of worms here too. I very much doubt that I'll ever 
be comfortable with this.

To give a concrete example, I actually really like the notion of 
telling a naïve client to store sent items in a folder which has an 
APPEND/COPY Sieve script which relocates the message elsewhere, 
depending on who it's sent to - I think it's a compelling use-case. 
The only problem is that if a client then chooses to assemble the 
message via CATENATE in that folder, quite innocently, then it'll 
never find it again, so in practise a neat hack turns into serious 
breakage. Which is a shame, as I think that's a reasonable thing for 
a client to try doing, and worse, I cannot think of a workaround that 
will interoperate with Lemonade Profile 1 clients.

That's not a "concern", for me, that's a complete showstopper.

> might WANT some of this stuff done on the server.
Automated filing of older messages sounds cool to me. Hence an 
explicit RUNSIEVE command, fully supported by the client.

> these functions are speculative.  See below.
Sure. In general, I think you're on to something good. I just think 
you're being a little *too* speculative. :-)

Let's just stick with notifications, and optionally flag changes, for 
now at least. I think reject needs a little more thought - I'm not 
convinced it's a good idea. For a start, I realised that the 
sync/non-sync literal problem doesn't even register - a Sieve script 
is likely to want the entire message before deciding to reject it.

Fileinto/discard really do scare the hell out of me.

> into an "Is so! / Is not!" match.
Aw. We could do "Oh, yes, it is! / Oh, no, it isn't!" if you're up 
for a bit of Pantomime, it is the season, after all.

>> No, it's not. It's up to the one client author who doesn't like a 
>> two phase delete, and sticks an innocuous checkbox marked "Delete 
>> Immediately" somewhere.
> 
No, it means that one client author can not only instruct the server 
to change the semantics for them, but impose their views on everybody 
else. That's quite a major change, I think.

> as I look again, I see that it doesn't.  OK.
Mark's newer draft does allow ommission of the UIDPLUS responses 
(COPYUID and APPENDUID), but only, IIRC, in the case where the 
mailbox declares itself as having non-sticky UIDs. In principle, I'd 
be fine with dropping the response even so, but there's no method for 
ommitting a single UID, in the case of MULTIAPPEND/COPY.

>> anyone thinks. Message content in IMAP is immutable. The only way 
>> to allow EditHeader would be to allocate a new UID for the 
>> message, which in turn destroys the cache.
> 
> existing
You misunderstood, and reading my own writing, I'm not surprised. I 
meant allocate a new UID for the message compared with the one 
generated by the normal APPEND/COPY mechanism and returned by 
UIDPLUS. It doesn't help that I don't think of APPEND as operating on 
a new message - messages are only copied in IMAP, in my mind, 
creation happens outside its scope.

> unchanged,

Exactly. Hence where EditHeader is used, the clients see the message 
instantly destroyed, exactly like your fileinto/discard proposals.

[[
1. Searching with Sieve, a imap-sieve hack: Create a fresh mailbox. 
Add a Sieve script which discards messages you don't want on COPY. Go 
through all source mailboxes you fancy, performing COPY 1:*. Finally, 
open the fresh mailbox and see what you find. It's much more powerful 
than the inbuilt IMAP SEARCH, and allows full "virtual scrolling" 
through the results.
]]

Dave.

---

From: Curtis King <[email-masked]>
Date: Thu, 22 Dec 2005 12:41:22 -0800

On 21-Dec-05, at 9:27 AM, Cyrus Daboo wrote:

> These three things combined together can do the majority of what is  
> needed, and won't require clients/servers to have sieve script  
> handling capability - just a few (pretty basic) extensions to the  
> IMAP protocol.
>
> That then leaves the question of whether it makes sense to use  
> sieve for notifications. I would argue not - if its just for  
> notifications. Lets define another IMAP extension for setting  
> notification information. Maybe it results in something that is  
> 'script-like' and if so perhaps SIEVE syntax could be borrowed for  
> that, or maybe we stick with parenthesised lists as being more IMAP- 
> like.

I agree with Cyrus and this is the best solution to date. Anyone else  
want to move in this direction?

ck

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 11:29:58 -0800

On 23-Dec-05, at 5:06 AM, Arnaud Taddei wrote:

> Curtis King wrote:
>>> These three things combined together can do the majority of what  
>>> is  needed, and won't require clients/servers to have sieve  
>>> script  handling capability - just a few (pretty basic)  
>>> extensions to the  IMAP protocol.
>>>
>>> 'script-like' and if so perhaps SIEVE syntax could be borrowed  
>>> for  that, or maybe we stick with parenthesised lists as being  
>>> more IMAP- like.
>>>
>>
>> I agree with Cyrus and this is the best solution to date. Anyone  
>> else  want to move in this direction?
>>
> Isn't it what Chris Newman is working on these days?
>

Yes, and is an other good reason to consider Cyrus idea, only one  
type of notifications. Does any else think it's better to have the  
more general solution of extending SEARCH, adding a pipe operation  
and macros then incorporating sieve into IMAP? I like it because not  
only does it provide the functionality of incorporating sieve, but  
also improves IMAP as a whole and allows the servers to cache results  
for the macros.

ck

---

From: Mark Crispin <[email-masked]>
Date: Fri, 23 Dec 2005 12:01:14 -0800 (PST)

On Fri, 23 Dec 2005, Curtis King wrote:
> Yes, and is an other good reason to consider Cyrus idea, only one type of 
> notifications. Does any else think it's better to have the more general 
> solution of extending SEARCH, adding a pipe operation and macros then 
> incorporating sieve into IMAP? I like it because not only does it provide the 
> functionality of incorporating sieve, but also improves IMAP as a whole and 
> allows the servers to cache results for the macros.

I think that it is a terrible idea to add any more bloat to IMAP.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 12:09:06 -0800

On 23-Dec-05, at 12:01 PM, Mark Crispin wrote:

>> and macros then incorporating sieve into IMAP? I like it because  
>> not only does it provide the functionality of incorporating sieve,  
>> but also improves IMAP as a whole and allows the servers to cache  
>> results for the macros.
>

Adding sieve isn't adding more bloat? There is already suggest to add  
a RUNSIEVE command.

ck

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 12:22:04 -0800

On 23-Dec-05, at 11:44 AM, Barry Leiba wrote:

> This would then argue for pulling notifications out of Sieve
> altogether, and doing that function by using IMAP notifications on the
> mailboxes.  The only thing I worry about there is that we make sure  
> the
> notification mechanism is independent of any client actions, because
> they have to happen whether there's a client connected or not.  Which
> means they can't happen as the direct result of a SEARCH command --  
> but
> could be done by having a persistent SEARCH that the server handles
> thereafter.

I agree and to me either sieve or persistent SEARCH is the same  
amount of work. I see persistent SEARCH being useful outside of  
notifications or I maybe missing something. The question I'm  
struggling with which one would be a cleaner design?

ck

---

From: Mark Crispin <[email-masked]>
Date: Fri, 23 Dec 2005 12:35:37 -0800 (PST)

On Fri, 23 Dec 2005, Curtis King wrote:
> Adding sieve isn't adding more bloat? There is already suggest to add a 
> RUNSIEVE command.

Opposing one form of bloat does not constitute support for another form of 
bloat.

The tendency to take the easy path of "add just one more new feature to 
IMAP" is something that should be vigorously resisted.

-- Mark --

http://panda.com/mrc

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 13:37:53 -0800

On 23-Dec-05, at 12:35 PM, Mark Crispin wrote:

>> Adding sieve isn't adding more bloat? There is already suggest to  
>> add a RUNSIEVE command.
>
> Opposing one form of bloat does not constitute support for another  
> form of bloat.
>
> The tendency to take the easy path of "add just one more new  
> feature to IMAP" is something that should be vigorously resisted.

So, I gather your not a fun of either sieve or macros ;) I thought it  
was an either/or discussion.

The simplest way to get notifications is have a new notification  
protocol which IMAP servers, clients, sieve scripts, and out-band  
push notifications can all use. This is simple to implement, no  
extensions needed to IMAP, can reuse existing notification  
technologies, etc. I do believe you and Lyndon have suggested this  
before. I'm very curious why there is no interest in implementing  
such a service?

ck

---

From: Mark Crispin <[email-masked]>
Date: Fri, 23 Dec 2005 14:19:24 -0800 (PST)

On Fri, 23 Dec 2005, Curtis King wrote:
> The simplest way to get notifications is have a new notification protocol 
> which IMAP servers, clients, sieve scripts, and out-band push notifications 
> can all use. This is simple to implement, no extensions needed to IMAP, can 
> reuse existing notification technologies, etc.

I agree.

> I do believe you and Lyndon 
> have suggested this before. I'm very curious why there is no interest in 
> implementing such a service?

I am interested in implementing such a service.

In 2002, I even implemented a prototype of such a service.  It was a nice, 
sweet protocol, easy to implement.  What's more, it had working code.

I'm not the only person to have done something like this, either.

The problem is that every time someone tries to produce a write up of such 
an effort, everybody immediately starts making the spurious "if <X> 
protocol does not have <Y> functionality, then <X> will die" argument to 
press to add everything but the kitchen sink.  Each time this happens, the 
guy who did the work decides "I don't need this headache" and drops it.

This is, sadly, a systemic problem throughout the IETF.

-- Mark --

http://panda.com/mrc

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 14:52:40 -0800

On 23-Dec-05, at 2:19 PM, Mark Crispin wrote:

>
> In 2002, I even implemented a prototype of such a service.  It was  
> a nice, sweet protocol, easy to implement.  What's more, it had  
> working code.
>

+1

It's how I implemented IDLE. (btw large phone controls systems use a  
very simple event notification service)

>
> The problem is that every time someone tries to produce a write up  
> of such an effort, everybody immediately starts making the spurious  
> "if <X> protocol does not have <Y> functionality, then <X> will  
> die" argument to press to add everything but the kitchen sink.   
> Each time this happens, the guy who did the work decides "I don't  
> need this headache" and drops it.

For what is worth, I would support such a simple protocol as it  
becomes possible to handle all the various race conditions. Just  
refer draft-ietf-lemonade-imap-sieve-00 looking for atomic operations  
to some of them. I found there is no limit on functionality it just  
get implemented in the client instead of the protocol. There is the  
add bonus for this working group of being able to addresses one of  
big issues, push notifications very quickly. Isn't that worth something?

ck

---

From: Dave Cridland <[email-masked]>
Date: Fri, 23 Dec 2005 23:11:18 +0000

On Fri Dec 23 22:19:24 2005, Mark Crispin wrote:
>> technologies, etc.
Yes, same. The tricky bit is that IMAP servers need to implement 
this, when the mail server package as a whole may already support 
Sieve, and there are also multiple Sieve implementations available 
easily.

Arguably, adding some form of Sieve support to IMAP - possibly 
including adding Sieve test or scripts as a SEARCH criterion - 
effectively adds one item of bloat to IMAP, and passes the entire 
problem onto Sieve implementors, who're working hard and actively to 
develop Sieve, and solve many of the problems that IMAP would benefit 
from being solved. (I phrase this in this awkward way to avoid you 
thinking I mean "IMAP will die..." - because it won't. But it would 
be better.)

To give an example, I've yet to find a conclusive, efficient, method 
of searching for messages with parts of a particular type without 
searching for strings likely to be found in the type's content, and 
then recutting the search locally - that's painful. The better 
alternative is to use Sieve, but that's only available at the point 
of delivery, so the method ends up being "tag mails with a user 
defined flag on delivery, hope that this is done before we actually 
need to find those parts". I'm not a huge fan of methods which 
include the word "hope".

Extending SEARCH to allow for Sieve would solve this, and every issue 
Cyrus raised in his point 1.

That all said, I think the interesting uses of Sieve are in 
post-delivery filtering and SEARCH, and not notifications.

>> I do believe you and Lyndon have suggested this before. I'm very 
>> curious why there is no interest in implementing such a service?
> 
I'm sort of. I think that developing a generalized notification 
service is tricky. I think that developing a method for the IMAP 
server to be made to issue notifications (on IDLE or NOOP) might be 
possible.

> I'm not the only person to have done something like this, either.
The outline you kindly sent me in private mail had one line in it 
that greatly interested me.

You said, and I'm not quoting, something along the lines of using 
URLAUTH as an authentication-by-proxy for notification access.

I mentioned this in passing to Philip Guenther, and between us (but 
mostly Philip), we arrived at the following concept: [Note this uses 
Magic in the approved sense of non-IETF]

1) Client requests from GENURLAUTH some form of URL not currently 
supported, such as SEARCH URLs, mailbox URLs, LIST URLs, etc. Server 
signs these and returns.

2) Client hands these via Magical Means to a Magic Notification 
Server. The interesting thing here is that said Magical Means might 
be SMS, WAP, or TCP. (Mark's implementation is TCP, in fact).

3) Said Magic Notification Server then hands these to IMAP server, 
using them to request notifications. These notifications are 
content-free, much like existing in-band IMAP notifications. Should a 
client which finally receives, or at least responds to, these 
notifications require content, it must connect, authenticate, and 
retrieve the content normally.

This means that the nasty hideous part of notifications - that is, 
actually getting the notifications to the client - is actually placed 
neatly well out of scope of the servers. Thus a Magic Notification 
Server could be provided by a mobile network operator, or a 
mailserver ISV. Or even both, covering different actualy notification 
protocols.

The IMAP extensions needed are fairly minimal - yes, we're extending 
the protocol, but we're not spending as much time extending the 
software required.

> This is, sadly, a systemic problem throughout the IETF.

No, there's a worse systemic problem, but that involves the creation 
of new protocols within the IETF. I think what you describe is merely 
a minor symptom of it. But that's another story for another time.

For now, Merry Christmas for those that intend getting merry, Happy 
Yule for those willing to be happy, and best wishes for the new year, 
for those that actually have a new year at roughly this time.

Dave.

---

From: Curtis King <[email-masked]>
Date: Fri, 23 Dec 2005 16:52:04 -0800

> Sieve, and there are also multiple Sieve implementations available  

But, you still need to extend sieve to support notifications. I think  
imap-sieve is overloaded by providing a notification service. Instead  
it should reference the notification service being both a generator  
and consumer of events just like IMAP servers and clients.

ck

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 26 Dec 2005 16:02:31 +0100

> struggling with which one would be a cleaner design?

We implemented persistent searches and found it clean and pleasant. 
Surprisingly so - suddenly we sat there looking at each other asking 
«that's it? we're done?»

A draft's sort of ready, all it needs is a references section.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Dec 2005 11:56:08 -0800 (PST)

On Mon, 26 Dec 2005, Arnt Gulbrandsen wrote:
> $B+(Bthat's it? we're done?$B;(B

You're talking about Alexey's draft-melnikov-imap-search-res proposal, 
right?

-- Mark --

http://panda.com/mrc

---

From: Arnaud Taddei <[email-masked]>
Date: Fri, 23 Dec 2005 14:06:38 +0100

Curtis King wrote:

>> That then leaves the question of whether it makes sense to use  sieve 
>> for notifications. I would argue not - if its just for  

A++

-- 
<http://www.sun.com> 	* Arnaud Taddei *
Senior Architect Communications Products

*Sun Microsystems, Inc.*
Phone x13541/+41(22)99904-41
Mobile +41 (0)79 506 1129
Email [email-masked]

---

From: Barry Leiba <[email-masked]>
Date: Fri, 23 Dec 2005 14:44:52 -0500

This would then argue for pulling notifications out of Sieve
altogether, and doing that function by using IMAP notifications on the
mailboxes.  The only thing I worry about there is that we make sure the
notification mechanism is independent of any client actions, because
they have to happen whether there's a client connected or not.  Which
means they can't happen as the direct result of a SEARCH command -- but
could be done by having a persistent SEARCH that the server handles
thereafter.

Barry

---

From: "Stephane H. Maes" <[email-masked]>
Date: Tue, 27 Dec 2005 05:59:18 -0800

I think we may be mixing how notification are integrated with IMAP versus how they can be bound to the event and generated via standard scripts.

I am working on generating the S2C notification draft to address the first part.

The current draft focuses on how SIEVE can be used to realize the binding that generates the notifications.

Stephane
_____
Stephane H. Maes, PhD,
Director of Architecture, Mobile, Voice, and Communications Platforms, Oracle.
Ph: +1-203-300-7786 (mobile/SMS); Fax / Office UM: +1-650-607-6296
e-mail: [email-masked]
IM: shmaes (AIM/Y!/skype) or [email-masked] (MSN Messenger)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Friday, December 23, 2005 11:45 AM
To: Curtis King; Arnaud Taddei
Cc: Cyrus Daboo; Enhancements to Internet email to support diverse service enivronments; Chris Newman
Subject: Re: [lemonade] draft-ietf-lemonade-imap-sieve-00

Barry

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 27 Dec 2005 23:49:29 +0100

Mark Crispin writes:
> You're talking about Alexey's draft-melnikov-imap-search-res proposal, right?

No, about a new one of mine. Would you describe search-res as 
"persistent searches"?

(Btw, PLEASE fix your MUA. You wrote RFC 1469 - if anyone can comply 
with it, you can.)

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Tue, 27 Dec 2005 15:14:21 -0800 (Pacific Standard Time)

On Tue, 27 Dec 2005, Arnt Gulbrandsen wrote:
> No, about a new one of mine. Would you describe search-res as "persistent 
> searches"?

I didn't say that.  I'm trying to figure out what you are talking about. 
If it is not search-res, then what is it and what are its characteristics 
that make it better than search-res?

> (Btw, PLEASE fix your MUA. You wrote RFC 1469 - if anyone can comply with it, 
> you can.)

I did not write "IP Multicast over Token-Ring Local Area Networks"; you 
must be talking about RFC 1468.

The message left my MUA in correct ISO-2022-JP (and yes, my MUA did 
translate them).  Something downstream mangled it.

In any case, you should stick to ASCII when sending messages to an 
international forum.

-- Mark --

---

From: "Stephane H. Maes" <[email-masked]>
Date: Wed, 28 Dec 2005 00:14:13 +0000 GMT

Vfolder is based on permanent search a la psearch from pimap.

Stephane
_____
Stephane H. Maes, PhD,
Director of Architecture, Mobile, Voice, and Communications Platforms, Oracle
Ph: +1-203-300-7786 (mobile/SMS/skype); Fax / Office UM: +1-650-607-6296.
e-mail: [email-masked]
IM: shmaes (AIM/Y!) or [email-masked] (MSN Messenger)

-----Original Message-----
From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 27 Dec 2005 23:49:29 
To:[email-masked]
Cc:Mark Crispin <[email-masked]>, Barry Leiba <[email-masked]>
Subject: Re: [lemonade] draft-ietf-lemonade-imap-sieve-00

Arnt

---

From: Curtis King <[email-masked]>
Date: Thu, 29 Dec 2005 07:59:17 -0800

On 27-Dec-05, at 5:59 AM, Stephane H. Maes wrote:

>
> I think we may be mixing how notification are integrated with IMAP  
> versus how they can be bound to the event and generated via  
> standard scripts.

True, but would be nice to have the referenced draft before the  
referring draft or I'm being too picky?

> I am working on generating the S2C notification draft to address  

The notifcation protocol should be seperate from IMAP for the single  
reason other protocols (SMTP/LMTP) need to generate events as well.

ck

---

From: Curtis King <[email-masked]>
Date: Thu, 29 Dec 2005 08:04:05 -0800

> No, about a new one of mine. Would you describe search-res as  

It sure covers part of the problem space.

ck

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Dec 2005 21:26:41 +0000

> Surprisingly so - suddenly we sat there looking at each other asking
Persistent searches are also quite easy client-side, and don't have a 
huge cost. Persistent SORT and THREAD are not, though, and nor are 
server-side enumerations.

> A draft's sort of ready, all it needs is a references section.
I'll look forward to it.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 30 Dec 2005 19:51:27 +0100

Dave Cridland writes:
> huge cost.

I'd say they're easy if all clients are kept in sync somehow, e.g. by 
using only a single client or by using only clients that store the 
search in something like ACAP. That's too limited from my point of 
view.

(There is a difference in server-side cost, btw. It's a little complex 
and depends on what extensions are used and what clients access the 
mailboxes in question. With search-res, condstore and either no 
primitive clients or no big mailboxes, client-side works well. Again, 
I'm not wild about that.

>> A draft's sort of ready, all it needs is a references section.

Future tense? I blush. Mea culpa. I've left too many people looking 
forward to things I should do in the past three months and a day.

Arnt