Subject: Re: WGLC on IMAP Sieve (draft-ietf-lemonade-imap-sieve)

From: Alexey Melnikov <[email-masked]>
Date: Fri, 23 May 2008 17:14:51 +0400

Alexey Melnikov wrote:
> While this is a Lemonade WG document, this document is of relevance to 
> the Sieve WG mailing list and should be reviewed here.
>
> So please review 
> <http://www.ietf.org/internet-drafts/draft-ietf-lemonade-imap-sieve-05.txt> 
> before May 5th.
Folks,
I haven't seen any reviews of this. Can you please send me a quick "I've 
reviewed it and it looks fine" or "this needs more work" to me, if 
you've done a review.

Thanks,
Alexey

---

From: Ned Freed <[email-masked]>
Date: Fri, 23 May 2008 08:40:57 -0700 (PDT)

> reviewed it and it looks fine" or "this needs more work" to me, if

I finally had a chance to look at this. Having done so, the question I really
want to ask is has anyone implemented this, and if they  did, how well did it
work? And if the answer to that is "no implementations yet", I'd then like to
hear if anyone is planning to implement this, and if so, when and for what
puropse?

The general problem I see here is that Sieve is specified to operate in a
fairly narrow context: At or around the time of final delivery. This has
influenced Sieve design choices in subtle ways and, now that we have such an
extensive body of work on Sieve-related stuff, tracking down all of the
implicit assumptions of context and reworking them so that Sieve can operate in
the context of the message store is very difficult. There are almost certainly
going to be things we miss or get wrong, and I think implementation experience
is about the only way we're going to find all the impedance mismatches.

Having said all this, I don't know how it translates into a plan of action for
the document. The obvious thing is to do is approve it as experimental, but
this may have the opposite of the indended effect - it may cause the
specification to be ignored because it is "not ready' for standardization.
OTOH, if it approved as proposed but then requires substantive changes, we
could easily end up in a situation with large differences between
implementations. That's not good either.

Now, obviously if someone has implemented this without difficulty then these
concerns are baseless. But if they haven't, I'd like to know to what extent
there are plans to code this, so we have some idea of how to proceed.

Moving on to specifics... Overall the document seems basically OK. I don't see
fundamental problems that would invalidate the entire approach. That said,
there are quite a few details I think need a bit of work.

First up is the interactions with annotations. I can see the value of being
able to get at annotations through Sieve. The trouble is there seems to be a
missing piece: I don't believe a Sieve extension has been defined that get at
annotation data. GIven that that's the case I'm not sure I see the point of the
changedannotation environment item. All this is going to tell you is what
annotations changed - it doesn't tell you whether they were added, or deleted,
or modified, or what's in them if they are still there. Unless there's either
an extension in the works for getting at annotation information (in which case
it needs to be referenced) or there's a use-case for knowing what annotations
have changed (in which case there should be an exmaple of that usage
somewhere), I have to recommend removing the changedannotations environment
item. If need be it can be defined in a subsequent Sieve annotate extension
specification.

Next up is the imap4flags extension. Flag handling is such a key part of IMAP I
really have to wonder how useful this extension is without imap4flags support.
I wonder if it doesn't make more sense to simply require imap4flags support -
given the way imap4flags can be used without variables it isn't especially
difficult to implement. It would certainly simplify the document by removing
lots of qualifiers all over the place.

The imap4flags extension essentially provides two different ways to manipulate
flags: With variables or with the so-called default flag internal variable.
Given some of the examples it appears that the default flag internal variable
is supposed to be set to the current flags on the emssage before the script is
run. I don't believe this is explicitly stated in the document, and it needs to
be. (This document should also reiterate that changes to the default flags
attach to the messages produced by the actions the sieve performs and the final
state of the default flags apply to the implicit keep.)

I note in passing that RFC 5232 doesn't actually say what the initial setting
of the default flags is supposed to be. Empty seems like the only choice for
regular Sieve use and we should spell that out in the next update.

And now for one of those subtle issues: Flag value inspection and manipulation.
A fairly common thing for scripts to do is make copies of their inputs so they
can refer back to the original values independent of any modifications the
script makes. (This is especially common practice in large scripts built up
from various component pieces from different sources.) This is a nonissue
for the imap4flags extension in normal usage: The initial setting of the
default flag internal value is empty. But it isn't empty at startup in this
new usage context. And unless I've missed something, there's no way to capture
the state of this internal variable. The best you can do is grab the first
flag in the set:

    if hasflag :matches "*" {set "doof" "${0}";}

Nowhere near good enough. Or, if you know the names of all the flags you're
ever going to be interested in, you can do:

    setflag "initialflags" "";
    if hasflag :is "flag1" {addflag "initialflags" "flag1";}
    if hasflag :is "flag2" {addflag "initialflags" "flag1";}
    ...
    if hasflag :is "flagn" {addflag "initialflags" "flagn";}

But this is extremely fragile, not to mention verbose and tacky.

The obvious way to address this is to add an additional environment item:
initialflags. This makes it easy to, say, reset the default flags to their
initial settings:

    if environment :matches "initialflags" "*" {setflag "${0}";}

Moving on... redirect. Redirect assumes that the message in hand is suitable
for submission. This is a reasonable assumption in normal Sieve use because the
message is either in transit or has just been in transit. However, I don't
believe, say, IMAP APPEND imposes sufficient requirements on messages for them
to be considered ready for submission. The draft needs to deal with this issue,
but I'm not exactly sure how - perhaps an explicit reference to the SMTP submit
specification plus some statement that if the message is unsuitable for
submission a DSN SHOULD be generated.

Next up is editheader and another subtle but extremely important issue. The
draft quite correctly bans editheader modifications to the message at hand
because such modifications  would violate IMAP semantics. However, this ignores
the extremely important usage of editheader as a meanss of annotating
redirected messages, where no such semantics problem exists. For example, it is
quite common to use editheader to prevent redirection loops:

        /* Don't redirect if we already redirected */
        if not header :contains "X-Sieve-Filtered"
                ["<[email-masked]>", "<[email-masked]>"]
        {
                addheader "X-Sieve-Filtered" "<[email-masked]>";
                redirect "[email-masked]";
        }

(This example is straigth from the editheader draft itself.) Given the added
risks of loops created by this document I really don't want to lose this
capability.

For that matter I'm not sure I see the harm in allowing:

        keep;
        addheader "comment" "extra copy for posterity"
	fileinto "archive";

Unless I'm missing something there should be no reason why an additional
copy of the message created with fileinto has to be identical to the
original.

And there's also the use of editheader as a kind of variable surrogate:

	if <something nasty and complex>
	{ addheader "flag" "whatever";}
	...
	if header :is "flag" "whatever" ...
	...
	deleteheader "flag";

I think this is a bit screwy and of course unnecessary in any implementation
that support variables, but I've seen several customer scripts do it. (And yes,
we support variables.)

I therefore think the outright ban on editheader needs to be lifted and
replaced with someething more nuanced, e.g., editheader, if used, has no
effect on implicit or expliccit keep, but can affect other actions.

Next up is spamtest and virustest. Neither is mentioned in the draft, and IMO
this is a SERIOUS omission. I can say with absolute certainty that one of the
main things, if not the main thing, our customers are going to want to do in
sieves attached to mailboxes: Check appended messages for spam and viruses. And
since headers cannot be used to communicate spam status because of message
immuatability, that leves spamtest and virustest as the only games in town.

And that segres into the next to last item on my list: reject. I think I
understand the reasons for wanting to ban it outright, but this is another one
that just isn't going to play in Peoria. Like it or not, people are going to
want to use this facility to impose access restrictions on IMAP APPEND, and
they are going to be pissed if they can't do it. In fact this is one that
unless there's a fundamental reason we cannot implement this in our store by
havingt it turn a reject into a NO response to APPEND I will recommend that we
simply ignore this restriciton and allow reject.

One final subtle issue. Another thing I'm fairly confident people will want to
do with Sieve in IMAP is implement finer grained access control. (When you
think about it, access control is one of Sieve's main purposes.) But unlike
final delivery, where there's no identity associated with the entity performing
the delivery, there's always an identity associated with someone doing IMAP
stuff. Accordingly, Sievce in IMAP needs to have a way to access this
identity so it can use it as part of its access control checks. THe obvious
way to do it is through another environment item, e.g., user-identifier. So 
you could say stuff like:

    if allof(environment :is "cause" ["append", "copy"],
             not environment :is "user-identifier" "ned",
	     size :over 64K {
      reject "Sorry, I don't allow random bozos to append big messages to my mailbox";
    }

That's it for now, but if nothing else I hope this review has driven home how
easy it is for these impedance mismatch issues to creep in unobserved.

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Sat, 24 May 2008 00:33:49 +0100

> hear if anyone is planning to implement this, and if so, when and  
The whole thing just leaves me a quivering mess, whimpering in a  
corner and yearning for the good old days, when if one APPENDed  
message literal in a MULTAPPEND set was accepted, one could breathe a  
sigh of relief and use LITERAL+ to do heavy pipelining on the rest.

I'm still not entirely sure how IMAP-SIEVE would signal *which*  
message failed in a MULTIAPPEND set, actually - I did look, and  
Section 2.2.2's coverage of this particular problem is, well,  
exceedingly economic.

But this is a specific issue, rather than the more general problem.

> is about the only way we're going to find all the impedance  
I have to admit to having had some concern over the document since  
its inception, but I've really been unable to pin down exactly what  
was bothering me about it - but in general terms it was indeed the  
unspoken assumptions.

The scary thing is that what you describe is a whole class of  
assumption I hadn't even considered - I was merely considering the  
class of assumption that IMAP client authors - who are hard enough to  
persuade into doing any actual IMAP client authoring, frankly, at the  
best of times - might well have made, and probably will wish to  
continue making, over time. A halfway predictable message store would  
be one of them, speaking personally.

Another class of assumption is that IMAP server developers are used  
to the requirements that certain operations are atomic, and running  
an arbitrary SIEVE script atomically doesn't quite fit into this  
frame of thought. Still, there are worse things, such as repeatedly  
stabbing myself in the navel with a rusty fork, and I suppose I  
shouldn't complain, since in as much as I can see, there is no  
requirement for self-mutilation anywhere in the specification. Nor,  
though, is there any mention of this in, say, Section 2.2.2, where it  
talks about the MULTIAPPEND case. (I was looking for a discussion of  
atomicity, not fork-stabbing, in case that's not clear. I did have a  
quick scan for self-mutilation, but aside from a brief mention of  
ManageSIEVE, there's nothing).

Uploading viruses, or uploading messages over a certain size, are  
both sensible things to want to prevent - but I have to admit to  
struggling to understand why an administrator would prefer to use a  
complicated script instead of a simple configuration setting,  
especially given that the complicated script will get executed for  
every flag change, too. Yay, go efficiency.

This ignores the interesting cases of whether an implementation  
ignores, for the purposes of Sieve, events caused by Sieve scripts,  
since otherwise two mailboxes could have conflicting scripts that  
cheerfully bounced an APPENDed message backwards and forwards for all  
eternity. The Security Considerations section clearly states that  
implementations might choose to do so, maybe, but only for  
flagchanges on Tuesdays in Summer during non-leap years.

There may very well be useful cases for this extension, above and  
beyond bringing on nervous breakdowns in IMAP implementors, hence the  
reason I'm obviously trying to appear positive about it, but these  
are not only likely to be quite rare and uncommon, but if they're  
not, then better solutions are undoubtedly possible.

So, to get back to your original question, I have not yet implemented  
it, and I don't think we have any imminent plans to do so, but I  
cannot speak for my employer in this regard.

I hope this helps answer your question.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 24 May 2008 11:24:35 +0400

Dave Cridland wrote:
> message failed in a MULTIAPPEND set, actually
Dave, I don't understand how is this different from MULTIAPPEND without 
IMAP Sieve: an IMAP Sieve script can't "fail" append. It can only 
discard message after it gets appended (see section 3.6).
> - I did look, and Section 2.2.2's coverage of this particular problem 
> is, well, exceedingly economic.
>
 [...]
> eternity.
Sieve fileinto caused by IMAP Sieve is not the same as IMAP COPY, but in 
order to avoid any doubts I agree that the Security Considerations 
should also mention that.
> The Security Considerations section clearly states that 
> implementations might choose to do so, maybe, but only for flagchanges 
> on Tuesdays in Summer during non-leap years.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 24 May 2008 11:28:43 +0200

I looked at imap-sieve long ago and decided that I wanted no part of it.

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 24 May 2008 13:53:56 +0400

> I looked at imap-sieve long ago and decided that I wanted no part of it.
Can you elaborate why?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 24 May 2008 16:00:25 +0200

> Can you elaborate why?

Dave summed it up better than I could.

I realise there are use cases for some, most or perhaps all of 
imap-sieve. But the use cases aren't all there is: there's also an 
apparently infinite number of nasty scary cases.

In case it's not obvious, I would prefer that lemonade profile-bis avoid 
any reference to imap-sieve.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Sun, 25 May 2008 15:17:27 -0700 (PDT)

> discard message after it gets appended (see section 3.6).

And as I said in my comments, I view this as a significant omission that needs
to be corrected. As for the MULTIAPPEND issue, well, I've already commented on
what I view to be the inadequacy of IMAP error reporting in other contexts.

> should also mention that.

I agree with this as far it goes, but it caused me to review the text on
fileinto in the base Sieve specification as well as in the IMAP sieve document,
and I'm afraid what turned up is another impedance mismatch that needs to be
addressed.

The base spec basically says that

   fileinto "foo";

causes the message to be placed in a mailbox named "foo". It then goes on to
discuss the access control issues this can cause, especially in the case where
the mailbox doesn't already exist. Conspiciously absent from any of this is
discussion of mailbox hierarchy - what, say,

   fileinto "foo/bar"; 

means is entirely implementation dependent. And if this case is interesting,
something like

   fileinto "/foo/bar";

has even more interesting implications. The reason for not nailing any of this
down is of course that Sieve does not presuppose a particular store backend of
any kind.

Also implicit in this discussion is that the sieve in question has an
identifiable owner that has access rights to the store. Also, in practice the
argument of fileinto is also interpreted in the context of where the user's
default mailbox is located - foo is normally placed at the same level in the
hierarchy as their inbox.

Now consider how this changes when Sieves are used in IMAP. Now we're dealing
with a specific sort of store with very specific semantics. So now these
semantics can and should be nailed down, but the draft currently fails to do
so.

And then there's the access control issue. Presumably if a script is attached
to a user mailbox is "owned" by that user and inherits his or her access
rights. (If so, this also needs to be explicitly stated.)

But what about scripts attached to public mailboxes, ones that have no owner?
Who owns them and what rights do such scripts have?

And what about the server script? Who owns it and what rights does it have?

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 26 May 2008 07:24:09 +0100

Ned Freed wrote:

> to be corrected.

Possibly (I haven't made my mind on this yet).
I would just point out that your change would create an extra failure 
case that isn't present in IMAP Sieve now.

> As for the MULTIAPPEND issue, well, I've already commented on

This is solvable by defining a new response code (or a new response).

> semantics can and should be nailed down, but the draft currently fails 

Maybe so. I am not entirely clear what needs to be written to clarify that.

Isode's implementation just treats the fileinto argument as an IMAP 
mailbox name. So if it contains a shared folder prefix, this would cause 
mail to be filed into the shared folder.

> rights. (If so, this also needs to be explicitly stated.)

Right.

> And what about the server script? Who owns it and what rights does it 

These are all interesting questions. I agree that they should be discussed.