Subject: [sieve] Poll: how to report Sieve runtime errors to the user?

From: Stephan Bosch <[email-masked]>
Date: Tue, 24 Aug 2010 12:10:47 +0200

Hi,

I attended the last meeting in Maastricht through XMPP.  As I explained 
there, I feel that something is currently missing in the Sieve world: a 
standard means of handling and reporting runtime errors to the user. It 
was suggested to ask around how people currently implement this and, 
possibly, write up a best-practices document. I promised to perform this 
task, since I was the one coining the issue in the first place. Now I 
finally have the time to put this on the list.

Concretely, I'm talking about the implementation of the following 
requirement in RFC5228, at the end of Section 2.10.6:

"When an error happens, implementations MUST notify the user that an 
error occurred and which actions (if any) were taken, and do an implicit 
keep."

The RFC doesn't specify how this is to be implemented and, particularly 
for virtual users, this is an interesting design problem. Therefore, I 
would like to invite implementors on this list to report their current 
approach.  We can then discuss the benefits and limitations of the 
various reported implementations and exchange ideas on possible 
alternatives.

The reason I brought this up is that my own implementation is currently 
very limited and unsuitable for virtual users: it writes an error log 
file in the directory where the main active script is located (typically 
in the user home directory).

The alternative approaches/ideas I've seen/had thus far include:

- disable the offending script and e-mail the script owner to report 
that there was an error, and that their script is disabled until they go 
and fix it (Barry)
- file a message containing the error report, e.g. in a special IMAP 
folder, for each failed delivery. Clearly, this can explode into a large 
number of messages when the script is fundamentally broken. This needs 
some mechanism to avoid reporting the identical problem for subsequent 
deliveries.
- extend ManageSieve with a means to get notified about runtime errors 
and a means to retrieve (and possibly configure) runtime logs.

Regards,

Stephan

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 24 Aug 2010 06:15:49 -0700

On Aug 24, 2010, at 3:10 AM, Stephan Bosch wrote:

> Hi,
> 
> I attended the last meeting in Maastricht through XMPP.  As I explained there, I feel that something is currently missing in the Sieve world: a standard means of handling and reporting runtime errors to the user. It was suggested to ask around how people currently implement this and, possibly, write up a best-practices document. I promised to perform this task, since I was the one coining the issue in the first place. Now I finally have the time to put this on the list.
> 
> Concretely, I'm talking about the implementation of the following requirement in RFC5228, at the end of Section 2.10.6:
> 
> "When an error happens, implementations MUST notify the user that an error occurred and which actions (if any) were taken, and do an implicit keep."

Personally, the first part (before the "and") is an improper MUST IMO as it fails to take into consideration the various cases in which an implementation cannot notify the user.  (It also not terribly clear what "user" is being referred to here.  If a email domain administrator installed script fails, does the error notice go the email domain administrator, system operator, the owner of the email account, or combination thereof?)

I think one could argue that "implicit keep"ing the message is itself a form of notification.  For instance, if I have a rule that does a fileinto that cannot be processed (e.g., mbox doesn't exist), having the message stored in my inbox can be an effective notification that an error occurred in the processing the fileinto.

I would suggest that upon revision of this spec, "MUST notify the user ... and do an implicit keep" be changed to "SHOULD notify the script author ... and MUST, if possible, do an implicit keep".

> The RFC doesn't specify how this is to be implemented and, particularly for virtual users, this is an interesting design problem. Therefore, I would like to invite implementors on this list to report their current approach.  We can then discuss the benefits and limitations of the various reported implementations and exchange ideas on possible alternatives.
> 
> The reason I brought this up is that my own implementation is currently very limited and unsuitable for virtual users: it writes an error log file in the directory where the main active script is located (typically in the user home directory).
> 

 - log the error, expose logs to script authors.

And for notification, maybe in addition to "implicit keep", add a special header to the message.  But personally "implicit keep" (or bounce if not keepable) is enough notification for me.

> 
> - disable the offending script and e-mail the script owner to report that there was an error, and that their script is disabled until they go and fix it (Barry)

Yikes!  The script might just have a run-time error hit only when a particular condition is true.  Disabling the entire script would be a bad thing.  I argue the script should be left in force.

> - file a message containing the error report, e.g. in a special IMAP folder, for each failed delivery. Clearly, this can explode into a large number of messages when the script is fundamentally broken. This needs some mechanism to avoid reporting the identical problem for subsequent deliveries.

If the error is "disk full", likely the special folder will also be "disk full"... (and the implicit keep will also likely fail).  Log and bounce.

> - extend ManageSieve with a means to get notified about runtime errors and a means to retrieve (and possibly configure) runtime logs.

> 
> Regards,
> 
> Stephan
> 
> 
> 
> 
> 
> _______________________________________________
> sieve mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sieve

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 24 Aug 2010 16:05:29 +0200

Hi Kurt,

Op 24-8-2010 15:15, Kurt Zeilenga wrote:
>> <...>
> Personally, the first part (before the "and") is an improper MUST IMO as it fails to take into consideration the various cases in which an implementation cannot notify the user.

Could you provide examples of such cases?

> (It also not terribly clear what "user" is being referred to here.  If a email domain administrator installed script fails, does the error notice go the email domain administrator, system operator, the owner of the email account, or combination thereof?)

Good question. My current approach is to report the error to whomever 
has or could have caused it and/or should fix it.

> I think one could argue that "implicit keep"ing the message is itself a form of notification.  For instance, if I have a rule that does a fileinto that cannot be processed (e.g., mbox doesn't exist…

Right, but then the act of finding the cause of the problem remains an 
issue. It would be quite cumbersome if users need to ask (bug) the 
administrator for assistance when they produce bugs in their own Sieve 
scripts. Some form of access to the actual error messages for the users 
is desirable at least.

> And for notification, maybe in addition to "implicit keep", add a special header to the message.  But personally "implicit keep" (or bounce if not keepable) is enough notification for me.

Ok, this pertains to my remark above. How do you 'expose the logs to 
script authors'. A web interface of some sorts? Would your proposed 
header contain the error message or just a status flag?

> Yikes!  The script might just have a run-time error hit only when a particular condition is true.  Disabling the entire script would be a bad thing.  I argue the script should be left in force.

If I remember correctly, this strategy was followed for 'fatal' errors 
only. Not sure what is considered a fatal error in this case. I think 
Barry can explain this in more detail.

> If the error is "disk full", likely the special folder will also be "disk full"... (and the implicit keep will also likely fail).  Log and bounce.

Good point, but this is a special case that can be handled separately 
(and if it is not quota-related it cannot be solved by the user anyway).

Regards,

Stephan.

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 24 Aug 2010 07:47:27 -0700

> Could you provide examples of such cases?

The cases would depend upon the particulars of the mechanism the implementation used to notify the users.  For instance, if an email server's sieve engine notified by email, that email may simply not be deliverable (or sendable).

> Good question. My current approach is to report the error to whomever has or could have caused it and/or should fix it.

But this can be hard to determine.  For instance, consider a script installed on behalf of a user versus a script installed by that user.  The sieve engine cannot necessarily distinguish these two cases.  This argues for a method for specifying in the script how notifications are to handled.

> Right, but then the act of finding the cause of the problem remains an issue. It would be quite cumbersome if users need to ask (bug) the administrator for assistance when they produce bugs in their own Sieve scripts. Some form of access to the actual error messages for the users is desirable at least.

> Ok, this pertains to my remark above. How do you 'expose the logs to script authors'. A web interface of some sorts? Would your proposed header contain the error message or just a status flag?

I'd suggest the header include indication of the nature of the error and an optional URI where additional information about the error may be found.

> If I remember correctly, this strategy was followed for 'fatal' errors only. Not sure what is considered a fatal error in this case. I think Barry can explain this in more detail.

Off hand, the only case where I can think it appropriate to disable the script is when the script would no longer be acceptable as a whole, such as require statement failing at run time (which is possible due to changes in engine support for extensions). 

> Good point, but this is a special case that can be handled separately (and if it is not quota-related it cannot be solved by the user anyway).

---

From: [email-masked]
Date: Tue, 24 Aug 2010 07:35:21 -0700 (PDT)

> finally have the time to put this on the list.

It is important to keep in mind that Sieve errors are, or should be, pretty
rare in practice. A lot of Sieves - in our case virtually all of then - are
automatically generated, and an automatic generator should not be generating
broken Sieves.

> in the user home directory).

I don't know what a "virtual user" is or why being one makes error handling
difficult, but FWIW, in our implementation every Sieve script has an associated
"responsible address". For a user Sieve this is simply the user's own address,
for Sieves associated with an entire domain it's the domain postmaster, for
system Sieves it's an administrator address. If there's an error, it's reported
by sending a message to the responsible address, whatever it may be. Such
messages are themselves exempted from Sieve processing, which prevents loops.

> that there was an error, and that their script is disabled until they go

Disabling the Sieve in the event of an error isn't as good an idea as it may
first appear. For a Sieve syntax error that renders the entire Sieve
nonfunctional, maybe, but given that syntactic validity can and should be
checked before storing the Sieve, such errors should be quite rare.

The errors that slip through tend to be ones that are conditional in some way
on the message being processed. In such cases disabling the script entirely
prevents it from being applied to messages that don't trip the error condition.
It's easy for this to turn into a cure that's much much worse than the disease.
In fact this behavior could be exploited as part of some sort of attack.

> some mechanism to avoid reporting the identical problem for subsequent

You're seriously overstating this risk. See above.

> and a means to retrieve (and possibly configure) runtime logs.

Unless the user's MUA does regular queries for this information, I don't think
this is a viable approach. It is quite common for the update interval for
Sieves to be measured in months or even years.

				Ned

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 24 Aug 2010 17:41:26 +0200

Hi Ned,

> automatically generated, and an automatic generator should not be 

Right.

> difficult, ...

Virtual users are those that are no actual system users and therefore 
(typically) have no shell, ftp etc. access to the system. My log files 
are thus not available to virtual users. Providing access to these logs 
would require some more effort (given that these are currently not 
mailed to the users).

> ... but FWIW, in our implementation every Sieve script has an associated

Right, that is sensible. However, in the arguably rare but not 
impossible case that a single Sieve frequently fails with a runtime 
error caused by a single problem, a message is sent to the responsible 
person on each failed delivery. I think this situation should be 
accounted for somehow.

> In fact this behavior could be exploited as part of some sort of attack.

I agree.

> You're seriously overstating this risk. See above.

Ok, I wouldn't call this a risk per se. It is more of a nuisance that 
should be prevented somehow. See above.

> Sieves to be measured in months or even years.

You are right, for the actual notification it would therefore not be 
suitable. It could however still be used to obtain detailed information 
about the error, e.g. after being notified about an error by some other 
means (also see postings by Kurt).

Regards,

Stephan.

---

From: [email-masked]
Date: Tue, 24 Aug 2010 09:27:45 -0700 (PDT)

> > I don't know what a "virtual user" is or why being one makes error
> > handling difficult, ...

> would require some more effort (given that these are currently not

Then all our users are virtual users. I think this is now the norm. And while
providing log access may meet the letter of the requirement, I think it falls a
bit short on the user friendliness side.

> person on each failed delivery. I think this situation should be

I remain unconvinced that this is worth worrying about. Again, operationally
speaking this has been a complete nonissue for us. To the extent we've had
issues with Sieve error reporting, they've always been of the form, "I got this
error, what does it mean?" (What it usually means is that the Sieve didn't get
built properly by whatever front end interface is in use, and since often as
not people don't even understand that Sieve is what underlies all this, they
don't understand what messages saying stuff like "fileinto used without first
being activated with a require or ihave" mean.)

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Tue, 24 Aug 2010 18:49:17 -0400

Clarifying things here:

The server I was talking about was a research prototype that Wolfgang
and I did.  It was meant to eventually turn up in a product, but that
never wound up happening, and some of the decisions we made for
conditions like this were based on the fact that it was a research
prototype.

We didn't disable the script for things like "fileinto mailbox does
not exist," but syntax errors and other sorts of things got the script
disabled.  We didn't use anything like managesieve ... basically, we
had a special mailbox that users could use IMAP APPEND to stick a
message into, and the content of that message would be used as the
sieve script.  Compilation didn't happen until the script ran, and the
scripts were hand-coded by programmers.

This is obviously not the normal situation.

> don't understand what messages saying stuff like "fileinto used without
> first being activated with a require or ihave" mean.)

Indeed.  Telling an end user that her sieve script had an error is
somewhat like asking an end user what she wants the browser to do with
a security certificate problem: it's a WTF? situation.

I really think that Kurt and Ned are saying the same thing here: when
Kurt says that "do an implicit KEEP is enough of an indication that
something's wrong," he's saying what Ned means when he says "I'm not
convinced this is a problem we need to deal with."

I agree with them.  Perhaps if we revise 5228, at some point, we can
change the MUST in the sort of manner that Kurt suggests, but maybe
aim it more at informing an administrator, rather than a user, and add
some vague statement about the method of informing being up to the
implementation, but implementations should keep in mind that it's a
Bad Idea to complain to users about things they won't understand.

Barry

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Tue, 24 Aug 2010 19:22:00 -0400

> convinced this is a problem we need to deal with."

I think doing an implicit KEEP is enough of an indication that something is 
wrong.  However, it doesn't provide the information that a user or 
developer of a script generator would need to figure out _what_ is wrong. 
And telling the user to contact the administrator and get him/her to help 
go through the logs is really not a solution that scales to the Internet.

-- JEff

---

From: [email-masked]
Date: Tue, 24 Aug 2010 16:39:32 -0700 (PDT)

> wrong.

That's probably the case if the actions of the Sieve are obvious, as in 
reject or discard. But when it comes to more subtle stuff like adding
headers, maybe not.

> However, it doesn't provide the information that a user or

You actually need both: Send an error message and keep a log that can be
checked easily. The error message is good enough most of the time, but
there are plenty of users so clueless that they will be unable to find or
may even have deleted the error. In such cases tech support has the unenviable
choice between telling them to wait for the problem to come up again or
check the logs.

				Ned

---

From: Derek Diget <[email-masked]>
Date: Tue, 24 Aug 2010 19:52:20 -0400 (EDT)

On Aug 24, 2010 at 18:49 -0400, Barry Leiba wrote:
=>> not people don't even understand that Sieve is what underlies all 
=>> this, they don't understand what messages saying stuff like 
=>> "fileinto used without first being activated with a require or 
=>> ihave" mean.)
=>
=>Indeed.  Telling an end user that her sieve script had an error is
=>somewhat like asking an end user what she wants the browser to do with
=>a security certificate problem: it's a WTF? situation.

As an operator...yup.  Most just want it fixed.

=>I really think that Kurt and Ned are saying the same thing here: when
=>Kurt says that "do an implicit KEEP is enough of an indication that
=>something's wrong," he's saying what Ned means when he says "I'm not
=>convinced this is a problem we need to deal with."
=>
=>I agree with them.  Perhaps if we revise 5228, at some point, we can
=>change the MUST in the sort of manner that Kurt suggests, but maybe
=>aim it more at informing an administrator, rather than a user, and add
=>some vague statement about the method of informing being up to the
=>implementation, but implementations should keep in mind that it's a
=>Bad Idea to complain to users about things they won't understand.

As an operator of Ned's implementation I like the user getting a error 
e-mail as it then prompts the user to do 1) nothing, 2) figure out the 
problem themselves, or 3) contact our support desk.  As Ned mentioned, 
users construct their filters via a web interface that should not make 
syntacticly incorrect filters, but we did have a while were a single quote 
(') in a certain field was not escaped correctly by the web interface 
and it caused a "broken" filter.  (This was a couple of years ago.)  I 
fixed the user's filter by hand when a support case was opened.  (I 
don't remember if I ever went looking in the server logs as the admin to 
see if I could proactively fix filters for users that did nothing.)

The only part that I would have liked to see done differently was that 
the "notice/error" e-mail message be delivered BEFORE the implicit KEEP 
message.  As it was delivered after, I would get a question like "Why 
are you notify me about a message I already got?"  But, then you have 
started to enter the user's "WTF? situation". :)

So, make it a admin choice.  If they want their user's notified, great.  
If not, the implementation should still be logging it somewhere for the 
admin so see.

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************

---

From: [email-masked]
Date: Tue, 24 Aug 2010 16:53:06 -0700 (PDT)

> a security certificate problem: it's a WTF? situation.

Well ... yes and no. I agree that the error isn't going to be meaningful to 
the users, but the difference between these two cases is that although the user
may not understand what the Sieve error means, it will be meaningful when
included in any trouble ticket the user opens. Note that text in and around the
error message can include details of what to do to address the problem, e.g.,
visit http://myfiltersaregrinched.isp.net and report the problem, etc. (We
allow these messages to be customized so this sort of site-specific text can be
included. It also allows for localization/internationalization of the response.
We ship several by default, but of course there are many more languages in use
than we've localized for)

The certificate situation, OTOH, is truly a complete WTF, because it's not an
error, not exactly, and the user hasn't the faintest idea what to do. And
neither would a tech support person, even assuming an ISP would be willing to
open a ticket about a certificate problem for, say, a random web site. Heck,
it's not an infrequent occurance for me to run into expired or broken certs at
work, and in some of those cases I'm not sure what to do either.

				Ned

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Tue, 24 Aug 2010 20:41:30 -0400

> choice between telling them to wait for the problem to come up again or

Agreed.

---

From: Dilyan Palauzov <[email-masked]>
Date: Fri, 27 Aug 2010 23:20:12 +0200

Hello,

in my case the best option is to send email to the user, notifying  
about the error.  The users do not have mailboxes on my server, just  
aliases, so when an error occurs and I mail them, there shall be no  
problem with delivering the error message.

In my case the scripts are executed during SMTP time, not final  
delivery, and the most common actions are stop, reject or keep, with  
vacation being the only other possible action.  The point is that  
implicit keep is not always good: it could be smtp rejected by the  
subsequent mail servers -> bounced by my server (but I have no better  
ideas that implicit keep in case the script is broken).

Greetings, Dilian

----- Message from [email-masked] ---------
     Date: Tue, 24 Aug 2010 12:10:47 +0200
     From: Stephan Bosch <[email-masked]>
  Subject: [sieve] Poll: how to report Sieve runtime errors to the user?
       To: [email-masked]

>  Hi,
>
> I attended the last meeting in Maastricht through XMPP.  As I  
> explained there, I feel that something is currently missing in the  
> Sieve world: a standard means of handling and reporting runtime  
> errors to the user. It was suggested to ask around how people  
> currently implement this and, possibly, write up a best-practices  
> document. I promised to perform this task, since I was the one  
> coining the issue in the first place. Now I finally have the time to  
> put this on the list.
>
> error occurred and which actions (if any) were taken, and do an  
> implicit keep."
>
> The RFC doesn't specify how this is to be implemented and,  
> particularly for virtual users, this is an interesting design  
> problem. Therefore, I would like to invite implementors on this list  
> to report their current approach.  We can then discuss the benefits  
> and limitations of the various reported implementations and exchange  
> ideas on possible alternatives.
>
> The reason I brought this up is that my own implementation is  
> currently very limited and unsuitable for virtual users: it writes  
> an error log file in the directory where the main active script is  
> located (typically in the user home directory).
>
> that there was an error, and that their script is disabled until  
> they go and fix it (Barry)
> folder, for each failed delivery. Clearly, this can explode into a  
> large number of messages when the script is fundamentally broken.  
> This needs some mechanism to avoid reporting the identical problem  
> for subsequent deliveries.
> - extend ManageSieve with a means to get notified about runtime  
> errors and a means to retrieve (and possibly configure) runtime logs.
>
> Regards,
>
> Stephan
>
>
>
>
>

----- End message from [email-masked] -----

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 28 Aug 2010 00:56:15 +0200

> The alternative approaches/ideas I've seen/had thus far include:

A fourth possibility is to get rid of runtime errors, which isn't as 
hard as it sounds.

Arnt

---

From: [email-masked]
Date: Fri, 27 Aug 2010 22:33:50 -0700 (PDT)

> and it caused a "broken" filter.  (This was a couple of years ago.)

Yes, quoting problems are probably the most common source of incorrectly
constructed sieves.

> I

> started to enter the user's "WTF? situation". :)

That's a very interesting observation. The reason it works this way is that
we're in the middle of processing the message for delivery when the error
occurs. The error message is a separate message, and it has to be enqueued and
undergo regular message processing, so it's pretty much guaranteed to arrive
later.

I'll give some thought to how to change this behavior, but right now I don't
see a way to do it that wouldn't involve doing something fairly nasty to delay
the original message. And even then there's no guatantee that this will result
in the "correct" order.

> If not, the implementation should still be logging it somewhere for the

Also an interesting idea, but my reading of RFC 5228 says this violates the
MUST at the end of section 2.10.6. That said, the case in our implementation
where errors are most likely is actually in the system-level Sieves that are
typically hand-constructed by admins, and in this context a system log _is_
visible to the "user" who owns the incorrect filter. So in this case the RFC
isn't being violated. And this is also the case that's most likely to lead to
an excessive number of error notifications.

In other words, this sounds like a good feature to implement for system-level
sieves at least. Thanks for the suggestion!

				Ned

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 30 Aug 2010 18:02:55 -0400

--On Saturday, August 28, 2010 12:56:15 AM +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

> A fourth possibility is to get rid of runtime errors, which isn't as hard
> as it sounds.

You're kidding, right?
Since runtime errors may be environmental and/or data-dependent, 
eliminating them entirely is virtually impossible.

-- Jeff

---

From: [email-masked]
Date: Mon, 30 Aug 2010 15:05:09 -0700 (PDT)

> eliminating them entirely is virtually impossible.

Even if you ignore the data dependency issue and reject all scripts that
_could_ lead to a runtime error given the proper inputs (which I don't think
you're allawed to do), this is still an intractable problem.

Since Sieve suppports boolean expresions of arbitrary complexity, determining
whether or not a given statement will be reached is equivalent to the
satisfiability (SAT) problem. Cook showed that SAT is NP-complete.

So the best you can do is try and catch as many errors as possible when the
script is stored. But this in turn depends on having complete control
over Sieve storage. Sometimes this is operationally achievable, e.g., everyone
accesses storage through managesieve. But lots of times it isn't, e.g, Sieves
are stored as one field in a user database.

				Ned