Subject: Re: Updated Sieve notification draft

From: Alexey Melnikov <[email-masked]>
Date: Sun, 02 Oct 2005 22:01:03 +0100

Michael Haardt wrote:

>On Tue, Sep 27, 2005 at 12:10:22PM +0100, Nigel Swinson wrote:
>  
>
>>    This document does not dictate the notification method used.
>>    Examples of possible notification methods are Zephyr and Jabber. The
>>-    method shall be site-defined.
>>+    available methods shall be site-defined.
>>
>>Which makes me wonder if each desired "method" should really be available via a require command?
>>    
>>
>
>I prefer a require commend for each method
>
I don't want to require a "require" statement for each URI schema. An 
URI can be fetched using a test from an external server (e.g. it can be 
an LDAP attribute value ;-)), in which case the script might not know 
all URI schemas in advance.

>and syntactically checking the URL, generating an error on bad syntax.
>
I am fine with this part. I've added to 3.1:

    If the :method tag contains a supported URI schema, then the URI MUST
    be checked for validity. An invalid URI syntax or an unsupported URI
    extension MUST cause an error.

Is this what you want?

>  And I would go as far as
>allowing semantic errors to cause errors on behalf of the implementation.
>  
>
>I ran various SMS gateways over the years and if there is one common
>problem, it is people embedding letters and all kinds of special
>characters in phone numbers, like @ and 8-bit characters, or doubling
>the number or the area prefix which yields way too long numbers.
>Simply dropping those numbers results in complaints why important SMS
>were not sent.  At one time I had almost 10% bad numbers and I am sick
>of complaints.
>  
>

---

From: Michael Haardt <[email-masked]>
Date: Tue, 4 Oct 2005 15:17:59 +0200

>    extension MUST cause an error.

I suggest "syntactic validity" instead of "validity" and I would add

"Semantic errors MAY cause an error".

Foreign phone numbers are syntactically fine, but may not be allowed as
SMS targets by a site.  I think such a configuration should be allowed
to generate an error.  Most sites can not check if a phone number exists,
thus being unable to generate an error.  Hence MAY.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 13 Oct 2005 23:31:20 +0100

Michael Haardt wrote:

>On Tue, Sep 27, 2005 at 05:56:04PM +0200, Kjetil Torgrim Homme wrote:
>  
>
>>one general issue which I think should be specified in 3.1, is how to
>>handle charsets when the notification method doesn't support full UTF-8.
>>something like: "Characters in the message which can't be represented by
>>the notification method SHOULD be replaced with a symbol indicating an
>>unknown character, e.g., a question mark."
>>    
>>
>
>Forget about that.  If you connect to a SMSC and speak UCP, it is your
>choice how to convert the message text to IA5.  If you use a regular
>gateway by SMTP or HTTP, you can not deliver IA5, but usually ISO-8859-15,
>and you have no influence on how that is converted to IA5.  The ones I
>saw so far just dropped characters they can not translate.
>  
>
I think the best we can do is to say that a notification mechanism 
specified how characters that can't be represented by the notification 
method must be handled handled.

>I choose to use UCP, if I can, but it is not offered by all SMS providers.
>  
>

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Oct 2005 11:38:35 +0100

Michael Haardt wrote:

>>    Usage:   notify [":method" string]
>>               [":id" string]
>>               [<":low" / ":normal" / ":high">]
>>               [":message" string]    
>>
>
>I miss a sender specification of some kind, like [":from" string], like
>we have for vacation.

So you want to distinguish between notifications caused by messages
being delivered into different accounts?

Fine with me, as long as it is optional.

>>    The format of the notification is implementation-defined. However,
>>    all content specified in the notify action, including Sieve actions
>>    taken on the message, SHOULD be included. If errors occurred in
>>    another action they SHOULD be reported in the notification. In
>>    addition, if the notification method does not provide a timestamp,
>>    one SHOULD be appended to the notification. Implementations SHOULD
>>    NOT include extraneous information.
>
>I don't like that, because it means I have to delay sending notifications
>until successful delivery of a message, like transferring it to the mail
>store or forwarding it to another host.

Based on numerous comments I've updated the paragraph to read:

    The format of the notification is implementation-defined and
    is also affected by the notification method used (see below).
    However, all content specified in the notify action SHOULD be
    included. It is RECOMMENDED that a timestamp is included in
    the notification. Implementations SHOULD NOT include extraneous
    information.

This should eliminate any confusion.

>What if the user is over quota
>or the remote host is down? That's not a successful delivery.  Yet I would
>like to get a notification that the mail was seen by the filter.
>
In my implementation overquota checks are done after Sieve, so
notifications will be always sent.

Anyway, I hope the new text clarifies this.

>I guess what I do not like it that notify sounds like DSNs.  To me,
>it should be just some action.  If I decide to send a notification and
>then discard the message in order to process mail alarms, I don't need
>to have notify tell me the message was discarded.
>
>>    If the :method tag is not specified, the default
>>    implementation defined notification method is used.  The
>>    possible values of this will be site-specific.  If an URI schema is
>>    specified that the implementation does not support, the notification
>>    MUST be ignored. An implementation treats this as a warning
>>    condition and execution of the sieve script MUST continue.
>>    
>
>As I wrote in another mail, people have a hard time to get phone numbers
>right, like not including letters in them.  I prefer to generate errors
>if that happens.  That way they see something is wrong instead of yelling
>at the SMS provider.

Ok, the new text reads:

    If an URI schema is specified that the implementation does not
    support, the notification MUST be ignored.

And later:
    If the :method tag contains a supported URI schema, then the URI MUST
    be checked for syntactic validity. An invalid URI syntax or an
    unsupported URI extension MUST cause an error.

>May an implementation choose not to send any notification if no method is
>given? I run a bunch of systems without a default notification mechanism.

I am Ok with that, but I would like to hear more opinions.

Alexey

---

From: Michael Haardt <[email-masked]>
Date: Fri, 14 Oct 2005 13:19:31 +0200

> method must be handled handled.

How about:

The implementation of a notification mechanism may modify the
final message, e.g. truncating it, if it exceeds a length limit,
or modify characters that can not be represented in the
target character set.  The implementation SHOULD keep such
modifications to a minimum.

Michael

---

From: Michael Haardt <[email-masked]>
Date: Fri, 14 Oct 2005 13:34:30 +0200

> Fine with me, as long as it is optional.

Some notification methods, like SMS, support the concept of a sender.
I just want to be able to influence that, perhaps even different senders
for the same account.  I see some similarity between notify and vacation,
and suggest, as written in section 4.3 of the vacation draft:

   Implementations MAY also impose restrictions
   on what addresses can specified in a ":from" parameter; it is
   suggested that values which fail such a security check simply be
   ignored rather than causing the vacation action to fail.

Reading that, shouldn't it be listed under "security considerations"
as well?

> This should eliminate any confusion.

Yes, that is very clear.

> notifications will be always sent.

Same here.

>    support, the notification MUST be ignored.

I still do not particularly like that, but if it is required by
technical reasons, I can live with it.  To me, it is odd if a
typo in the URI scheme is ignored, but typos later on cause errors.

>    unsupported URI extension MUST cause an error.

May I add:

A semantically invalid URI may cause an error?

That would allow me to cause errors for phone numbers my implementation
knows to be crap, although they are syntactically valid.

Michael

---

From: Michael Haardt <[email-masked]>
Date: Fri, 14 Oct 2005 15:36:13 +0200

On Fri, Oct 14, 2005 at 02:05:02PM +0100, Alexey Melnikov wrote:
> How about the following:
> 
>    An implementation MAY enforce
>    other semantical restrictions on URIs, for example an SMS URI can only
>    contain phone numbers in a particular geographical reason. Violation
>    of such semantical restrictions MUST also cause an error.

Semantical restrictions is exactly the term I couldn't put into words!

Is the MUST useful? If an implementation wants to ignore violating such
restrictions, it could just pretend there are no restrictions and let
things fail silently, hence it it not forced to treat them as error,
despite the MUST.

The above would make sense if absence of errors would guarantee delivery
of a notification (making notify transactional), but unless semantic
validity could be checked completely, I see no way to offer that.

Compared to the original specification that enforced errors for
syntactically invalid URLs and allowed them for nothing else, I would
like to additionally allow generating errors upon semantic restrictions,
but not enforce them.

While saying that, we have a similar situation with "redirect".
Would anybody object if "redirect" caused an error for syntactically
valid addresses that violate e.g. a site policy of not allowing mail to
be forwarded to external addresses? Some sites may check that in the
central mail hub, thus not allowing to detect the error inside Sieve
and ignoring it.

Kind of like: If I can detect a violation, I flag an error, but if I
can't until it is too late, I ignore it.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Oct 2005 14:05:02 +0100

How about the following:

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Oct 2005 14:53:48 +0100

>modifications to a minimum.
My current text is:
    The notification method must also specified how Unicode
    characters that that can't be represented by the notification method
    are to be handled. Notification methods MUST be able to represent any
    US-ASCII character with exception of control characters.

Should I just replace it with what you've suggested, or do people feel 
that the last sentence is appropriate?

---

From: Michael Haardt <[email-masked]>
Date: Fri, 14 Oct 2005 17:48:44 +0200

> that the last sentence is appropriate?

IA5, the GSM character set, does not contain a backquote character
(US-ASCII dec. 96).  There goes "any US-ASCII character".  Do pagers
still exist? The devices I remember from the past contained only numbers.

Michael

---

From: Ned Freed <[email-masked]>
Date: Fri, 14 Oct 2005 14:03:32 -0700 (PDT)

> still exist? The devices I remember from the past contained only numbers.

A small point of clarification here: IA5 and the GSM 03.38 character set are
not the same, and IA5 does include grave accent (aka backquote) as one of its
"IRV graphic character allocations".

These days IA5 is for all intents and purposes the same as US-ASCII. (The
variants that include things like the universal currency sign rather than
dollar sign are rarely used.) The GSM charset, on the other hand, has no direct
representation for most of the IRV graphic characters and does include a bunch
of characters that aren't in IA5 or ASCII (e.g., an ae ligature).

In any case, specifications of mappings to various more restrictive charsets
have been widely used for a long time and seem to work pretty well. X.400-1984
discusses several such mappings in detail, for example. I would therefore
suggest changing the text from "able to represent" to "able to handle" or
something similar.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Oct 2005 18:07:43 +0100

>typo in the URI scheme is ignored, but typos later on cause errors.
You have a point: it is not possible to distinguish a mistyped URI 
schema name from an unsupported URI type.

The case I had in mind is as follows: The notification URI is stored 
elsewhere (as an LDAP attribute or an IMAP annotation) and Sieve scripts 
uses Sieve variables extension to specify the notification method. Is it 
desirable for an unrecognized notification method to cause runtime 
error? I am open to suggestions.

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Oct 2005 17:50:29 +0100

Mark E. Mallett wrote:

>As for a few specific comments (pared down a little):
>
>    > 1.  Introduction
>    > 
>    > This is an extension to the Sieve language defined by [SIEVE] for
>    > providing instant notifications of sieve actions that have been
>    > preformed. It defines the new action "notify".
>
>That definition suggests that executing the "notify" action will cause
>a notification.
>  
>
Right. I've deleted the rest of the sentence after "for providing 
instant notifications".

>    > 1.0
>    > 
>    > Sieve interpreters for which notifications are impractical or is not
>    > possible SHOULD ignore this extension.
>
>What does this mean... are you recommending that interpreters implement
>the extension as no-ops, so that the "require" will work but the actual
>notify statements will not?  I mean, this seems to be recommending
>something other than the usual thing of having the "require" fail to
>operate if the capability is not available.  If all this means is
>"have the 'require' fail if the capability isn't available" -- doesn't
>that go without saying?
>  
>
You are right, this is meaningless and I've deleted the sentence.

[...]

>
>
>> The notify action is compatible with itself.
>
>Meaning?
>
Meaning that multiple executed notify actions are allowed. I've changed 
the sentence to read:

    The notify action is compatible with all actions.
    Multiple executed notify actions are allowed.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Oct 2005 16:13:33 +0100

>still exist? The devices I remember from the past contained only numbers.
Ok, you've convinced me. I've used your text with minor editorial changes.
I've also replaced the last sentence with:

    Allowed modifications should be documented in a standard track or an 
informational document.

---

From: Michael Haardt <[email-masked]>
Date: Mon, 17 Oct 2005 10:04:56 +0200

> error? I am open to suggestions.

People who use notifications usually consider some (or all) mails very
important.  Silently ignoring errors, in particular when variables are
involved and the error only happens every now and then, will make them
think notifications were sent when they were not.  SMS are not reliable
and they may complain about the service.

If I specified some attribute of a message wrong and the message arrives,
I will see what's up.  An error might have been better, but then again,
I may be happy something worked at all.

If I make a mistake and nothing is sent, but I don't get an error either,
I am lost.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 17 Oct 2005 14:01:57 +0100

Michael Haardt wrote:

>important.  Silently ignoring errors,
>
Maybe the document is not clear, but I am not suggesting silently 
ignoring errors: an error message can be logged, but the script 
execution should continue.

>in particular when variables are

---

From: Michael Haardt <[email-masked]>
Date: Wed, 19 Oct 2005 16:33:06 +0200

> execution should continue.

The base spec (section 2.6.10, at the end) requires that the user is
informed of errors, in which case the message is kept.  In case this means
there must be means by which Sieve can also send messages in absence of
errors, we really need to document that in the base spec.  Otherwise,
you can not rely on being able to generate warnings.

Michael

---

From: Ned Freed <[email-masked]>
Date: Mon, 24 Oct 2005 11:08:07 -0700 (PDT)

> think notifications were sent when they were not.

I agree that error notification is important and should be done when possible.
However, I also think it is important to keep some perspective here. Many
notification services are in fact NOTORIOUSLY unreliable. Even worse, error
reporting when failures occur is in many cases impossible as a result of how
notification services are set up or even as a result of the design of the
service.

> SMS are not reliable

Right, but this cuts both ways. Errors should be reported when possible in
order to prevent blame from being laid at the wrong door - if it is SMS that
failed fine, but if there's a busted sieve script the user needs to know that
in order to distinguish it from an SMS failure. OTOH, people routinelly reach
bogus conclusions about where fault lies and no amount of persuasion will
convince them they're wrong. So, while requiring error reports may help in some
cases, in many others it will make no difference.

				Ned