Subject: Re: [sieve] Notify (RFC 5435) questions

From: Alexey Melnikov <[email-masked]>
Date: Thu, 1 Oct 2009 22:23:45 +0100

On Tue, Sep 29, 2009 at 8:55 AM, Barry Leiba
<[email-masked]> wrote:
 [...]
> I agree that it'd be better to put it into the variables spec.  You
> know I'm not suggesting going through everything to say where you can
> and can't do variable substitution; you should know me better than
> that.  But Hannah has a point: the variables spec says this:
>
>   This extension changes the semantics of quoted-string, multi-line-
>   literal and multi-line-dotstuff found in [SIEVE] to enable the
>   inclusion of the value of variables.
>
> ...and the grammar for "importance" uses none of those three syntax
> constructions.  Taken at its word, that would imply that variable
> substitution doesn't apply here.

I am fine with this being reported as an Erratum. Do you want to do that?

> We might change RFC 5229 to add that
> variables can substitute for literal strings as well.  Or we might
> decide that it's clear enough, and doesn't need changing.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 5 Oct 2009 09:26:34 -0400

>> decide that it's clear enough, and doesn't need changing.

...and no one's specifically commented on it.  If we have
rough/vague/sorta-maybe consensus for making a small change to 5229,
I'll submit an erratum.  Otherwise, we ought to leave it as is.  I
particularly want to hear Ned's opinion, as well as, of course, those
of others.

The change I would propose is to make the quoted text, above, change to this:

     This extension changes the semantics of quoted-string, multi-line-
     literal and multi-line-dotstuff found in [SIEVE], including any
literals that are
     directly specified in the grammar, to enable the inclusion of the
value of variables.

As far as I can tell, that is the only place in RFC 5229 that says
anything about what variables can substitute for.

Barry

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 05 Oct 2009 15:13:21 +0100

>anything about what variables can substitute for.
I think the Enotify spec needs to say that variable expansion applies to 
the importance field.

If you want to add a statement to the Variables spec, we can do that 
too. But I don't believe this would eliminate the need to clarify Enotify.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 5 Oct 2009 10:26:27 -0400

> I think the Enotify spec needs to say that variable expansion applies to the
> importance field.
>
> If you want to add a statement to the Variables spec, we can do that too.
> But I don't believe this would eliminate the need to clarify Enotify.

Hm.
I think that gets into Ned's concern, with which I agree, of having to
specify in every extension that variables apply here and here and
here.

If 5229 (as modified) makes it clear that grammatical structures such as
   foobar = "foo" / "bar"
are covered by variable substitution -- which is what I think my
proposed change does -- then why would we also need to update any of
the notify specs?

Barry

---

From: [email-masked]
Date: Mon, 05 Oct 2009 07:47:29 -0700 (PDT)

> I am fine with this being reported as an Erratum. Do you want to do that?

I'm not fine with it and don't want to do it. Again, this is a slippery slope.
We have any number of specifications: relational (:count, :value), imapflags
(list-of-flags), and date (:zone, :originalzone) to name a few which, like
notary, use an overlay ABNF syntax to further restrict the allowed values of a
partameter and which have the same "problem" you cite here. If we're to
"correct" this "error", we need to "correct" all the others, and it places yet
another silly requirement on future specifications that do this.

I will also point out that it isn't even necessary for ABNF to be involved.
This overly creative reading (and like it or not, that's what it is) of syntax
restriction could be extended to other paramters whose syntax is restricted in
other ways, such as date-part values in the date extension, the reason value in
the vacation extension, and so on.

Look, I get that overlay syntaxes are inherently subject to this sort of
confusion. It's why got rid of all of them from MIME in the transition from RFC
1521 to RFCs 2045 and 2046. The diffference is in MIME the overlay syntaxes
turned out not to be essentially useless. THis is not the case in Sieve.

Again, the place to deal with this is in the variables specification. Something
along the lines of "specifications for other Sieve extensions may employ ABNF
or other syntax specification tools to restrict the allowed values of certain
strings. When this is done it doesn't imploy that variable substitutions cannot
take place. Rather, the ability for variable substitution to occur is linked to
the string's role in the core Sieve grammer specified in RFC 5228". I can
support this sort of erratum.

> > decide that it's clear enough, and doesn't need changing.

I don't know what you mean by "literal string" here, so I don't know
what purpose, if any, this serves.

				Ned

---

From: [email-masked]
Date: Mon, 05 Oct 2009 08:10:51 -0700 (PDT)

> particularly want to hear Ned's opinion, as well as, of course, those

I just suggested some possible text. I don't think it's great, but it's
a start.

>      directly specified in the grammar, to enable the inclusion of the

This is too narrow. Any sort of syntax restriction, whether to a set of
specific strings, or to a time zone, or whatever, could possibly be construed
as not allowed variable usage. What needs to be conveyed is that these syntax
restrictions apply to the _result_ of variable substitution if such
substitutions are used.

> anything about what variables can substitute for.

Agreed.

				Ned

---

From: Philip Guenther <[email-masked]>
Date: Mon, 5 Oct 2009 08:15:10 -0700

On Mon, 5 Oct 2009, [email-masked] wrote:
...
> Again, the place to deal with this is in the variables specification. 
> Something along the lines of "specifications for other Sieve extensions 
> may employ ABNF or other syntax specification tools to restrict the 
> allowed values of certain strings. When this is done it doesn't imploy 
> that variable substitutions cannot take place. Rather, the ability for 
> variable substitution to occur is linked to the string's role in the 
> core Sieve grammer specified in RFC 5228". I can support this sort of 
> erratum.

+1

---

From: [email-masked]
Date: Mon, 05 Oct 2009 08:18:54 -0700 (PDT)

> turned out not to be essentially useless. THis is not the case in Sieve.

Correction: I meant to say "turned out to be essentally useless".

				Ned

---

From: Aaron Stone <[email-masked]>
Date: Mon, 5 Oct 2009 09:58:47 -0700

> restrictions apply to the _result_ of variable substitution if such

Certainly the result of a variable expansion in a syntax-restricted  
parameter must be checked to conform to that syntax after the expansion.

>> anything about what variables can substitute for.

There are parameters that restrict against variable expansion, but  
they are specified one-off in each document.

Aaron

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 05 Oct 2009 19:29:12 +0100

>support this sort of erratum.
Ok.

---

From: [email-masked]
Date: Mon, 05 Oct 2009 11:52:35 -0700 (PDT)

> parameter must be checked to conform to that syntax after the expansion.

Absolutely. The sequence is:

   (1) Syntax check of the core sieve grammar.
   (2) Perform variable expansion on the strings where it is appropriate.
   (3) Parameter-specific syntax checks.

> they are specified one-off in each document.

Indeed there are, but they are fairly uncommon (IMO they are actually too
uncommon - the body extension should have stated that variable substitutions
aren't allowed in search strings but did not). Explicit specification of such
uncommon exceptions is the right thing to be doing. Specification of the common
reality as it if was in fact uncommon is not.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Mon, 5 Oct 2009 17:18:16 -0400

> Explicit specification of such
> uncommon exceptions is the right thing to be doing. Specification of the
> common reality as it if was in fact uncommon is not.

And no one is suggesting doing the latter.  Relax on that point.  All
we're talking about here is making sure that 5229 is clear with
respect to the core 5228 syntax.

But I still contend the definition in 5229 of what bits of the core
5228 syntax can be replaced by variables isn't sufficiently clear.
Unfortunately, (1) we all know what we meant, and (2) neither of us
has yet come up with text that cleanly fixes the issue.

As an aside, there *is* an unarguable erratum in 5229: where the text
says "multi-line-dotstuff" it should be "multiline-dotstart", and
where it says "multi-line-literal" it should say "multiline-literal".

For an example of the problem of clarity, let's look at an example from 5229:

          fileinto "INBOX.lists.${2}"; stop;

I contend that's not valid from a strict reading of the grammar,
though we clearly want to allow it.  We've said that a variable can
stand in for a "quoted-string", but we did not say it could stand in
for a PORTION of a quoted string -- specifically, for a quoted-text
construction (and I'm not sure even that would do it, given the 5228
grammar).

I'll also note that since we don't include "number" in the
constructions that can be substituted, we can't use variables in the
"size" test, for example.  So this is syntactically invalid:

       if size :over ${max} {
           discard;
       }

I'm not sure whether or not that was intended.

Barry

---

From: Philip Guenther <[email-masked]>
Date: Mon, 5 Oct 2009 14:53:25 -0700

On Mon, 5 Oct 2009, Barry Leiba wrote:
...
> where it says "multi-line-literal" it should say "multiline-literal".

Yep, those are clearly typos.

> construction (and I'm not sure even that would do it, given the 5228

Umm, what?  5229, section 3p2 says:
   When a string is evaluated, substrings matching variable-ref SHALL be
   replaced by the value of variable-name.

*Substrings* matching variable-ref are replaced.  That seems to me to be 
an explict statement declaration that it "stands in" for a portion of a 
string.  Where do you see syntax or requirements that say that the entire 
string must match the variable-ref syntax?

> I'm not sure whether or not that was intended.

I seemed clear enough to me at the time.  The fact that we call out the 
syntax elements for quoted-string and multiline strings would seem to back 
up that people knew we were just talking about strings.

Philip Guenther

---

From: Barry Leiba <[email-masked]>
Date: Mon, 5 Oct 2009 18:18:06 -0400

> string.

Yes, good.  I'd missed interpreting that, in the mess I was looking
at.  That's good.  I think we are, indeed, set on this, then, and no
clarification (other than fixing the typos in the syntax names) is
needed.

> up that people knew we were just talking about strings.

I don't remember discussing it at all, so I'm not sure whether we
intended it or not.  Because the size test is the only thing affected
by this, it could easily have been an oversight (our other use of
numbers, as in :count and :importance, puts them in as strings).

Does anyone else remember specific discussion of variables not
substituting for numbers?  Do we care?

Barry

---

From: [email-masked]
Date: Mon, 05 Oct 2009 15:25:35 -0700 (PDT)

> substituting for numbers?  Do we care?

It did briefly come up at one point. I believe the consensus can be accurately
described as, "Meh."

				Ned

P.S. Numbers have certainly been a minority taste in Sieve arguments, but there
is at least one other place where they are used: vacation :days.

---

From: [email-masked]
Date: Mon, 05 Oct 2009 15:31:11 -0700 (PDT)

> up that people knew we were just talking about strings.

In case it isn't obvious, I will also point out that this is a change to core
Sieve syntax. That's a very big deal and not a change to be considered lightly.

				Ned

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 05 Oct 2009 19:26:55 -0400

--On Monday, October 05, 2009 03:31:11 PM -0700 
[email-masked] wrote:

>> syntax elements for quoted-string and multiline strings would seem to
>> back up that people knew we were just talking about strings.

I'd be inclined to agree.  The variables extension as published does not 
cover non-strings, and while it might be nice to be able to use variables 
in place of numbers, adding that feature would be a real change, not a mere 
correction to an oversight.

> In case it isn't obvious, I will also point out that this is a change to
> core Sieve syntax. That's a very big deal and not a change to be
> considered lightly.

Very true.  I think I'd sooner define a new test which works like size but 
uses a string.

-- Jeff

---

From: Barry Leiba <[email-masked]>
Date: Mon, 5 Oct 2009 19:45:48 -0400

> where it says "multi-line-literal" it should say "multiline-literal".

So, I'll await further comment by Alexey and a consensus call from the
chairs, but I'll tentatively plan to submit an erratum to correct the
errors above, and nothing more.

Barry