Subject: [sieve] Variable expansion issues

From: Hannah Schroeter <[email-masked]>
Date: Tue, 6 Oct 2009 17:55:30 +0200

Hello!

I've read the recent mails.

I agree to the general gist that it's best to specify once and for all
in the variables extension (RFC 5229) that variables expand in every
string position unless explicitly specified otherwise and that
additional ABNF-like language in the other specifications means
restrictions on the string result *after* variable expansion.

However, there are a few places where I would like/have liked such an
explicitly stated restriction on variable expansion.

The variables extension (RFC 5229) *does* specify such a restriction on
require (though a bit indirectly: "The require clauses themselves are
not affected by this extension.", first paragraph in section 1, it'd be
more explicit if the language using the term "constant string" would've
been used on "require", too), and on "set" (section 4, "The name MUST be
a constant string").

I'd think, then, it'd be logically consequential to require *constant*
strings in other places where variable *names* are used, such as in
- Imap4flags (RFC 5232),
 * The hasflag test (unlike the "string" test it doesn't use
   interpolation but a variable name itself). I do *not* mean
   "list-of-flags", but "variable-list".
 * The setflag, addflag, removeflag actions (section 3.1 through 3.3),
   parameter "variablename" (I don't mean to talk about
   "list-of-flags"!). Here, "variablename" is just like "name" in
   the "set" action in RFC 5229, and "name" for "set" is required to
   be constant. It'd be non-consequential to allow variable names be
   given by other variables here, when RFC 5229 just does *not*
   allow that (indirect variable references).
- Similarly: draft-ietf-sieve-mime-loop-09.txt,
 * The "varname" parameter of the "extracttext" action isn't specified
   to be constant there. The same argument applies though.

This especially makes sense as the use of constant strings in "set" and
not allowing indirect variable references in the interpolation syntax
allows for statical analysis of how many and which variables are used.
That is thwarted if one allows other extensions to use variables in
variable *names*. I think there those changes would be virtually
necessary.

The other way of consistency would be to drop the restriction on
"set" and to allow indirect variable references on interpolation, too,
e.g. "${x_${bar}}", but then enforcing limits on the number on variables
used must be deferred to run-time generally.

In fact there are other places where at least *I* would rather like
*constant* strings (or even tags) to be used, like the match types of
the relational extension. (And in fact, the ABNF there *did* suggest
constant strings to me!) IMO, variables don't really buy you much there
and it just adds unneeded complexity to a Sieve engine that translates
the script (from a simple enumeration [e.g. C-style enum] of match types
to a much more complex data type, using a combination of an enumeration
[basic match types] and a Sieve string, but the latter only sometimes).

Kind regards,

Hannah.

---

From: [email-masked]
Date: Tue, 06 Oct 2009 09:03:46 -0700 (PDT)

> been used on "require", too), and on "set" (section 4, "The name MUST be

I actually disagreed with the exclusion of variables from the first argument to
set here. But that's water long since under the bridge.

>    to be constant there. The same argument applies though.

Since I disagree with the exclusion of set, I also am opposed to this
expansion. And even if I were to agree to this in principle (which I don't),
I would still object to this change on the grounds that it makes
currently conforming implementation nonconformant for no good reason.

> variable *names*. I think there those changes would be virtually

I understand the logic behind such exclusions. I just don't agree that the
supposed benefits are worth the loss in flexibility.

> used must be deferred to run-time generally.

Doesn't follow. I think exclusion of indirect variable references in
expressions was a good thing, mostly because this is exactly the sort of thing
implementors mess up, but it went too far when it was applied to set, where
it's actually more work to implement the exclusion. However, now that set has
this exclusion, I think more substantial justification is needed to change it
now.

> [basic match types] and a Sieve string, but the latter only sometimes).

While it's true that use cases for variables in relational parameters, or
comparator parameters, seem pretty limited if not entirely nonexistant, in the
absence of a problem this  has caused I don't think a change is justified.

Justification for me for making such a change in whether or not variables are
allowed would be a concrete use case where the change provides a significant
tangible implementation benefit. There may be several cases of this, but the
one I know of is in the body extension. The fact that variables are allowed  in
the search string means that body has to either be implemented using an
unlimited number of passes over the message or by performing extensive
preprocessing of the message (e.g., building a Patricia tree). This limitation
makes body effectively unimplementable in high performance environments. But
take away variable substitutions and you can precompute all the search strings
and implement body in a single pass. This IMO is sufficient justification to at
least consider not allowing variable substitutions in body search strings.

				Ned

---

From: Stephan Bosch <[email-masked]>
Date: Thu, 22 Oct 2009 17:26:25 +0200

Hi Ned,

> set here. But that's water long since under the bridge.
[...]
> currently conforming implementation nonconformant for no good reason.
[...]
> supposed benefits are worth the loss in flexibility.

Although I tend to agree with your appreciation for flexibility, I also 
value consistency. My implementation does not support constructing 
variable names with variable substitutions anywhere, since I assumed all 
variable name arguments to have similar requirements as the set 
command's variable name argument (e.g. for hasflag). Now I am wondering 
whether I should implement support for this anyway. Did you?

I think consensus on issues like these is very important to guarantee 
interoperability.

Regards,

Stephan.

---

From: [email-masked]
Date: Thu, 22 Oct 2009 08:52:03 -0700 (PDT)

> whether I should implement support for this anyway. Did you?

Well, I wouldn't go so far as to say "implement support", because that sounds
like a bunch of work. The routine in my implementation that computes parameter
values takes an argument specifying whether or not to perform variable
expansion in a given case. So for me changing the semantics so that, say, the
first argument of set allows variable expansions is simply a matter of changing
FALSE to TRUE in one call.

I even have the parameter set to TRUE for arguments to body, but if you
actually use a variable substitution in such an argument you're very likely to
get a run time error. As I explained previously, the alternative was not to
implement any form of body test.

				Ned