Subject: Re: variables draft (draft-homme-sieve-variables-00.txt)

From: [email-masked]
Date: Sun, 06 Apr 2003 07:09:40 -0700 (PDT)

I have just finished implementing this draft. Nice work -- It was considerably
easier to do than I expected.

Of course I now have a number of comments:

(1) The ${0}, ${1} all-digit variables should be set and modified by the
    glob-style :matches mechanism in addition to :regex. It really should
    not be necessary to implement regex in order to get the ability to
    set variables based on message inputs. For example:

    require ["variables", "envelope", "subaddress"];
    if allof(envelope :all :is "from" "a@b",
             envelope :detail :matches "to" "*")
      redirect "d+${1}@e"

    Of course glob style patterns don't have the ability to specify
    which wildcard characters capture and which do not, but I really don't
    this this will be a problem in practice.

    The presence of "variables" in the require list should be sufficient
    to enable setting of the all-digit variables by :matches.

(2) Implementations should be able to impose limits on the total number of
    variables as well as the size of any particular variable. With such
    limits it is easy to write scripts that consume an unreasonable amount
    of space:

    require "variables";
    set a "aaaaaaaaaa";
    set a "${a}${a}${a}${a}${a}${a}${a}${a}${a}${a}";
    set a "${a}${a}${a}${a}${a}${a}${a}${a}${a}${a}";
    set a "${a}${a}${a}${a}${a}${a}${a}${a}${a}${a}";
    set a "${a}${a}${a}${a}${a}${a}${a}${a}${a}${a}";
    ...

    That's already 100000 characters, with no end in sight.

    I suggest minimum maximums of 64 variables that can each hold 998
    characters.

(3) The draft needs to make it clear that ${} constructs are not interpreted
    in a special way inside of strings unless the "variables" extension is
    listed in a require clause.

(4) There's a comment to the effect that fileinto needs to be extended to
    create folders if they don't already exist. As far as I can tell the
    specification of fileinto in RFC 3028 is silent on the folder creation
    issue. Our implementation certainly allows folder creation by fileinto;
    I suspect others do as well. But regardless, I think this is a matter
    that needs to be clarified in the base sieve specification; I don't think
    it calls for a separate extension.

(5) Other than we decide on something and stick with it, I have no preference
    for modifier ordering.

(6) The selection of specific lowercasing or uppercasing rules for lower: and
    upper: needs to be done by specifying a comparator. Comparators have
    implement case conversion rules. Chris Newman has a new draft that
    cleans up the situation surrounding the comparator registry but it isn't
    out as an I-D yet.

(7) The setdate operator accept a "time zone" argument. And this is exactly
    what it should take. However, what's currently specified is a time offset,
    not a time zone -- the former is something like -0800 while the latter is
    something like "US/Pacific". Unfortunately the IETF does not presently have
    a time zone rules registry. This has been a problem for some time and
    I don't see it changing any time soon. I suggest that the correct
    approach to take is the one RFC 2445 uses. I would also like to allow
    a time offset as an argument to setdate, but zone support is what's needed.

(8) The default zone for setdate should probably be the zone local to the
    server. I think this makes a lot more sense than GMT.

(9) The specification of setdate needs to state that every setdate in a
    given script operates on a single, consistent date/time value that does not
    change throughout the execution of the script. If you don't do this you
    can get some unpleasant boundary cases. Note that it is entirely
    reasonable to use setdate multiple times in a single script, either
    because of automated script construction or else because someone
    wants the current date in different time offsets.

(10) I had been planning to write up a specification for currentdate and date
     tests. The former would allow testing of current date information while
     the latter would be similar to the address test except it would operate on
     fields containing date information.

     Setdate and currentdate provide equivalent functionality: Setdate combined
     with the string test give you the ability to test current date information
     while currentdate combined with :matches and the set action give you the
     ability to get date information into the variable of your choice. Of course
     each one is better syntax-wise at doing what it is directly designed to
     do:

    setdate "+0200"                     if currentdate :year :matches "*"
    fileinto "${year}-${month}"           set year "${1}"
                                        if currentdate :month :matches "*"
                                          set month "${1}"
                                        fileinto "${year}-${month}"

    setdate "+0200"                     if currentdate :year :is "2003"
    if string :is "${year}" "2003"        ...
      ...

     I'm inclined to go with setdate at this point, mostly because the
     setdate syntax seems simpler to use overall. My one remaining concern
     is that setdate doesn't match up well with the obvious syntax the
     date test needs to have. So I'd appreciate feedback from others on this
     point.

That's enough for now. As I said at the beginning, I think this is very good
document and I hope we can advance it sooner rather than later.

				Ned

---

From: Lawrence Greenfield <[email-masked]>
Date: Mon, 7 Apr 2003 17:08:53 -0400

Date: Sun, 06 Apr 2003 07:09:40 -0700 (PDT)
   From: [email-masked]

This leads to ambiguity, however.

What does:

envelope :detail :matches "to" "**"
or how about "*a*"?

do? Is it greedy?

Also, this draft might be a little surprising to implementors since
something like:

           if header :regex "List-ID" ["<alpha(.*)@", "(.*)beta@"] {
             fileinto "lists.${1}"; stop;
           }

can no longer short circut due to:

"The decimal value of the numeric variable name will index the list of
matching strings from the each of the group operators in the latest
regular expression match."

So _all_ regular expressions have to be tried and the last of the
matches causes the binding.

   (5) Other than we decide on something and stick with it, I have no
       preference for modifier ordering.

Modifiers should be omitted from this specification. These are
functions; if we're going to define functions, fine. If not, fine.

There are many other problems with this draft, ranging from i18n
concerns: "Variable names are case insensitive." to ones that will
potentially significant problems later on ("All variables have global
scope.").

Larry

---

From: [email-masked]
Date: Mon, 07 Apr 2003 14:39:06 -0700 (PDT)

> This leads to ambiguity, however.

Good point, but it is an ambiguity that is easily resolved simply by saying
glob patterns have to implement greedy matching when variables are used.

> envelope :detail :matches "to" "**"

${1} - full detail field
${2} - empty string

> or how about "*a*"?

${1} - String before the last a
${2} - String after the last a.

> do? Is it greedy?

AFAIK 3028 didn't say, so the variables document should say that greedy matching
is to be employed.

> regular expression match."

I don't think that was the intent, and if it was, I think it should be
changed. I read "latest" as meaning "the latest operation that was performed.
So if you have

	    if header :matches "subject" "*a*" ...
	    if header :matches "subject" "*b*" ...

where the subject contains both an "a" and a "b" the values of ${0}, ${1}, ...
would be set by the "*b*" pattern.

> matches causes the binding.

Big mistake IMO. Short circuit rules should still apply. The base specification
recommends short circuit evaluation of lists. This should be changed to a
requirement when variables are in use. The same is also true of anyof. (Allof
doesn't matter as far as I can tell.)

> functions; if we're going to define functions, fine. If not, fine.

I have no problem with this approach either.

> potentially significant problems later on ("All variables have global

Given that variable names are explicitly limited to ASCII alphanumeric I fail
to see an issue on this point. As for scope, I think global scope is the right,
and indeed the only realistic, choice.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 02:07:18 +0200

[[email-masked]]:
>
>   I have just finished implementing this draft. Nice work -- It was
>   considerably easier to do than I expected.

thank you.

>       Of course glob style patterns don't have the ability to
>       specify which wildcard characters capture and which do not,
>       but I really don't this this will be a problem in practice.

your example shows that it is possible to do useful things with it, at
least.

it is possible to add, though.  one method is to split the match
string on the meta characters.  consider the string "*@*.com" -- we
split this in four, "*", "@", "*" and ".com".  ${2} and ${4} will
always be "@" and ".com" (if there's a match), but ${1} and ${3} will
contain the matching characters.

does this sound too complicated for too little gain?  isn't it likely
that those who implement variables will also implement regex?

>       The presence of "variables" in the require list should be
>       sufficient to enable setting of the all-digit variables by
>       :matches.

I agree.

>   (2) Implementations should be able to impose limits on the total
>       number of variables as well as the size of any particular
>       variable.

good catch!

>       I suggest minimum maximums of 64 variables that can each hold
>       998 characters.

while there is a tradition for 998 characters as a maximum line
length, I think it is very low for this application.  remember that
some people may want to put longish vacation messages inside
variables.  I suggest 30000 characters (not octets) and an even
hundred variables as minimum.

>   (3) The draft needs to make it clear that ${} constructs are not
>       interpreted in a special way inside of strings unless the
>       "variables" extension is listed in a require clause.

not sure what you want here.  either a clarification in the
introduction like:

   This is an extension to the Sieve language defined by [SIEVE].  It
   adds support for storing and referencing data in string variables.
+  The mechanisms detailed in this document will only apply to Sieve
+  scripts which include a require clause for the "variables"
+  extension.

or, do you want to outlaw making a different extension which is
mutually incompatible with "variables"?  is that even possible to do
without updating the base specification?

>   (4) There's a comment to the effect that fileinto needs to be
>       extended to create folders if they don't already exist. As far
>       as I can tell the specification of fileinto in RFC 3028 is
>       silent on the folder creation issue. Our implementation
>       certainly allows folder creation by fileinto; I suspect others
>       do as well. But regardless, I think this is a matter that
>       needs to be clarified in the base sieve specification; I don't
>       think it calls for a separate extension.

okay.  clearly independent of "variables", anyhow.

>       preference for modifier ordering.

let's stick to prefixing (function call like), then.

>   (6) The selection of specific lowercasing or uppercasing rules for
>       lower: and upper: needs to be done by specifying a
>       comparator. Comparators have implement case conversion
>       rules. Chris Newman has a new draft that cleans up the
>       situation surrounding the comparator registry but it isn't out
>       as an I-D yet.

interesting.  do you know where I can get a copy?  I wonder how to
cram this into the modifier syntax, though.  perhaps the current
approach is completely wrong.  rather than stick modifiers inside the
strings, we could make them modify the SET action.  it seems a better
fit to fit other elements of Sieve (both syntax and semantics).

    SET :upper :comparator "i;ascii-casemap" name "${1}"

"i;ascii-casemap" would be the default comparator for SET.  I'm not
sure what the meaning of using "i;ascii-numeric" or "i;octet" with SET
should be.  they may make more sense if other modifiers are added in
the future.

sorry, I should have thought of that approach earlier.

>   (7) The setdate operator accept a "time zone" argument. And this is
>       exactly what it should take. However, what's currently
>       specified is a time offset, not a time zone -- the former is
>       something like -0800 while the latter is something like
>       "US/Pacific". Unfortunately the IETF does not presently have a
>       time zone rules registry. This has been a problem for some
>       time and I don't see it changing any time soon.

what would it take to get it changed?  it seems to me they could just
take (a subset of) the Olson database and decree it as official.

>       I suggest that the correct approach to take is the one RFC
>       2445 uses. I would also like to allow a time offset as an
>       argument to setdate, but zone support is what's needed.

interesting, thanks for the reference.  do you mean to include the
whole vocabulary to define a time zone?  that seems very complex to
me.

one pragmatic approach is to allow "well-known" time zones with a
fallback offset value supplied by the user:

  setdate "US-Eastern" "-0500"

to allow the user to discover that the time zone was unknown, a
variable called "timezone" would be set to the value actually used.
there should be a note that if IETF establishes a registry for time
zones, the values in it SHOULD be used.  if no fallback value is
specified, use the server's default.

actually, with ${timezone} available, the fallback can be implemented
by the user.  we simply need to mandate that the time offset syntax is
supported as a valid "time zone".

    setdate "US-Eastern";
    if not string "${timezone}" "US-Eastern" {
        setdate "-0500";
    }

>   (8) The default zone for setdate should probably be the zone local
>       to the server. I think this makes a lot more sense than GMT.

the objection to that was that a Sieve script should behave the same
way even if it was moved to a server in a different time zone.  e.g.,
imagine a local ISP in the UK being swallowed up by a pan-European ISP
which centralises all email servers in Brussels.  that must lead to
breakage for customers.

>   (9) The specification of setdate needs to state that every setdate
>       in a given script operates on a single, consistent date/time
>       value that does not change throughout the execution of the
>       script.

agreed.  how about this?

    These variables SHOULD reference the time when execution of the
    Sieve script reaches the statement.
+   The result of all calls to setdate MUST refer to the same point in
+   time.

>   (10) I had been planning to write up a specification for currentdate
>        and date tests. The former would allow testing of current
>        date information while the latter would be similar to the
>        address test except it would operate on fields containing
>        date information.
>    [...]

the currentdate syntax is obviously easier to read in tests.

>        I'm inclined to go with setdate at this point, mostly because
>        the setdate syntax seems simpler to use overall.

it allows general use without an IF and SET for every value needed,
yeah.

>        My one remaining concern is that setdate doesn't match up
>        well with the obvious syntax the date test needs to have.

what date tests are needed?  Sieve is quite expressive already.  an
example:

  require [ "variables", "vacation", "relational" ];
  setdate "+0200";
  if string :value "le" "${year}-${month}-${day}" "2003-09-18" {
        vacation "I'm on my holidays, I'll be back on Sept. 19th";
  }

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 04:18:10 +0200

[Ned Freed]:
>   [Lawrence Greenfield]:
>   I don't think that was the intent, and if it was, I think it
>   should be changed. I read "latest" as meaning "the latest
>   operation that was performed.

yes.  I've changed it to "most recently evaluated match".

>   Big mistake IMO. Short circuit rules should still apply. The base
>   specification recommends short circuit evaluation of lists. This
>   should be changed to a requirement when variables are in use.

short circuit is fine, but:

    if header :matches [ "To", "Cc" ] [ "coyote@*.com", "*@*.com" ] {
         ...
    }

which tests are done first?  if we care about effiency, it is best to
do all tests against the same match expression first.

   "To" :matches "coyote@*.com"
   "Cc" :matches "coyote@*.com"
   "To" :matches "*@*.com"
   "Cc" :matches "*@*.com"

this seems intuitively correct, since the first list contains
"synonomous" headers for the purpose of the match.  it also gives
natural precedence to the order of the match strings (most important
first).

>   I have no problem with this approach either.

since folder names are case sensitive, a method for normalising input
into all lower case or similar is highly desirable.

>   Given that variable names are explicitly limited to ASCII
>   alphanumeric I fail to see an issue on this point. As for scope, I
>   think global scope is the right, and indeed the only realistic,
>   choice.

if the need ever arises, we can define a action (say, "LOCAL") to set
variables with block scope.  I don't intend to turn Sieve into a
general purpose programming language, so I'll leave it out for now :-)

-- 
Kjetil T.

---

From: Lawrence Greenfield <[email-masked]>
Date: Mon, 07 Apr 2003 23:24:50 -0400

More problems:

* Semantics

>    Strings SHOULD be evaluated every time control reaches the statement
>    it is a part of, to make sure the expanded variable values are up-to-
>    date.

This is central to the semantics of the language; this isn't a SHOULD.

It should also be made clear that variables are expanded once, not 
repeatedly.

* Syntactic structure

The "set" action doesn't fit into the Sieve grammar, as something that is 
identical to an "identifier" is not a valid argument.

* Scope

--On Monday, April 07, 2003 2:39 PM -0700 [email-masked] wrote:

> As for scope, I think global scope is the right, and indeed the only
> realistic, choice.

Global scope is a wonderful choice that years of programming language 
theory has shown to be an extremely bad thing.

When Sieve scripts are run in serial (as many implementations now allow) do 
variables carry over from one to another? Why or why not?

If and when people start thinking about includes and functions, scope 
becomes even more important.

* Numeric variables

When does the side effect of tests have it's effect on numeric variables? 
After the test? Only inside the if?

Does the ${1} in the second test refer to the variable set in the first 
test? Or does it refer to the previously set variable?

if (allof (header :regex "Foo" "${1}(.*)", header :regex "Bar" "${1}(.*)")) 
{
}

* Numeric variables

What does nested grouping do?

header :regex "Foo" "((a|b))*"?

* Numeric variables

Does a failed matching clear all numeric variables?

* Program analysis

I guess some of what's bothering me is that there are several invasive 
language changes with this draft that make it significantly more difficult 
to analyze the language to make sure it isn't mutually contradictory, and 
more difficult to analyze a piece of a program without global analysis.

. Tests now alter global state (thus the short-circuit problem)

. Variable expansion can drastically change patterns to match against, 
preventing certain pre-compilation strategies.

Larry

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 06:27:05 +0200

[Lawrence Greenfield]:
>
>   More problems:
>   
>   * Semantics
>   
>   >    Strings SHOULD be evaluated every time control reaches the
>   >    statement it is a part of, to make sure the expanded variable
>   >    values are up-to-date.
>   

the wording isn't good, I welcome help.  the intent was to allow
static analysis to optimise variable references away.  perhaps an
"as-if" rule is understood, so changing the SHOULD into a MUST is a
non-problem.

>   It should also be made clear that variables are expanded once, not

good point.

    When the string is evaluated, substrings matching variable-ref shall
    be replaced by the value of variable-name.
+   Only one pass through the string shall be done.

one of the examples ("${dollar}{beep}") demonstrates this.

>   * Syntactic structure
>   
>   The "set" action doesn't fit into the Sieve grammar, as something
>   that is identical to an "identifier" is not a valid argument.

I don't understand what you mean here.

>   Global scope is a wonderful choice that years of programming
>   language theory has shown to be an extremely bad thing.
>   
>   When Sieve scripts are run in serial (as many implementations now
>   allow) do variables carry over from one to another? Why or why
>   not?

does require carry over?  I'd say "no".  variables shouldn't either.

>   If and when people start thinking about includes and functions,
>   scope becomes even more important.

let's address that when the need arises.  for now, consider SET to
implicitly declare the variable to have file scope.  add LOCAL or a
generic DECLARE action if you want them.  I think they are overkill
for Sieve, but we'll see.

>   * Numeric variables
>   
>   When does the side effect of tests have it's effect on numeric
>   variables? After the test? Only inside the if?

after the test.

as it happens, I just rewrote that part of the spec to cater for
:matches as well as :regex.  here's the new text:

3.2 Numeric variables

   The decimal value of the numeric variable name will index the list of
   matching strings from the most recently evaluated match of type
   ":matches" or ":regex".  The list is empty if the match was
   unsuccessful.

   For ":matches", the list will contain one string for each wildcard in
   the match pattern.  Each string holds what the corresponding wildcard
   expands to, possibly the empty string.  The wildcards expand
   greedily.

   For ":regex", the list will contain the strings corresponding to
   the group operators.  The groups are ordered by the position of the
   opening parenthesis, from left to right.

   The first string in the list has index 1.  If the index is out of
   range, the empty string will be substituted.  Index 0 returns the
   number of strings in the list.

   Example:
        require [ "fileinto", "regex", "variables" ];

        if header :regex "List-ID" "<(.*)@" {
             fileinto "lists.${1}"; stop;
        }

        # this is equivalent to the above:
        if header :matches "List-ID" "<*@" {
             fileinto "lists.${1}"; stop;
        }

        if header :matches [ "To", "Cc" ] "coyote@**.com" {
             # ${0} is always "2", and ${2} is always the empty string.
             fileinto "business.${1}"; stop;
        } else {
             # ${0} is always "0"
             stop;
        }

=== excerpt ends ===

>   Does the ${1} in the second test refer to the variable set in the
>   first test? Or does it refer to the previously set variable?
>   
>   if (allof (header :regex "Foo" "${1}(.*)", header :regex "Bar"
>   "${1}(.*)")) {
>   }

it refers to the variable set in the first test.

>   header :regex "Foo" "((a|b))*"?

the traditional thing, see above.

>   Does a failed matching clear all numeric variables?

yes.

>   * Program analysis
>   
>   I guess some of what's bothering me is that there are several
>   invasive language changes with this draft that make it
>   significantly more difficult to analyze the language to make sure
>   it isn't mutually contradictory, and more difficult to analyze a
>   piece of a program without global analysis.
>   
>   . Variable expansion can drastically change patterns to match
>   against, preventing certain pre-compilation strategies.

there is no doubt optimisation becomes harder, but in the end system
efficiency could improve since some long Sieve scripts now can be
expressed much more compactly.

-- 
Kjetil T.			|  read and make up your own mind
				|  http://www.cactus48.com/truth.html

---

From: Tony Hansen <[email-masked]>
Date: Tue, 08 Apr 2003 09:37:21 -0400

>     if not string "${timezone}" "US-Eastern" {

Isn't "X/Y", as in "US/Eastern", the accepted format for compound 
timezone names?

Question:

Should the date variables ALWAYS be preset without executing the setdate 
command?

Other languages have benefitted from namespaces. It might be worthwhile 
exploring that concept, particularly if there are preset variables. For 
example, instead of the preset variable name ${year}, you'd instead have 
${time.year}. Thoughts?

	Tony Hansen
	[email-masked]

---

From: Tony Hansen <[email-masked]>
Date: Tue, 08 Apr 2003 10:01:06 -0400

> Should the date variables ALWAYS be preset without executing the setdate 

Let me expand on this a bit. The idea was that the purpose of setdate 
changes to simply resetting the values to a different timezone. If you 
don't execute the setdate command, then the variables get preset to a 
value based on UTC.

     Tony Hansen
     [email-masked]

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 18:47:41 +0200

>   ${2} - String after the last a.

while this is easy to explain and understand, my original suggestion
was

   ${1} - String before the last a
   ${2} - "a"
   ${3} - String after the last a.

which I now realise may have merit when matching two lists against
eachother or when using the anyof test.

   if header :matches [ "To", "Cc" ] [ "foo@*.no", "bar@*.com" ] {
        redirect "kjetilho+${1}ifi.uio.no";
   }

the match order for [ a, b ] [ A, B ] should be aA, bA, aB, bB.  this
means that messages sent to both addresses will be forwarded to
<[email-masked]>, i.e. the first match string has precedence.
in this example, what the wildcard matches is irrelevant.

---

From: Tony Hansen <[email-masked]>
Date: Tue, 08 Apr 2003 13:02:42 -0400

Kjetil Torgrim Homme wrote:
>>  Question:
>>  
>>  Should the date variables ALWAYS be preset without executing the
>>  setdate command?
> 
> no.  you may consider the setdate action a different kind of
> "require".  the intent is that other actions can be added later to set
> variables, without breaking compatibility since the user has to call
> the action(s) explicitly.  (if setdate was an extension, we would have
> no way of specifying timezone, so there is good reason to make it an
> action.)
> 
>>  Other languages have benefitted from namespaces. It might be
>>  worthwhile exploring that concept, particularly if there are
>>  preset variables. For example, instead of the preset variable name
>>  ${year}, you'd instead have ${time.year}. Thoughts?
> 
> how should the namespace namespace be governed?  a registry?  only
> available to extensions specified in RFCs?

If setdate is comparable to a "require", then defining such an addition 
should define the namespace that it places its data into. Perhaps it 
should  be ${setdate.year}, ${setdate.month}, etc. More complex packages 
  might want to use more fully qualified names:

	require "example";
	setfoo ...;
	setbar ...;

	${example.foo.handle}
	${example.bar.contenttype}

to make certain that they don't conflict with the "foo" and "bar" 
namespaces of other "require" constructs.

Whoever defines a "require" extension that sets variables somehow would 
be responsible for defining the corresponding namespace used by its 
variables' names.

	Tony Hansen
	[email-masked]

---

From: Lawrence Greenfield <[email-masked]>
Date: Tue, 8 Apr 2003 13:16:01 -0400

From: Kjetil Torgrim Homme <[email-masked]>
   Date: 08 Apr 2003 06:27:05 +0200

>From RFC 3028, section 8.2:

   command = identifier arguments ( ";" / block )

   argument = string-list / number / tag

   arguments = *argument [test / test-list]

Your "set" action is implicitly extending 'argument' to allow for
something that is lexically identical to an 'identifier'. This
extension needs to be made explicit.

You then have to resolve the parsing ambiguity, since

set variable "string";

can be produced by 

command -> identifier arguments ; 
        -> identifier *argument ;
        -> identifier variable *argument ;
        -> identifier variable string-list *argument ;
        -> identifier variable string *argument ;
        -> identifier variable string ;

OR

command -> identifier arguments ;
        -> identifier *argument test ;
        -> identifier test ;
        -> identifier identifier arguments ;
        -> identifier identifier *argument ;
        -> identifier identifier string-list *argument ;
        -> identifier identifier string *argument ;
        -> identifier identifier string ;

which are obviously two different parse trees.

It's not clear to me. Nor is it clear that sites won't want to have
variables cascade.

[ ... explanations elided ... ]

I'm not convinced that Sieve-with-variables is a simple enough
language to explain without the use of formal semantics. I will
attempt to find a simple reference to post to the list.

Larry

---

From: [email-masked]
Date: Mon, 07 Apr 2003 19:23:02 -0700 (PDT)

> does this sound too complicated for too little gain?

I really don't see the point of storing the static parts of the pattern. It
will also confuse people familiar with capture systems for patterns, since
AFAIK none of them work this way.

So yes, I think this is too complex for too little gain.

> isn't it likely

Well, I for one intend to provide both, but regex will be disabled by default
and given some of the security risks I expect it will have I doubt if many
users of ours will be willing to enable it.

> hundred variables as minimum.

Well, it isn't clear to me that we want to encourage storing massive texts in
variables. I think the preferred approach is to use variables to store smallish
substitutions into larger texts.

Philosophy aside, I'm more concerned about limits on the all-digit variables
since they come into being automatically than I am on the limits on explicitly
specified variables. I could tolerate having a large limit on variables in
general as long as a smaller limit could apply to all-digit variables. Although
come to think of it, the base specification allows considerable leeway on
implementation limits, so I guess I could consider limits on :matches as an
implementation limit allowed by the base specification.

> +  scripts which include a require clause for the "variables"

This is what I meant.

> without updating the base specification?

No, nothing like that. I suppose it is possible for two extensions to
be incompatible in some way, but that's a bridge we can cross if we ever
come to it.

> let's stick to prefixing (function call like), then.

WFM, although there have been some other objections here...

> fit to fit other elements of Sieve (both syntax and semantics).

A copy was posted to the imapext list on 26-Feb. Hopefully Chris will
be able to post an updated version to the I-D directory soon.

>     SET :upper :comparator "i;ascii-casemap" name "${1}"

I like this a lot better than the present modifier syntax. You do need to
specify the combining rules for :upper/:lower/:upperfirst/:lowerfirst. I think
the rule should be that :upper and :lower are incompatiblei with each and so
are :upperfirst and :lowerfirst. And if :Xfirst is specified along with :Y, :Y
is done prior to :Xfirst.

> should be.  they may make more sense if other modifiers are added in

They would represent an identity case mapping I guess. I believe Chris plans to
deal with this issue in his document.

> take (a subset of) the Olson database and decree it as official.

The problem is that the Olson database is constantly being changed and updated,
which means any snapshot you take has a good chance of being at odds with
reality in fairly short order. (A political rather than technical base tends to
do this... I understand that there are 1-2 updates every month or so.) So
absent the wherewithall to basically have IANA act as a handler for the Olson
effort or something similar, I see serious problems here.

> whole vocabulary to define a time zone?  that seems very complex to

You can simply refer to it. No need to repeat it.

>     if not string "${timezone}" "US-Eastern" {

Interesting. I could go with this.

> which centralises all email servers in Brussels.  that must lead to

Then by all means make the zone one that's local to the user rather than to the
server. I really don't think having a default of UTC is the right answer here.

> +   The result of all calls to setdate MUST refer to the same point in

Works for me.

> the currentdate syntax is obviously easier to read in tests.

Yes, but harder to deal with in actions. I don't feel comfortable trying to
predict whether actions are going to be more common than tests.

> it allows general use without an IF and SET for every value needed,

Exactly.

>         vacation "I'm on my holidays, I'll be back on Sept. 19th";

Date would test date values in headers. There are all sorts of those, and
there is considerable value in being able to test them.

				Ned

---

From: [email-masked]
Date: Tue, 08 Apr 2003 10:23:11 -0700 (PDT)

> eachother or when using the anyof test.

Hmm. OK, I can see some utility here. Although I have to say this is getting
awfully tricky and I'd rather see such stuff done with two separate tests.

> in this example, what the wildcard matches is irrelevant.

Cute. I can live with it.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 18:29:10 +0200

>   Isn't "X/Y", as in "US/Eastern", the accepted format for compound

yes, that is the name used in Unix.  however, in the iCalendar
standard (2445) examples they use a dash.  reading more closely, they
do say

|      Note: This document does not define a naming convention for
|      time zone identifiers. Implementers may want to use the naming
|      conventions defined in existing time zone specifications such
|      as the public-domain Olson database [TZ]. The specification of
|      globally unique time zone identifiers is not addressed by this
|      document and is left for future study.

so going with the slash is probably better.  thanks.

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 8 Apr 2003 13:40:55 -0400 (EDT)

> It's not clear to me. Nor is it clear that sites won't want to have

Currently Cyrus's include draft has require being scoped to the current
script.  I think this is fine for features that only enable certain forms
of syntax (since they basically dissapear at parsing time), but is
somewhat less good for things that change global state of the interpreter.

Implicit keep is another "global state" issue that jumps to mind.

I think I'm currently of the opinion that all global state
(including variables, implicit keep, etc) should be global for the entire
execution.  However, the effects of require which only affect parsing
should be limited to the current script only.

I do think Larry is correct that there are many cases that will be hard to
understand without formal syntax.

-Rob

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski | Andrew Systems Group * Research Systems Programmer
PGP:0x5CE32FCC | Cyert Hall 207 * [email-masked] * 412.268.7456
-----BEGIN GEEK CODE BLOCK----
Version: 3.12
GCS/IT/CM/PA d- s+: a-- C++++$ ULS++++$ P+++$ L+++(++++) E W+ N o? K-
w O- M-- V-- PS+ PE++ Y+ PGP+ t+@ 5+++ R@ tv-@ b+ DI+++ G e h r- y?
------END GEEK CODE BLOCK-----

---

From: [email-masked]
Date: Tue, 08 Apr 2003 10:27:45 -0700 (PDT)

> This is central to the semantics of the language; this isn't a SHOULD.

Agreed. This has to be a MUST. I will also point out that this has some
interesting implications when it comes to having a body test, implications that
may cause me not to support the use of string substitutions into body tests.
(Basically I want to be able to do a single scan of the message that does all
the body checks at  the same time.)

> It should also be made clear that variables are expanded once, not

Yep. Thought of that one but forgot to mention it.

> identical to an "identifier" is not a valid argument.

I guess that means we have to quote it. Yuck, but we do what we have to do...

> theory has shown to be an extremely bad thing.

And as any Pascal programmer such as myself can tell you, the complexity
associated with scoped variables has been shown to be a bad then when it isn't
necessary.

> variables carry over from one to another? Why or why not?

I don't think they should carry over. As for why, it is because it makes
all sorts of unpredictable behavior very likely.

> After the test? Only inside the if?

I would say after.

> test? Or does it refer to the previously set variable?

Cute. My implementation currently does the former, but I have no strong opinion
here.

> header :regex "Foo" "((a|b))*"?

That IMO is an issue for the regex draft. As for how it should be handled
there, I think that we should find some set of consistent capture semantics
from some regular expression system we like and follow it. Does the
Posix regexp spec cover capture?

> Does a failed matching clear all numeric variables?

My personal preference would be "no".

> preventing certain pre-compilation strategies.

Agreed, but IMO the benefits far outweigh the disadvantages. Like it or
not, there is a large class of operations sieve currently cannot do and which
people really really really want to be able to do. I have to tell customers
"no, you cannot do that at present with sieve" at least a couple of times
every month, and that they have to use procmail or something similar instead.
This really isn't an acceptable state of affairs IMO.

				Ned

---

From: Jutta Degener <[email-masked]>
Date: Tue, 8 Apr 2003 11:22:51 -0700

> Cute. I can live with it.

I'm not happy.  Before this, I could stop matching when any combination
matched.  Now I have to either keep reading to the end just in case a
variable assignment changes, or figure out in advance whether any of
the strings following use any variables and keep reading to the end only
if that is true.

By the way, coupling variable assignments and matches _really_ makes
life interesting for anyone implementing streaming matches such as "body".

Without them, it's possible to at least predict a finite number of
possible variable values and match those in parallel in one pass.

With variables, you may not be able to predict what string a later
match will match against, because the match pattern itself may contain
variables that are set as the result of previous match patterns.

	if body :matches "*call(*,*)*",
	{
		if body :matches "*response(${3},${2})*"
		{
			...
		}
	}

These are extreme examples that don't affect the ability to parallelize
the more common fixed choices, but I'm not looking forward to the
extra complexity, and I don't think it's justified by functional
benefits.

Jutta

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 20:32:43 +0200

[Ned Freed]:
>
>   > while this is easy to explain and understand, my original
>   > suggestion was
>   >
>   > which I now realise may have merit when matching two lists
>   > against eachother or when using the anyof test.
>   
>   Hmm. OK, I can see some utility here. Although I have to say this
>   is getting awfully tricky and I'd rather see such stuff done with
>   two separate tests.

you're right, the added expressiveness is not worth the complication.
if N tests is deemed unacceptable by the user, he will probably find
that :regex is available in most implementations anyway.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 21:10:43 +0200

>   extension needs to be made explicit.

oh dear.  I've missed that entirely.

(it's not lexically identical, ${_foo} is not allowed, though that
should be changed for internal consistency.)

>           -> identifier variable *argument ;

so your grammar change here is

-   argument = string-list / number / tag
+   argument = variable / string-list / number / tag

?

>   which are obviously two different parse trees.

sorry, I don't follow this expansion.  does this assume a different
grammar change?

so, we have (at least) three options.

  a) change SET into taking two strings.

         SET "from" "${1}";

     the slightly nasty consequence of this is that the string
     specifying the variable itself can contain variable references.
     it is impossible to know at compile time whether the resulting
     variable name is valid.

  b) change SET into using a tag for the variable name.

         SET :from "${1}";

     this looks funny, and also creates a namespace problem if we add
     modifiers such as :upper.

  c) change the base grammar of Sieve.  my suggestion is

       - command = identifier arguments ( ";" / block )
       + command = 1*identifier arguments ( ";" / block )

     as far as I can tell, this does not introduce any ambiguity.

>   It's not clear to me. Nor is it clear that sites won't want to
>   have variables cascade.

isn't this an issue that should be dealt with in an "include"
extension or something?

thank you for your feedback!

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 21:28:33 +0200

[Rob Siemborski]:
>
>   Currently Cyrus's include draft has require being scoped to the
>   current script.

ah, I haven't seen that.  perhaps you could put it up at
http://www.cyrusoft.com/sieve/#documents ?

>   I think this is fine for features that only enable certain forms
>   (including variables, implicit keep, etc) should be global for the
>   entire execution.  However, the effects of require which only
>   affect parsing should be limited to the current script only.

if script one require variables, and set variable foo to bar, and
script two doesn't require variables, and contain the string "${foo}",
what should happen?  it seems to me you're saying that the string
should be left alone.  also, if script two does require variables, the
value of foo will become available, and the string will be expanded to
"bar".

this seems reasonable to me.
-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 21:47:58 +0200

[Tony Hansen]:
>   >>  preset variables. For example, instead of the preset variable
>   >>  name ${year}, you'd instead have ${time.year}. Thoughts?
>   >
>   > how should the namespace namespace be governed?  a registry?
>   > only available to extensions specified in RFCs?
>   
>   If setdate is comparable to a "require", then defining such an
>   addition should define the namespace that it places its data into.
>   Perhaps it should be ${setdate.year}, ${setdate.month}, etc.

this would work.  (there need not be a one-to-one relationship between
the name of the action/extension and the namespace, "date.year" would
work just as well.)

essentially, this is "only available to extensions specified in RFCs".
the question is if it's needed, the variable names themselves will be
carefully controlled and collisions can be avoided relatively easily.
the main benefit is that system variables look syntactically different
from user variables, and we could also make them read-only if we like.

it seems to me that namespaces are more useful if the user are allowed
to use them.  however, the user can do that, simply by imposing his
own rules for variable naming.  e.g., prefix all variables with "kj_".

>   More complex packages might want to use more fully qualified
>   names:
>   
>   	require "example";
>   	setfoo ...;
>   	setbar ...;
>   
>   	${example.foo.handle}

nice, but I think this is over-engineering, and most of the effect can
be had by substituting an underscore for the period.

-- 
Kjetil T.

---

From: Tony Hansen <[email-masked]>
Date: Tue, 08 Apr 2003 16:11:55 -0400

> value of foo will become available, and the string will be expanded to

I'm of the other mind, though. If script one sets ${foo}, I would NOT 
expect script two to see the value of ${foo}. I'd expect the variable 
state to be reset between scripts.

	Tony Hansen
	[email-masked]

---

From: Tim Showalter <[email-masked]>
Date: Tue, 08 Apr 2003 14:05:56 -0700

>glob patterns have to implement greedy matching when variables are used.
I implemented a mostly-linear time glob algorithm by implementing 
minimal matching left-to-right.  I am opposed to any semantic that would 
result in pattern matching being fairly expensive (specifically, any 
algorithm that results in * being recursive is bad).

If I had done minimal-matching right-to-left, would the result have been 
greedy?  (This would resolve my objection.)

Let me know if this doesn't make sense; I'm not an expert on this.

Tim

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 08 Apr 2003 23:08:27 +0200

[Jutta Degener]:
>   > > the match order for [ a, b ] [ A, B ] should be aA, bA, aB,
>   > > bB.  this means that messages sent to both addresses will be
>   > > forwarded to <[email-masked]>, i.e. the first match
>   > > string has precedence.  in this example, what the wildcard
>   > > matches is irrelevant.
>   > 
>   I'm not happy.  Before this, I could stop matching when any
>   combination matched.
>
>   Now I have to either keep reading to the end just in case a
>   variable assignment changes, or figure out in advance whether any
>   of the strings following use any variables and keep reading to the
>   end only if that is true.

okay, I see your problem.  so which match is returned should be
undefined?  that's fine with me.  if the user needs a specific
ordering, he can split the expression.

>   By the way, coupling variable assignments and matches _really_
>   makes life interesting for anyone implementing streaming matches
>   such as "body".
>

neat!

>   These are extreme examples that don't affect the ability to
>   parallelize the more common fixed choices, but I'm not looking
>   forward to the extra complexity, and I don't think it's justified
>   by functional benefits.

one sledgehammer is to declare that no variable expansion is performed
on match patterns.  it may seem a little arbitrary in the spec, and
indeed it can't be expressed in the grammar without _large_ changes to
it, but efficiency of implementation may trump expressiveness here.

since body isn't out as an RFC yet, an alternative approach is to
change _that_.  for instance, implementations can be allowed to refuse
searches with non-constant matching patterns.  (this can be determined
at script upload time, right?)

-- 
Kjetil T.

---

From: [email-masked]
Date: Tue, 08 Apr 2003 14:21:34 -0700 (PDT)

> the strings following use any variables and keep reading to the end only

Er, I never said that expressions shouldn't short circuit. I think they
should. But this issue, it seems to me, is independent of short circuiting.

The point here is that the saved non-wildcarded parts of the pattern can
be useful. Not that expressions must be fully evaluated. On the contrary,
if they are fully evaluated the non-wildcarded parts aren't nearly as useful.

> life interesting for anyone implementing streaming matches such as "body".

Coupling variable assignmens and matches doesn't change the problem in 
any material way. How about:

   set foo "bar"
   if header :contains "to" "foo"  {set foo "bletch"}
   if body :contains "${foo}"

So as I noted previously, the addition of variables and their use in body
effectively eliminates streaming matches.

> possible variable values and match those in parallel in one pass.

Yep. I plan to either preclude the use of variables and body or else
disallow variables in body arguments because of this.

> extra complexity, and I don't think it's justified by functional

The functional benefits of variables are enormous -- I receive many more
requests that can be satisfied with variables than I do for the ability
to search bodies.

				Ned

---

From: [email-masked]
Date: Tue, 08 Apr 2003 14:38:40 -0700 (PDT)

>      it is impossible to know at compile time whether the resulting

It isn't like there aren't all sorts of error conditions that can arise in
sieve scripts already. I have no problem with creating one more.

>      modifiers such as :upper.

Bad idea IMO.

>      as far as I can tell, this does not introduce any ambiguity.

While it isn't a matter of consequence to my own implementation, I really
dislike making such a change at this point.

I think (a) is the way to go.

> isn't this an issue that should be dealt with in an "include"

Exactly. There is a big difference between having multiple sieve scripts
and building up a single script from subscripts. Variables should not
cross from one to the other in the former case, they should in the latter
since it is purely a construction technique for producing a single
sieve script.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 09 Apr 2003 01:00:00 +0200

[Ned Freed]:
>
>   Well, it isn't clear to me that we want to encourage storing
>   massive texts in variables. I think the preferred approach is to
>   use variables to store smallish substitutions into larger texts.
>   
>   Philosophy aside, I'm more concerned about limits on the all-digit
>   variables since they come into being automatically than I am on
>   the limits on explicitly specified variables. I could tolerate
>   having a large limit on variables in general as long as a smaller
>   limit could apply to all-digit variables. Although come to think
>   of it, the base specification allows considerable leeway on
>   implementation limits, so I guess I could consider limits on
>   :matches as an implementation limit allowed by the base
>   specification.

I added this section to the draft:

6.  Implementation Limits

An implementation of this draft MUST support at least 128 distinct
variables.  The supported length of variable names MUST be at least 32
characters.  The supported size of a variable's content MUST be at
least 4095 octets.  Attempts to set the variable to a value larger
than the implementation supports MUST be treated as an error.

Numeric variables ${1} through ${9} MUST be supported.  Referencing
higher indices than is supported is a syntax error which MUST be
discovered at compile-time.  If the string matching a wildcard or a
regex group operator exceeds the maximum variable size, the
implementation SHOULD truncate it and MUST NOT treat it as an error.
=== ends ===

I hope this addresses your concerns.

>   > +  scripts which include a require clause for the "variables"

added, thanks.

>   be able to post an updated version to the I-D directory soon.

thanks, it's at http://article.gmane.org/gmane.ietf.imapext/1436

>   I like this a lot better than the present modifier syntax.  You do
>   need to specify the combining rules for :upper / :lower /
>   :upperfirst / :lowerfirst.  I think the rule should be that :upper
>   and :lower are incompatible with each and so are :upperfirst and
>   :lowerfirst.  And if :Xfirst is specified along with :Y, :Y is
>   done prior to :Xfirst.

so tags in Sieve MUST be commutative?

>   > what would it take to get it changed?  it seems to me they could
>   > just take (a subset of) the Olson database and decree it as
>   > official.
>   
>   The problem is that the Olson database is constantly being changed
>   and updated, which means any snapshot you take has a good chance
>   of being at odds with reality in fairly short order. (A political
>   rather than technical base tends to do this... I understand that
>   there are 1-2 updates every month or so.) So absent the
>   wherewithall to basically have IANA act as a handler for the Olson

ok.  my current wording:

  Note: There is currently no registry for time zones.  If IETF
  establishes one, its names SHOULD be used.  In the absence of such a
  registry, [TZ] is the most widespread collection of time zone
  definitions and its use as a reference is RECOMMENDED.

(where [TZ] refers to Olson et al. of course)

>   You can simply refer to it. No need to repeat it.

oh, I was worried about the implementation :-).  also, now the
responsibility for keeping the time zone definition up-to-date rests
on the Sieve client, not the server.  also, the time zone definition
could be implemented by the client using a bunch of IFs and a couple
of calls on SETDATE...

>   Interesting. I could go with this.

great :-)

>   Then by all means make the zone one that's local to the user
>   rather than to the server. I really don't think having a default
>   of UTC is the right answer here.

I'm with you, and these changes in SETDATE makes it easier to bear.

>   Date would test date values in headers. There are all sorts of
>   those, and there is considerable value in being able to test them.

ah...  yes, this is useful functionality, especially for filtering
email archives.  perhaps we could generalise SETDATE?

   SETDATE :now "+0200";	# old behaviour
   SETDATE :header "Date";
   SETDATE :header "Date" "+0000";

the default time zone for :header should be the one specified in the
header.

(what to do if there is no such header, or it can't be parsed?)
-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 09 Apr 2003 01:32:06 +0200

[Tim Showalter]:
>   minimal matching left-to-right.  I am opposed to any semantic that
>   would result in pattern matching being fairly expensive
>   (specifically, any algorithm that results in * being recursive is
>   bad).
>   
>   If I had done minimal-matching right-to-left, would the result
>   have been greedy?  (This would resolve my objection.)

yes.  just reverse both strings to see this.  here's an example:

"ababbab" ~ "*a*b?*" => [ "ab", "b", "a", "b" ]  (greedy)
"babbaba" ~ "*?b*a*" => [ "b", "a", "b", "ba" ]  (minimal)

or the other pair:
"babbaba" ~ "*?b*a*" => [ "babb", "a", "", "" ]  (greedy)
"ababbab" ~ "*a*b?*" => [ "", "", "a", "bbab" ]  (minimal)

---

From: [email-masked]
Date: Tue, 08 Apr 2003 14:56:17 -0700 (PDT)

> algorithm that results in * being recursive is bad).

I'm fairly sure that the greediness or non-greediness of the match doesn't
have to affect the efficiency of the operation. I've written both greedy
and non-greedy pattern matchers, and I believe all of them exhibit the
same sort of mostly-linear never-exponential behavior you describe.

(The trick I used in the greedy case, FWIW, is to scan the remainder of the
pattern and back off the initial match so it doesn't cover material needed to
satisfy the rest of the pattern.)

> greedy?  (This would resolve my objection.)

I don't think so. I think you have to tweak your approach.

> Let me know if this doesn't make sense; I'm not an expert on this.

The reason I prefer a greedy match is that it seems to be what most people have
come to expect. Capturing globs and regexps tend to be greedy by default.

				Ned

---

From: [email-masked]
Date: Tue, 08 Apr 2003 17:38:05 -0700 (PDT)

> "ababbab" ~ "*a*b?*" => [ "", "", "a", "bbab" ]  (minimal)

Yep, you're right. My mistake. Sorry.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 9 Apr 2003 12:34:17 +0200

>>    have been greedy? (This would resolve my objection.)

No. Consider the match for a*a in ababa: A minimal match takes aba no 
matter whether it's right-to-left or left-to-right, a greedy one takes 
ababa.

--Arnt

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 09 Apr 2003 14:21:58 +0200

[Arnt Gulbrandsen]:
>   No. Consider the match for a*a in ababa: A minimal match takes aba
>   no matter whether it's right-to-left or left-to-right, a greedy
>   one takes ababa.

doh!  I assumed anchored matching, sorry!

-- 
Kjetil T.

---

From: [email-masked]
Date: Wed, 09 Apr 2003 09:48:49 -0700 (PDT)

> doh!  I assumed anchored matching, sorry!

We're discussing sieve here, and sieve uses anchored matches for :matches.
While RFC 3028 doesn't have an explicit statement to the effect that :matches
is anchored, the only discussion of substring matches is specifically in the
context of :contains, and there are a couple of examples of the the form

    if header :matches "subject" "*make*money*fast*" ...

that don't make much sense if :matches is unanchored.

I also note that turning an anchored match into an unanchored match is
easy. (The opposite is not true -- another reason why sieve  matches need to be
anchored.)

We probably want to add an explicit statement when RFC 3028 is revised.

				Ned

---

From: Lawrence Greenfield <[email-masked]>
Date: Wed, 9 Apr 2003 14:47:17 -0400

Date: Tue, 08 Apr 2003 10:27:45 -0700 (PDT)
   From: [email-masked]

   Agreed, but IMO the benefits far outweigh the disadvantages. Like
   it or not, there is a large class of operations sieve currently
   cannot do and which people really really really want to be able to
   do. I have to tell customers "no, you cannot do that at present
   with sieve" at least a couple of times every month, and that they
   have to use procmail or something similar instead.  This really
   isn't an acceptable state of affairs IMO.

Whatever the benefits may be, they don't let us adapt a random-ass
approach to language design.

Formal operational semantics and careful thought on how to make
various features orthogonal would go a long way so we don't have every
specification referencing every other specification and subtly
altering their meanings.

Larry

---

From: [email-masked]
Date: Wed, 09 Apr 2003 14:30:19 -0700 (PDT)

> approach to language design.

Nor was I suggesting any such thing be done.

> specification referencing every other specification and subtly

As is always the case in the IETF, willingness to do the work counts for a lot.
I have no problem with someone doing a formal analysis here; in fact I would
welcome it. What I object to is making such analysis a requirement to
proceed.

As for careful thought, well, isn't that why we're discussing the various
details of this extension on the list right now?

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 10 Apr 2003 00:38:16 +0200

[Lawrence Greenfield]:
>   various features orthogonal would go a long way so we don't have
>   every specification referencing every other specification and
>   subtly altering their meanings.

no disagreement in general.  however, variables are usually a
first-class member of a language, so when we add this after the fact,
it is to be expected that most specifications have their functionality
altered (hopefully enhanced).

in my draft, this is done by changing the semantics of the basic type
string.  the alternative seems to be to make variable references a
construct only available when explicitly stated in the grammar for the
action, which in turn requires new actions and tests to be defined
whereever variables might be useful.  I obviously favour the first
approach.  it may be more painful now, but I believe it will pay off
in the long term.

now, how to do it?  in my first attempt at a draft, I changed the
grammar of quoted-string, multi-line-literal and multi-line-dotstuff.
this does seem more invasive, but it may also seem more honest.

basically, it was done like this:

       quoted-string  =  DQUOTE *(CHAR / variable-ref) DQUOTE
        variable-ref  =  "${" variable-name "}"

which is bogus, of course.  doing it right is actually quite hard (at
least for me), I don't see how to support the current syntax in the
draft without a _lot_ of grammar rules.  actually, I don't see how to
do it at all :-)

here's an attempt at a slightly simplified syntax:

  quoted-string = DQUOTE
                  *(qcontent / quoted-pair / variable-ref /
                    verbatim-dollar / "{")
                  *"$"
                  DQUOTE

        qcontent  =  %x01-21 / %x23 / %x25-5b / %x5d-%x7a / %x7c-1fffff
                ; all characters except NUL, double-quote, dollar,
                ; backslash and opening brace
     quoted-pair  =  "\" %x01-1fffff
    variable-ref  =  "${" variable-name "}"
   variable-name  =  num-variable / identifier
    num-variable  =  1*DIGIT
 verbatim-dollar  =  1*"$" (qcontent / quoted-pair / variable-ref)

the verbatim-dollar makes sure that "$${foo}" and "$\"bar" are allowed
and work as expected.  since there has to be a non-dollar character in
the verbatim-dollar expansion, strings ending in one or more dollar
characters are handled by an explicit *"$" just before the closing
quote.

using this syntax, "${!}" gives a parsing error rather than being left
verbatim.  backslash are processed differently than in my draft, a
backslash will now actually escape the variable reference, which I
think is an improvement.  "\${foo}" will parse as a quoted-pair ("\$")
followed by a "{" and "foo}".

the above syntax can be simplified further by requiring all dollars
outside variable references to be escaped using backslash.

  quoted-string = DQUOTE
                  *(qcontent / quoted-pair / variable-ref)
                  DQUOTE
        qcontent  =  %x01-21 / %x23 / %x25-5b / %x5d-1fffff
                ; all characters except NUL, double-quote, dollar and
                ; backslash

other expansions are the same as above.  strings like "$$$" and
"\\.(.*)$" now yield errors.  most users will use Sieve generators
that probably escape every dollar _anyway_ to keep the implementation
simple, so being non-intrusive in general texts doesn't seem
important.  the regexp gotcha may be the worst.

it might be useful to change string as well:

          string  =  quoted-string / squoted-string / multi-line
  squoted-string  =  "'" *(sqcontent / quoted-pair) "'"
       sqcontent  =  %x01-26 / %x28-5b / %x5d-1fffff
                ; all characters except NUL, single quote and backslash

that way the body extension can mandate the use of squoted-string for
specifying the match pattern.  yes, dependencies are tricky here.

...

btw, it seems this needs fixing in a Sieve revision:

   quoted-string  =  DQUOTE *CHAR DQUOTE
            CHAR  =  %x01-7F ; from [ABNF]
similarily:
    CHAR-NOT-DOT  =  (%x01-09 / %x0b-0c / %x0e-2d / %x2f-ff)
           ;; no dots, no CRLFs
and others doesn't include the complete Unicode repertoire.

also, CHAR includes the double quote, so "foo" "bar" could be parsed
as one quoted-string with contents ``foo" "bar''.

-- 
Kjetil T.

---

From: Marc Mutz <[email-masked]>
Date: Thu, 10 Apr 2003 20:24:21 +0200

On Tuesday 08 April 2003 21:10, Kjetil Torgrim Homme wrote:
<snip>
>      it is impossible to know at compile time whether the resulting
<snip>

Suppress/disallow variable expansion in the first arg of SET?

Marc

-- 
The first casualty of war is the truth.

---

From: Marc Mutz <[email-masked]>
Date: Thu, 10 Apr 2003 21:06:24 +0200

On Tuesday 08 April 2003 19:27, [email-masked] wrote:
<snip>
> That IMO is an issue for the regex draft. As for how it should be
> handled there,
<snip>

On a related note, I think the regex extension should include 
non-capturing parentheses: "make (?:money|dollars)"

Esp. in the light of the variables extension, which makes formerly 
non-capturing () capturing now. As everyone here will know, capturing 
in regexes imposes a non-negligible performance penalty on regex use, 
which is a critical component even without capturing.

Of course, this may mean that we can't use the posix regex spec as 
reference anymore. How about the ecmascript one? With libpcre being 
available under a very liberal license, I don't see a bigger 
implementation burden than with posix regexes...

Marc

-- 
"You're hackers, aren't you," the barman said, eyeing us. No one said
a thing. The darkness of the Eurotunnel rolled by. Apparently we'd
given ourselves away by talking too enthusiastically about IPv6. He
looked around conspiratorially, lowered his voice. "Can you get me
some credit card numbers?"
      -- James J. King "What's the shortest way to hack a Linux box?"
         Telepolis 2001/08/11 (#9293)

---

From: Jutta Degener <[email-masked]>
Date: Thu, 10 Apr 2003 13:36:18 -0700

On Thu, Apr 10, 2003 at 09:06:24PM +0200, Marc Mutz wrote:
Content-Description: signed data
> non-capturing parentheses: "make (?:money|dollars)"

Why not just omit capturing if the corresponding variable isn't used?

Jutta

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 11 Apr 2003 00:50:38 +0200

[Marc Mutz]:
>   [Kjetil Torgrim Homme]:
>   >   specifying the variable itself can contain variable
>   >   references.  it is impossible to know at compile time whether
>   >   the resulting variable name is valid.
>   

suppress makes the parser quite hairy unless we add a single quoted
string or similar.  disallow could work, but we still need to handle
the same syntax errors.  we only remove the ability to do indirect
setting of variables.  this isn't a very useful ability, since there
is no corresponding indirect lookup of variables, but I'm sure it can
be exploited somehow...

-- 
Kjetil T.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 11 Apr 2003 16:02:46 -0600

In order to move imapflags extension forward I need support for set/list variables.
I haven't thought too much how this is going to be implemented (so don't blame me if
I say something stupid ;-)), but here are some preliminary thoughts and questions:

1). Special syntax for list variables?
2). Can they be expanded inside strings? If yes, what does it mean?
3). Need new actions to add/remove values to the list:

addhead <listvariable> <value(s)>
addtail <listvariable> <value(s)>
remove <listvariable> <value(s)>

4). Need a way to convert a list variable to a string form (conversion in other
direction is easily achievable with existing capabilities: set listvar ["${var1}",
"${var2}", ...];)

New command:
 asstring <identifier> <listvariable> <delimeter_str>

<delimeter_str> defines the delimiter that would be inserted between values of the
listvariable.

Alexey
__________________________________________
R & D, ACI Worldwide/MessagingDirect
Watford, UK

Work Phone: +44 1923 81 2877
Home Page: http://orthanc.ab.ca/mel

I speak for myself only, not for my employer.
__________________________________________

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 13 Apr 2003 17:53:40 +0200

>   In order to move imapflags extension forward I need support for

I've read your draft, but I don't see why.  I'm currently reading the
complete archive for this list to get a better feel for the philosophy
of Sieve, but it takes some time :-)

>   I haven't thought too much how this is going to be implemented (so
>   don't blame me if I say something stupid ;-)), but here are some
>   preliminary thoughts and questions:
>   

the first two operate on a list, but remove operates on a set.

it has occured to me that

  SET "foo" ["hi", "there"];

seems a natural thing to write, which perhaps should be supported in
some way.  I'm not sure what it should be used for, though.

if we want to go for generality, it may be a good idea to offer a
associative array/hash/dictionary rather than an indexed list.  after
all, the hash can be used as if it was an indexed list, AWK does this.

  SET "foo" :index "bar" "zot";
  "${foo{bar}}" => "zot"

>   4). Need a way to convert a list variable to a string form
>   (conversion in other direction is easily achievable with existing
>   capabilities: set listvar ["${var1}", "${var2}", ...];)
>   
>   New command:
>   <delimeter_str> defines the delimiter that would be inserted
>   between values of the listvariable.

if the default delimiter is the empty string, the user can add
whatever delimiter he likes in the list.  in AWK, there is a magic
global variable OFS, output field separator, to control this
behaviour.

in my current draft, a variable can only be expanded inside a string.
clearly, this expansion can not result in a list of strings.  or can
it?

   SET rcpt [ "Alice", "Bob" ];
   "hello ${rcpt}, nice to see you" =>
        [ "hello ", "Alice", "Bob", ", nice to see you" ]

just some quick thoughts.

-- 
Kjetil T.

---

From: Jutta Degener <[email-masked]>
Date: Mon, 14 Apr 2003 09:18:16 -0700

> >   In order to move imapflags extension forward I need support for

I think we might be able to do without this.  What if you allow
(and ignore) empty strings in a [...] list?

That way, you could set

	set "aflag" "aflag"

	["${aflag}", "${bflag}", "${cflag}"]

and have that come out as

	["aflag" "" ""]

if bflag and cflag are unset.

There are still things that you can do with a list algebra
that you can't do with this approach -- infinitely many
flags based on the header and body contents of a message --
but I don't think we want to do those things badly enough
to warrant adding three new commands and a variable data type.

Jutta

---

From: [email-masked]
Date: Mon, 14 Apr 2003 10:15:23 -0700 (PDT)

> to warrant adding three new commands and a variable data type.

I agree with this assessment. The added complexity isn't worth it.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 Apr 2003 20:38:17 -0600

> to warrant adding three new commands and a variable data type.

Surely three new actions shouldn't scare you :-).
But I understand the complexity of multiple data types.

Ok, I can live with this.
It seems the only thing required to be said is that empty strings are
ignored in a list that specifies flags to be set.
I should also mention that each list member defines a separate flag and
spaces in flag names are not allowed (due to IMAP restriction).

Cheers,
Alexey

Work Phone: +44 1923 81 2877
Home Page: http://orthanc.ab.ca/mel
IETF standard
related pages: http://orthanc.ab.ca/mel/devel/Links.html

---

From: [email-masked]
Date: Wed, 16 Apr 2003 13:30:51 -0700 (PDT)

Folks, I think we have reached a point where a revised I-D for the
variables document is in order. I'm starting to lose track of where
we are on some of these changes...

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 17 Apr 2003 00:44:57 +0200

[Ned Freed]:
>   variables document is in order. I'm starting to lose track of
>   where we are on some of these changes...

I fully agree, but I'd like some feedback on a very important issue
first, the syntax vs. semantics issue.

semantics seems the easier bolt-on, but syntax is the smoothest change
IMO.  in particular, the backslash handling feels more natural.
consider "\${foo}".  if we change syntax, the backslash will quote the
dollar so that it isn't a variable reference.  if we change semantics
of strings, the backslash has no effect.

-- 
Kjetil T.

---

From: Jutta Degener <[email-masked]>
Date: Wed, 16 Apr 2003 16:25:55 -0700

> of strings, the backslash has no effect.

The gut reaction of a shell programmer will probably differ,
but I'd rather not change the syntax.  To me, string scanning
and variable extension are different stages and must not
affect each other.

If you do change the syntax, where do you stop?  Why should
\ be special for variable syntax, but not for :regex and
:matches pattern syntax?  I don't want to have to answer that.

Jutta

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 17 Apr 2003 01:46:40 +0200

[Jutta Degener]:
>   > semantics seems the easier bolt-on, but syntax is the smoothest
>   > change IMO.  in particular, the backslash handling feels more
>   > natural.  consider "\${foo}".  if we change syntax, the
>   > backslash will quote the dollar so that it isn't a variable
>   > reference.  if we change semantics of strings, the backslash has
>   > no effect.
>   
>   The gut reaction of a shell programmer will probably differ, but
>   I'd rather not change the syntax.  To me, string scanning and
>   variable extension are different stages and must not affect each
>   other.

duly noted.

>   If you do change the syntax, where do you stop?  Why should \ be
>   special for variable syntax, but not for :regex and :matches
>   pattern syntax?  I don't want to have to answer that.

interesting point!  I'm not sure I want to explain that, either :-)

-- 
Kjetil T.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 17 Apr 2003 09:57:34 -0600

> and variable extension are different stages and must not

I fully agree with this.

Alexey

---

From: [email-masked]
Date: Wed, 16 Apr 2003 16:36:03 -0700 (PDT)

> of strings, the backslash has no effect.

I'm personally in the semantics camp. Smooth changes are good.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 19 Apr 2003 01:21:53 -0600

> spaces in flag names are not allowed (due to IMAP restriction).

After I rewrote my example in "imapflags" I've realised how ugly this will
look, because
1). The script author has to remember all flags that were set on the
message and specify them all.
2). setflags has to be replaced with multiple "set"s as we have to clear
all flags.

So, I am thinking now about using a single string parameter (instead of a
string list) that contains space delimited flags. Adding flags (old
"addflags") to the list would be simpler, e.g.

set "flags" "${flags} \\Seen"

"removeflags" would be trickier to implement, but "string" test with
:matches can be used to test if a flag was set, e.g.

if string :matches "${flags}" "*\\Draft*" {
    set "flags" "${1}${2}"
}

Multiple spaces between flags must be treated as one.

Comments?

Alexey

---

From: [email-masked]
Date: Sat, 19 Apr 2003 04:54:09 -0700 (PDT)

> set "flags" "${flags} \\Seen"

I was wondering if we'd eventually get to this. I'm not delighted by it, but
I think it probably is the best solution to the problem.

> if string :matches "${flags}" "*\\Draft*" {

Very cute. How to folks who precompile all this stuff feel about allowing
this sort of variable pattern?

				Ned

---

From: Jutta Degener <[email-masked]>
Date: Mon, 21 Apr 2003 07:44:12 -0700

> > message and specify them all.

Hm!  I thought of that as an advantage, not a downside.

> > 2). setflags has to be replaced with multiple "set"s as we have to clear

Similarly, I don't think that it's so bad that we get a list
of what flags are available that way.

Maybe we have very different applications in mind.  To me, the range
of flags is severely limited by the ability of clients to display
them and use them in the interface, and by the ability of users to
remember and use them.

Can you describe a typical application of your multitude of flags?

> I think it probably is the best solution to the problem.

I'd like to get a better understanding of the problem first.

This is introducing a second list type with roughly the same function
as the existing one, making sieve in the long term more expensive to
implement and harder to understand; because now, every time you specify
a list, you'll have to ask yourself whether it was type A or type B.

> > if string :matches "${flags}" "*\\Draft*" {

The mistake in this expression -- it'll match \Draft as well
as \DraftBeer -- will be made many times over, and mostly without
harm, in scripts all over the world.  It can be fixed by using trailing
white space in both the matches and the flags string, but I don't
think most authors will remember to do that.

> this sort of variable pattern?

Well, you can't (precompile all this stuff), so what's the use.

Jutta

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 21 Apr 2003 22:44:21 -0600

[I've modified the subject once again to reflect our current discussion].

> Hm!  I thought of that as an advantage, not a downside.

See below.

> Can you describe a typical application of your multitude of flags?

I am think from the point of view of a user writing scripts by hand, not scripts
generated by UI.
Also, I am thinking more in IMAP terms, i.e. each script is a list of blocks like
this:

if <condition> add/remove <flag>;

I don't want an end user to require to remember at each point in the script which
flags are to be set on the message.
IMHO, this reduces script complexity and, as a result, user errors.

> a list, you'll have to ask yourself whether it was type A or type B.

I don't think I share your concern. If you are afraid that this will open gates to
multiple Sieve extensions that misuse strings,
Sieve mailing list exists not to let such extensions be standardized.

All functionality required is already available in the base Sieve document or the
variable draft. So, my proposal doesn't make it harder to implement.

Besides, if you want to be explicit, you can still use "${flag1} ${flag2}
${flag3}" construct instead of specifying a single variable.
So, UI-generated scripts are still possible with my proposal.

And yes, I grew up as a Pascal programmer as well and I miss set types and set
related operations ;-).

> as \DraftBeer

Good catch.

> -- will be made many times over, and mostly without
> harm, in scripts all over the world.

Hopefully, if a correct example is shown in the document, everybody will just
cut&paste it.

> It can be fixed by using trailing

Regards,
Alexey