Subject: variables: interaction with "require"

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Apr 2003 04:55:38 +0200

this can happen:

  require "variables";
  require "file${foo}into";

according to current spec, this should work, since ${foo} expands to
the empty string.  how about

  require [ "variables", "file${foo}into" ];

should it work?  and how about

== main script ==
  require [ "include", "variables" ];
  set "ext" "fileinto";
  include "/personal/test";

== /personal/test ==
  require "variables";
  require "${ext}";
  fileinto "INBOX";

this must not work, since each included script must be parsable on its
own (according to the current include draft).  but if you change it
into

== /personal/test ==
  require "variables";
  require "file${ext}into";
  fileinto "INBOX";

it will work on its own, but bomb when included from the main script.

I don't see how variable expansion in the arguments to require can
ever be useful.  I suggest adding to the introduction:

  The mechanisms detailed in this document will only apply to Sieve
  scripts which include a require clause for the "variables" extension.
+ The require clauses themselves are not affected by this extension.

-- 
Kjetil T.

---

From: [email-masked]
Date: Sun, 13 Apr 2003 22:20:56 -0700 (PDT)

> + The require clauses themselves are not affected by this extension.

I agree with this proposed solution to the problem.

				Ned

---

From: Jutta Degener <[email-masked]>
Date: Mon, 14 Apr 2003 09:22:31 -0700

On Sun, Apr 13, 2003 at 10:20:56PM -0700, [email-masked] wrote:
> [Kjetil] 

Now that the door has been opened to "strings that don't
have variables expanded in them", can we consider using
the same kind of strings as variable names in "set"
commands and as match patterns?

(So they can be precompiled and don't change contents
during program execution.)

Jutta

---

From: [email-masked]
Date: Mon, 14 Apr 2003 10:16:07 -0700 (PDT)

> commands and as match patterns?

I've thought about this and I believe I like it. I assume you're talking
about any match argument anywhere, not just when :matches or :regex
are in use. (The issue with body isn't addressed unless you do this.)

This needs to apply to the match argument in the string test as well. It cannot
apply to the source argument or the test becomes worthless.

> during program execution.)

Yep. Seems like a good idea to me.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Apr 2003 20:48:34 +0200

[Ned Freed]:
>
>   [Jutta]:
>   > Now that the door has been opened to "strings that don't have
>   > variables expanded in them", can we consider using the same kind
>   > of strings as variable names in "set" commands and as match
>   > patterns?
>   
>   I've thought about this and I believe I like it. I assume you're
>   talking about any match argument anywhere, not just when :matches
>   or :regex are in use. (The issue with body isn't addressed unless
>   you do this.)

REQUIRE is a special case in that it is only allowed as the first
action in a script.  it therefore makes sense that all REQUIRE are
processed before any of them take effect.  (this is also a useful rule
for resolving dependencies among extensions if that should occur in
the future.)

the parser/lexer will know which strings contain variable references,
and can easily set a boolean for "constant/non-constant".  I suggest
we use terminology like "<argument> must be a constant string without
variable references".  this makes "${foo}" an error, not a verbatim
value, in such a context.

>   This needs to apply to the match argument in the string test as
>   well. It cannot apply to the source argument or the test becomes
>   worthless.
>   

why can't you optimise it only if the string is static?  I can see
some utility in being able to match against a previous result.

  require [ "variables", "fileinto" ];
  if header [ "List-ID" ] :matches "*<*@*>*" {
     set "listaddr" "${2}@{3}";
     set "listname" "${2}";
     if address :is :all [ "to", "cc" ] "${list}" {
        fileinto "INBOX.lists.${listname}";
     } else {
        fileinto "INBOX.possibly-spam";
     }
  }

-- 
Kjetil T.

---

From: [email-masked]
Date: Mon, 14 Apr 2003 12:04:32 -0700 (PDT)

> value, in such a context.

Good point.

>         fileinto "INBOX.possibly-spam";

Fair enough, but the case that breaks the bank is body. You really really
really want to do all your body search operations in a single passs. If you
don't you can easily write scripts that have serious performance problems.

I don't especially care about the case of optimizing the header, address,
envelope, and string tests, but body is a serious problem. And it seems fairly
awkward to say that the match parameter doesn't get expanded in body but
does elsewhere.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Apr 2003 21:42:53 +0200

[Ned Freed]:
>
>   Fair enough, but the case that breaks the bank is body. You really
>   really really want to do all your body search operations in a
>   single passs. If you don't you can easily write scripts that have
>   serious performance problems.

agreed.

>   I don't especially care about the case of optimizing the header,
>   address, envelope, and string tests, but body is a serious
>   problem. And it seems fairly awkward to say that the match
>   parameter doesn't get expanded in body but does elsewhere.

how about:

(somewhere at the start of the specification:)
  A constant string is a string or multi-line value without variable
  references.

(in a section on interactions with old and new mechanisms?)
  Implementations MAY require match patterns to be constant strings
  when used with some test types.  Such implementations MUST reject
  the activation of scripts with disallowed match patterns.

is it too strong to require the error to be discovered at upload time?
I don't feel good about allowing it to happen during run-time, but
that might be a quality-of-implementation issue.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 14 Apr 2003 22:19:17 +0200

[Jutta Degener]:
>
>   Hm.  That's a reasonable offer, Kjetil, but I'd rather eat the
>   complexity as an implementor than to place it on the users and
>   client writers by making this be an implementation-specific
>   difference.

well, that is one side of it.  the other side is that of the server
administrators.  if you must support multipass searches in body, this
may be a performance hit for servers.  the administrator has few
options: pray that people won't use it too often, buy more hardware
(RAM!), or turn off the body extension altogether.

-- 
Kjetil T.

---

From: Jutta Degener <[email-masked]>
Date: Mon, 14 Apr 2003 13:04:54 -0700

> that might be a quality-of-implementation issue.

Hm.  That's a reasonable offer, Kjetil, but I'd rather eat
the complexity as an implementor than to place it on the users
and client writers by making this be an implementation-specific
difference.

Jutta

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 14 Apr 2003 16:43:36 -0400

Hi Kjetil,

--On Monday, April 14, 2003 10:19 PM +0200 Kjetil Torgrim Homme 
<[email-masked]> wrote:

|>   Hm.  That's a reasonable offer, Kjetil, but I'd rather eat the
|>   complexity as an implementor than to place it on the users and
|>   client writers by making this be an implementation-specific
|>   difference.
|
| well, that is one side of it.  the other side is that of the server
| administrators.  if you must support multipass searches in body, this
| may be a performance hit for servers.  the administrator has few
| options: pray that people won't use it too often, buy more hardware
| (RAM!), or turn off the body extension altogether.

Sadly we have already seen multiple instances of ISPs disabling the IMAP 
SEARCH command because they feel that is too 'expensive' CPU-wise and 
adversely effects performance. I would guess the same philosophy will be 
applied to body searches in sieve.

-- 
Cyrus Daboo

---

From: Jutta Degener <[email-masked]>
Date: Tue, 15 Apr 2003 15:42:03 -0700

> (RAM!), or turn off the body extension altogether.

I strongly feel that performance should be addressed on a global
level, not on the level of individual features.

An ISP might want to restrict, and a vendor of tools might offer
a way to restrict, the amount of time and memory used by a sieve
script's execution (or by an IMAP search, for that matter).
That's a good and safe thing to do.

Disabling individual features is a stopgap measure in the absence
of better tools, but while desperate ISPs will do it, we shouldn't
do it at the specification level.

(Neither should we specify mechanisms that are needlessly wasteful,
but I don't think these are.)

Jutta