Subject: Comments on draft-homme-sieve-variables-01.txt

From: Simon Josefsson <[email-masked]>
Date: Fri, 18 Apr 2003 15:16:50 +0200

* The setdate command is useful, but wouldn't it be even more useful
  if there was a command that extracts comparable time/date variables
  from Date: headers?  To handled obs-zone's timezones.  E.g.,
  consider if I want to sort messages sent (Date:) between 01 am and
  04 am (in the senders timezone) into a folder INBOX.rants.

  This would also make it easier for script writers to do more robust
  time-based filtering based on Date:, since the server could also
  contain functions for handling non-conforming but easily guessable
  times, if they want to support that.

  Looking only at the posts to this list shows people are still using
  obs-zone, and if people on this list uses such MUAs, chances are
  rather high that many people in the real world also does.

* A security consideration is that people should be aware that RFC
  2047 encoded-word, IDNA etc potentially encode information that the
  script wants to trigger on.  This should have been in RFC 3028, but
  as variables makes it even more likely that people will try to use
  sieve scripts to filter for virus etc it might be useful to mention
  it.  Compare the sieve-body draft:

   Filters relying on string matches in the raw body of an e-mail
   message may be more general than intended.  Text matches are no
   replacement for a virus or spam filtering system.

* The examples in Section 3.1 and 4.1.1 should be terminated with
  semi-colons to be complete Sieve commands.

* I feel variable variable names (e.g., ${foo-${bar}-baz}) would be
  useful, but I can't come up with a good example why it is really
  required so I'm just mentioning it if someone else can come up with
  a illustrative example.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 18 Apr 2003 16:11:28 +0200

[Simon Josefsson]:
>     if there was a command that extracts comparable time/date
>     variables from Date: headers?  To handled obs-zone's timezones.
>     E.g., consider if I want to sort messages sent (Date:) between
>     01 am and 04 am (in the senders timezone) into a folder
>     INBOX.rants.

like
  SETDATE :header "Date";
vs. just
  SETDATE;
to access the current time.

Ned, you had some ideas for more date related tests, what do you
think?

it seems to me that the only interesting data to be found in Date: is
the local time and time zone of the sender (I feel SETDATE should
normalise the timezone to offset format if it recognises it).  the
date itself should be no more than a few minutes earlier than current
time.  if it isn't, there is a misconfiguration or an outage going on.

oh!  if you run the Sieve script in batch, long after the delivery was
made, the current date is completely useless.  SETDATE should be
changed to initialise the variables to the time of delivery, not
current time.

>      replacement for a virus or spam filtering system.

I'd appreciate it if you could formulate a paragraph for me :-)

>     semi-colons to be complete Sieve commands.

thanks.

>     required so I'm just mentioning it if someone else can come up
>     with a illustrative example.

it could be used as a poor man's associative arrays.  there'd be no
function to list all keys in the "array", though.  if we do this, we
need a way of escaping the dollar character other than the ${dollar}
hack.  I don't think it is worth it.  if we want to support
associative arrays, let's do it explicitly.

-- 
Kjetil T.

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 18 Apr 2003 18:57:39 +0200

>>     01 am and 04 am (in the senders timezone) into a folder

Yes.  The string must be allowed to be a variable too, so you can
extract times from Received:, etc.

> time.  if it isn't, there is a misconfiguration or an outage going on.

The time in Date: are in general not the same as delivery time.
Differences of several hours and days are typical.

> changed to initialise the variables to the time of delivery, not

What is the difference between delivery time and current time?

Consider the batch example again, what would SETDATE be initialized to
in this case?  The delivery time is in general not available from a
RFC 2822 message, so the implementation would probably simply pick
current time.  A delivery agent invoked by the MTA would also pick
current time for this, since current time is delivery time in a
delivery agent.  I'm probably just dense, but I don't see a
practically useful distinction between delivery and current time.

> I'd appreciate it if you could formulate a paragraph for me :-)

Someone who knows english might be able to make the following legible:

    Carelessly constructed tests may be both more general than
    intended and also less general than intended.  This is a generic
    problem with Sieve, but may become more important when filtering
    decisions are made based on variables initialized from data in
    message.  The moral of the story is that Sieve tests are not a
    replacement for intelligent spam and virus detection mechanisms,
    rather a solution that is intended to work most of the time that
    can be circumvented by adaptive senders.

    For an example of a more general than expected test, consider
    looking for "xn--foo-bar" within a To: addr-spec in order to
    filter message addressed to a specific internationalized domain
    names into a separate folder, and an incoming message having an
    addr-spec of someone@(xn--foo-bar)example.org.  For an example of
    a less general than expected test, consider looking for Subject:
    containing "MAKE MONEY FAST" to filter spam into a separate
    folder, and an incoming message having a Subject: of
    "=?CP-1252?q?MAKE_MONEY_FAST?=.

That reminds me, Sieve should support internationalized domain names
by decoding them when they occur at the appropriate places (dot-atom's
inside domain inside addr-spec, for instance).

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 19 Apr 2003 04:14:00 +0200

[Simon Josefsson]:
>   > the date itself should be no more than a few minutes earlier
>   > than current time.  if it isn't, there is a misconfiguration or
>   > an outage going on.
>   

you should get a better email provider :-)

huge mailing lists may introduce a delay of half an hour.  several
hours is simply not an acceptable delay, and certainly not if it is
typical.

>   > oh!  if you run the Sieve script in batch, long after the
>   > delivery was made, the current date is completely useless.
>   > SETDATE should be changed to initialise the variables to the
>   > time of delivery, not current time.
>   
>   Consider the batch example again, what would SETDATE be
>   initialized to in this case?  The delivery time is in general not
>   available from a RFC 2822 message, so the implementation would
>   probably simply pick current time.

the latest Received header is a good bet.  the final local delivery
may be held up, but should be very uncommon.  in the current mail
system of my University this can happen since the spool disk (my home
directory) is accessed through NFS.  these days, delivery to local
storage is becoming the typical mode of operation.

>       intended and also less general than intended.  This is a
>       generic problem with Sieve, but may become more important when
>       filtering decisions are made based on variables initialized
>       from data in message.  The moral of the story is that Sieve
>       tests are not a replacement for intelligent spam and virus
>       detection mechanisms, rather a solution that is intended to
>       work most of the time that can be circumvented by adaptive
>       senders.

all right, I'm not a native speaker, but how about

   Sieve filtering should not be relied on as a security measure
   against hostile e-mail messages.  Sieve is designed to do simple,
   mostly static tests, and is not suitable for use as a spam or virus
   checker, where the perpetrator has a motivation to vary the format
   of the email in order to avoid filtering rules.

>       filter message addressed to a specific internationalized
>       domain names into a separate folder, and an incoming message
>       having an addr-spec of someone@(xn--foo-bar)example.org.

if you use :address for matching, this won't be a problem.  if you use
:header, there are more likely modes of failure.  e.g.,

   if :header :matches [ "To", "Cc" ] "root@*.example.com"

and a message with "To: [email-masked], [email-masked]"

although this may be a common pitfall, I'm not sure warnings to "use
the appropriate test" fits in here.

>       For an example of a less general than expected test, consider
>       looking for Subject: containing "MAKE MONEY FAST" to filter
>       spam into a separate folder, and an incoming message having a
>       Subject: of "=?CP-1252?q?MAKE_MONEY_FAST?=.

I believe that message should be filtered just fine.

>   That reminds me, Sieve should support internationalized domain
>   names by decoding them when they occur at the appropriate places
>   (dot-atom's inside domain inside addr-spec, for instance).

I think that is implied, really.  it does pose a problem for clients,
they can't know whether the server supports it or not, so they need to
duplicate their tests.  hmm, perhaps we do need an extension for it.

-- 
Kjetil T.

---

From: Philip Guenther <[email-masked]>
Date: Fri, 18 Apr 2003 20:57:58 -0700

>you should get a better email provider :-)

Not at all.  His provider has no control over the Date: header field as,
for the vast majority of messages, that header field was added by either
the sender's MUA or the MSA.  Desktop machines are often unsynchronized
and may have never been given the correct date, much less time.

Philip Guenther
<[email-masked]>

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 19 Apr 2003 12:49:52 +0200

> hours is simply not an acceptable delay, and certainly not if it is

Moderated lists are common, and if the moderator is out of town for a
week, the Date: will be a week late.

Offline clients are also common, and the typically put the current
time of hitting the "Send" button as the Date: value (as suggested by
RFC 2822).

> the latest Received header is a good bet.

Ah.  I think that should be mentioned, as one possible way to infer
the delivery time.

>    of the email in order to avoid filtering rules.

Very nice!

> the appropriate test" fits in here.

IMHO it is useful if all common pitfalls that lead to security
problems are discussed.

> I believe that message should be filtered just fine.

How?  Compare the following from RFC 3028.  Assuming the
implementation does not support CP-1252, I don't see how "MAKE MONEY
FAST" and "=?CP-1252?q?MAKE_MONEY_FAST?=" would match.  (The
assumption isn't a restriction, there will always be MUAs that
supports more charsets, or are less robust in their QP decoding, than
servers.)  Consider EBCDIC for a case where decoding the QP and only
using the ASCII characters would fail too.

,----
| 2.7.2.   Comparisons Across Character Sets
| 
|    All Sieve scripts are represented in UTF-8, but messages may involve
|    a number of character sets.  In order for comparisons to work across
|    character sets, implementations SHOULD implement the following
|    behavior:
| 
|       Implementations decode header charsets to UTF-8.  Two strings are
|       considered equal if their UTF-8 representations are identical.
|       Implementations should decode charsets represented in the forms
|       specified by [MIME] for both message headers and bodies.
|       Implementations must be capable of decoding US-ASCII, ISO-8859-1,
|       the ASCII subset of ISO-8859-* character sets, and UTF-8.
| 
|    If implementations fail to support the above behavior, they MUST
|    conform to the following:
| 
|       No two strings can be considered equal if one contains octets
|       greater than 127.
`----

> duplicate their tests.  hmm, perhaps we do need an extension for it.

The IDNA specification says that if IDNs are in IDN-unaware domain
name slots, it MUST be encoded as ASCII.  Since the Sieve
specification does not discuss IDN, all domain name slots in Sieve
scripts are IDN-unaware.

---

From: [email-masked]
Date: Sat, 19 Apr 2003 04:29:53 -0700 (PDT)

>   04 am (in the senders timezone) into a folder INBOX.rants.

A date test is also useful, but it is no replacement for the ability
to test the current date. Like it or not, the current date is a value
that may be reflected nowhere in the message, and may be the information
you want to use to decide on an action (e.g., I'm in a meeting and unable
to answer right away).

Additionally, if you are testing date information in the message header,
while it may be useful to test the Date: field, there are actually quite
a few other header fields that can contain date information that is even
more useful to test. For example, tests on Received: fields can be
very useful in determining if a message made it to a given point by 
a given time. Tests on the various expiry fields are another example.
All tests on the Date: field tell you is the time the originator's UA or
the originator's submission elected to attach to the message.

The problem with these sorts of tests is that they tend to be somewhat more
dependent on context  than we'd like (e.g., can you trust Received: fields or
not) and they introduce a substantial amount of additional complexity. For
exmaple, you really need a way to select a specific occurrance of a field  to
make Received: field tests worthwhile. As such, I beleve they should be cast 
as a separate operation and extension, not something we try and attach to
this draft.

>   times, if they want to support that.

Agreed. But this is not the place for it IMO. I'm trying to find the time
to write up a separate date test that accomplishes this.

>   rather high that many people in the real world also does.

Sure, this is an issue that has to be accomodated in such tests.

>   it.  Compare the sieve-body draft:

And the converse is also true: I may want to test for a specific sort of
encoded word (e.g., anyone sending a message to me with a subject in the gb2312
charset is probably sending spam) and that canont be done reliably at present.

>    replacement for a virus or spam filtering system.

Of course they aren't. Have we really reached the point where we 
have to belabor obvious points like this in security considerations sections?

>   required so I'm just mentioning it if someone else can come up with

They may be useful but they add substantial complexity to some implementations.
Absent a very compelling use case I don't think this is something we want
to support.

				Ned

---

From: [email-masked]
Date: Sat, 19 Apr 2003 04:44:04 -0700 (PDT)

> >     01 am and 04 am (in the senders timezone) into a folder

Cute. I hadn't thought of using this to get a likely time zone for the
originator.

> Ned, you had some ideas for more date related tests, what do you

I really don't want to try and force this into the setdate framework.
I'll see what I can do to write up the approach I have in mind for this.

> changed to initialise the variables to the time of delivery, not

Um, no, it may well be that it is the value in the Date: field that is
completely useless. It is all a question of what you're trying to achieve. If I
want a likely time zone for the originator I use Date:. If I want to guess at
when the message was actually sent use the oldest Received: field. If I want a
good estimate of when the message reached me use the newest Received: field.
(Actually, it may be necessary to  skip down a few to get the real scoop. This
is one of the reasons why this sort of test tends to be a bit context
sensitive.) If I want to know if the message has expired according to the
originator use one of the various expired fields. And if I want to alter
processing based on my current situation, use the current date.

There's a reason why there isn't a single grand unified time stamp on all
messages, and why we eventually want the ability to perform flexible
tests in this area.

> associative arrays, let's do it explicitly.

I completely agree.

				Ned

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 19 Apr 2003 14:30:39 +0200

[email-masked] writes:

> to test the current date.

Agreed, I want both.

> more useful to test.

Agreed, I want it the command to take a string that may be a variable,
so you can extract it from, e.g., Received headers.

> have to belabor obvious points like this in security considerations sections?

There are several solutions our there that claim to solve the email
spam and virus problems with Sieve, encoding common spam and virus
signatures in Sieve scripts (some are offering online access to
updated Sieve scripts with new signatures).  People are being told,
and believe, Sieve is the right tool for this.  Looking only at RFC
3028, I can't blame them.  It even contains examples on how to catch
spam.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 19 Apr 2003 15:29:50 +0200

[Ned Freed]:
>
>   I really don't want to try and force this into the setdate
>   framework.  I'll see what I can do to write up the approach I have
>   in mind for this.

thanks.

>   Um, no, it may well be that it is the value in the Date: field
>   that is completely useless. It is all a question of what you're
>   trying to achieve. [...]

all right, the original idea for adding SETDATE was to use it with
FILEINTO so that you could make archives "dynamically".  also, Sieve
is typically used by the delivery agent.  the case where a Sieve
client uses IMAP or similar to access the message store after delivery
has taken place, is an edge case.  furthermore, the spec says

|  These variables SHOULD reference the time when execution of the
|  Sieve script reaches the statement.

so such a client is allowed to use Received headers or whatever it
likes to find the date the user wants for SETDATE.  still, the intent
is that it references the time of delivery.  for the typical use, the
MDA, it doesn't make any difference.  for the standalone MFA (mail
filtering agent), it does make a difference, and I think we should be
explicit about what we want the MFA to do.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 19 Apr 2003 15:40:04 +0200

>   IMHO it is useful if all common pitfalls that lead to security

well, I don't think this is the place for it, it would fit better in a
revision of RFC 3028, or an informational RFC.

>   >>    For an example of a less general than expected test,
>   >>    consider looking for Subject: containing "MAKE MONEY FAST"
>   >>    to filter spam into a separate folder, and an incoming
>   >>    message having a Subject: of "=?CP-1252?q?MAKE_MONEY_FAST?=.
>   >
>   How?  Compare the following from RFC 3028 [2.7.2].  Assuming the
>   implementation does not support CP-1252, I don't see how "MAKE
>   MONEY FAST" and "=?CP-1252?q?MAKE_MONEY_FAST?=" would match.

oh, I assumed that an implementation would decode quoted-printable and
base64, but mark the string as "unknown charset".  that seems to "do
what I want" more often.

>   specification does not discuss IDN, all domain name slots in Sieve

good to know, thanks.
-- 
Kjetil T.

---

From: [email-masked]
Date: Sat, 19 Apr 2003 11:29:05 -0700 (PDT)

> 3028, I can't blame them.  It even contains examples on how to catch

Specific examples of how to catch a specific sort of spam message do not
a general filtering system make.

Every month Bruce Schneier has a "doghouse" section in his Cryptogram column
detailing one product or another he considers to be "snake oil security". In
many cases these systems try and gain credibility by claiming that they employ
one security standard or another. Yet we don't try and document this sort of
possible misuse in the security standards we write. Instead we focus on what
the system is good for and what its technical limitations are. We don't try and
deflect marketing pitches. It isn't a game that's appropriate to play in the
"once written never changed" context of an RFC.

				Ned

---

From: [email-masked]
Date: Sat, 19 Apr 2003 11:37:20 -0700 (PDT)

> has taken place, is an edge case.  furthermore, the spec says

In this case using the Date: field would be an exceedingly bad idea. You don't
want to let people add stuff to old archives simply by setting the Date: field
to a value in the past.

IMO the ideal value to use here is "date of insertion into the archive". The
current date is a reasonable thing to use for that. The Date: field is not.

> explicit about what we want the MFA to do.

I think it should refer to the time the filtering action was taken,
regardless of whether that was in the context of delivery or later filtering.

				Ned

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 19 Apr 2003 22:35:32 +0200

> base64, but mark the string as "unknown charset".  that seems to "do

That approach would only work, for this example, if the unknown
charset was ASCII compatible.  Of course, non-ASCII charset doesn't
occur in practice, but I was talking about the case where the sender
acts malicously to cause certain behaviour the user didn't think of.
If picking a non-ASCII charset makes the client behave badly, the
attacker will do so.

But I agree that your draft isn't the place for these generic Sieve
security considerations.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 20 Apr 2003 03:25:02 +0200

[Ned Freed]:
>   In this case using the Date: field would be an exceedingly bad
>   idea. You don't want to let people add stuff to old archives
>   simply by setting the Date: field to a value in the past.

absolutely, using Date is worse than useless.  I've seen quite a few
mail archives which do that on the web, though...

>   IMO the ideal value to use here is "date of insertion into the
>   archive". The current date is a reasonable thing to use for
>   that. The Date: field is not.

I (as a user) think the ideal value is "date of original delivery".
unfortunately, there is currently no standard way of obtaining that
value, you'll have to dig around in Received and apply some sanity
checking.

>   I think it should refer to the time the filtering action was
>   taken, regardless of whether that was in the context of delivery
>   or later filtering.

noted.

-- 
Kjetil T.

---

From: [email-masked]
Date: Sat, 19 Apr 2003 19:45:31 -0700 (PDT)

> If picking a non-ASCII charset makes the client behave badly, the

iso-2022-* charsets commonly occur in practice, and are not ASCII compatible
in this sense.

> But I agree that your draft isn't the place for these generic Sieve

Exactly.

				Ned