Subject: Questions regarding RFC 5228

From: Stephan Bosch <[email-masked]>
Date: Sun, 19 Oct 2008 12:31:58 +0200

Hello,

I am finishing up a first release of my Sieve implementation, and one of 
the TODO items that yet remains is getting some answers to questions 
that arose during development. I've collected these into a file an now I 
submit them to this list to get some clarification. Any help is greatly 
appreciated.

* RFC 5228 (Sieve) : 5.1.  Test address:
"Implementations MUST restrict the address test to headers that contain 
addresses, but MUST include at least From, To, Cc, Bcc, Sender, 
Resent-From, and Resent-To, and it SHOULD include any other header that 
utilizes an "address-list" structured header body."
   
  -> Will this cause a compile error, or are the disallowed headers 
simply ignored? My implementation currently considers this to be a 
compile error.
  -> Given the variables extension, sometimes the specified header names 
aren't known until runtime. If the previous answer was to cause a 
compile error, should this abort the script at runtime?
    
* RFC 5228 (Sieve) : 5.4.  Test envelope:
"The "envelope" test is true if the specified part of the [SMTP] (or 
equivalent) envelope matches the specified key.  This specification 
defines the interpretation of the (case insensitive) "from" and "to" 
envelope-parts.  Additional envelope-parts may be defined by other 
extensions; implementations SHOULD consider unknown envelope parts an 
error."
   
  -> Given the variables extension, sometimes the specified envelope 
parts aren't known until runtime. Should invalid ones abort the script 
or is ignoring them a better practice?

Regards,

--
Stephan Bosch
[email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 20 Oct 2008 11:21:40 +0200

Stephan Bosch writes:
> Hello,
>
> I am finishing up a first release of my Sieve implementation, and one 
> of the TODO items that yet remains is getting some answers to 
> questions that arose during development. I've collected these into a 
> file an now I submit them to this list to get some clarification. Any 
> help is greatly appreciated.
>
> "Implementations MUST restrict the address test to headers that 
> contain addresses, but MUST include at least From, To, Cc, Bcc, 
> Sender, Resent-From, and Resent-To, and it SHOULD include any other 
> header that utilizes an "address-list" structured header body."

So does mine.

> compile error, should this abort the script at runtime?

I don't have variables (yet?). I expect that I would try to give an 
error at compile time and to avoid runtime errors.

> extensions; implementations SHOULD consider unknown envelope parts an 

Again, I give an error at compile time, none at runtime.

>  parts aren't known until runtime. Should invalid ones abort the 
>  script or is ignoring them a better practice?

Sieve is a simple language... All commands in a script can be arranged 
into basic blocks, and the basic blocks form a DAG. (I don't remember 
whether this remains true with Ned's MIME/looping extension.)

I wonder whether it is possible to walk along the DAG and say "this 
assignment is invalid since it leads to an error when the variable is 
used".

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 20 Oct 2008 11:19:10 +0100

>>  simply ignored? My implementation currently considers this to be a 

My implementation allows "address" on any header - it will just try to 
parse it as an email address.

>> -> Given the variables extension, sometimes the specified header 
>> names aren't known until runtime. If the previous answer was to cause 
>> a compile error, should this abort the script at runtime?
>

How would you do that if you supported variables? Are you just going to 
reject the second parameter because it contains a variable reference?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 20 Oct 2008 12:28:32 +0200

Alexey Melnikov writes:
> How would you do that if you supported variables? Are you just going 
> to reject the second parameter because it contains a variable 
> reference?

It would be easy for me to walk back (at compile time) from where the 
variable is used to the locations where it's set, and see if each of 
those values look legal, illegal or indeterminate.

Don't know what I'd do when the value looks indeterminate, but I hope to 
get an idea from this thread.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Mon, 20 Oct 2008 09:53:18 -0700 (PDT)

> So does mine.

Our implementation does not. We allow address tests on all possible header
fields. The test fails if the field does not contain actual addresses.

IMNSHO what you are doing here is quite simply wrong. It might, and I emphasize
MIGHT, be OK to throw an error if an address test is done on a really well
defined field that is either unstructured or structured as something other than
an address list. (Although I see no point in doing that.) But for an unknown
field, you cannot assume that it isn't defined by something somewhere as an
address list. Even if you assume you can keep up with the fields people bother
to register (and if you think you can I have to wonder how you get all the
sites using your software updated to the new definitions in a timely way),
people define an use their own private fields all the time. And nothing we've
done or can do in the standards process is going to change this.

And what about fields that have address syntax but which aren't exactly  an
address-list? The obvious one is message-id, which I believe is a proper subset
of address syntax. I've seen quite a few scripts written that use address tests
on the message-id header. Heck, the way this is written fields defined with
Sender: or From: syntax would be excluded, and that's nothing short of absurd.

I will also point out that the standard doesn't say anything about throwing
errors in this case. "Restricting" a test could mean that, but I think having
the test fail is more than sufficient.

But let's suppose the specificaiton did say an error should occur. I have to
say I would happily violate the specification in that case. The entire point of
Sieve is to give users a rich set of tools they can use to process the messages
they receive, including some seriously messed up stuff. I'm sorry, but when the
need to be able to deal with the broken dreck people actually send around
conflicts with implemnetation purity, effective hanlding of the dreck wins.
Every. Single. Time.

> Again, I give an error at compile time, none at runtime.

We check this at runtime. Changing things to perform a compile time check
wouldn't be difficult, at least in the common case of no variable
substitutions, but since this can't be done when ihave is turned on and we're
headed towards a situation where the vast majority of scripts we process use
ihave, I don't see any point in spending the time to implement this.

> >  script or is ignoring them a better practice?

A script that manages to do an envelope test on an nonexistant envelope part is
broken in a fairly fundamental way. This needs to be noted, and the way you do
that is to throw an error.

> whether this remains true with Ned's MIME/looping extension.)

It's not my extension, but since it doesn't change the syntax or structure of
Sieve I don't see how it could impact the blocking. But it certainly can make
static analysis more difficult if not impossible.

> assignment is invalid since it leads to an error when the variable is

Sure, that's possible in a lot of cases, but not all. The envelope part name
could come from the message data itself (sounds like a really bad idea, but you
can certainly do it), so unless your compilation step is done in a
message-dependant way that's a fundamental limit on analysis. (We compile in a
message-independent way so the same compiled script can be applied to multiple
messages. We even have the ability to store compiled scripts on disk and read
them back.)

But even if message data isn't involved or can be built into the compilation
process, I'm pretty sure doing it in full generality is at least in NP.

The real question, then, is whether this check is worth the bother. Since I
don't think even checking static string arguments to envelope is worth doing at
compile time, it goes without saying where I stand on doing even more analysis
of this case.

				Ned

---

From: "Mark E. Mallett" <[email-masked]>
Date: Mon, 20 Oct 2008 14:28:02 -0400

> fields. The test fails if the field does not contain actual addresses.

[ snipped ]

Ditto; I was going to say the same thing, and for all the points
you mention.  This "MUST" (or at least that interpretation of it) is
something I disagree with - I think it's unreasonable for an
implementation to tell a script writer that they may not attempt to
examine or extract or test an address in a header field that the script
writer wants to examine or extract or test, even if that header field
might not be defined as addresses in it in a structured way.  The
message-id case is a perfect example, but there are others.

I think that that proscription is more of a script 'con' than a 'pro.'  :)

mm

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 20 Oct 2008 21:16:42 +0200

Mark E. Mallett writes:
> Ditto; I was going to say the same thing, and for all the points you 
> mention. This "MUST" (or at least that interpretation of it) is 
> examine or extract or test an address in a header field that the 
> script writer wants to examine or extract or test, even if that 
> header field might not be defined as addresses in it in a structured 
> way. The message-id case is a perfect example, but there are others.

I looked at this message and saw two more examples.

List-Id: <ietf-mta-filters.imc.org>
List-Unsubscribe: <mailto:[email-masked]?body=unsubscribe>

Both fields are very widely used, and people do want to handle list mail 
specially in sieves, so I guess that if the address test ought to work 
beyond ordinary address fields, then those two are high on the priority 
list. So has anyone here written the code necessary to have the address 
test handle List-Id, List-Unsubscribe and friends?

This sounds like a fine test, just the kind of thing to catch all 
mailing-list mail.

if address :is :all [ "list-id", "to", "cc" ] "[email-masked]" {
     fileinto "sieve";
}

> I think that that proscription is more of a script 'con' than a 'pro.' :)

Simplicity is a valuable feature.

Arnt

---

From: "Mark E. Mallett" <[email-masked]>
Date: Mon, 20 Oct 2008 15:48:08 -0400

> test handle List-Id, List-Unsubscribe and friends?

Yes, in that I don't do anything special to prohibit List-ID from being
used in an address test.  In fact I use an address test against
"List-ID" heavily in the script for my personal inbox, including for
this very mailing list.  (In kind of a layered way: info is pulled out
of fields using sieve statements and tested against metadata in a
database, but that's a tangent.

mm

---

From: Stephan Bosch <[email-masked]>
Date: Mon, 20 Oct 2008 23:42:39 +0200

(OOPS, used wrong identity)

> assignment is invalid since it leads to an error when the variable is 
In extreme cases, a user can extract data from the incoming message
using :matches or even :regex. Then, there is really no way of telling
what the variable is going to contain. But yes, in all other cases one
could do compile-time string constant evaluation and be done with it.

I am interested particularly in those extreme cases. Not that I see a
particular use for those in the mentioned cases, but I like to be thorough.

Regards,

Stephan.

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 21 Oct 2008 00:00:17 +0200

> fields. The test fails if the field does not contain actual addresses.
<SNIP>Long explanation about why this is wrong.</SNIP>

There is a reason why I pose this as a question. :) I am just trying to 
follow the standard here. Your points are valid and they do make me 
wonder why this was added in the new Sieve RFC. However, I do think we 
should reach some sort of consensus here to prevent ill portability of 
scripts that 'address' obscure headers.

> ihave, I don't see any point in spending the time to implement this.
I agree.

> that is to throw an error.
Ok.

Regards,

Stephan

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 21 Oct 2008 00:10:22 +0200

> if address :is :all [ "list-id", "to", "cc" ] "[email-masked]" {

I hope that users will not relying on their server converting
ietf-mta-filters.imc.org into [email-masked].  Likewise for
extracting stuff from a mailto URL.  I wouldn't object to this being
added to RFC 5228bis (or do we call it 3028ter?), but with the current
state of affairs these headers should be left alone, and the address
operator should fail.

(yes, I'd like to loosen the requirement of flagging errors for applying
this test on header fields known not to include addresses, too.)
-- 
regards,
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 21 Oct 2008 16:14:36 -0400

> state of affairs these headers should be left alone, and the address

Yes, Arnt's test should not attempt to combine "list-id" with "to" and "cc".
I suspect that was just a quick mistake.

But there's a good point in there (not about converting the format, but
about the lack of an '@').  My implementation does not require an '@' in
any address in an address test.  If an '@' is missing, :localpart and
:domain do not return anything, but :all returns the entire string.  I
believe this is correct behavior according to RFC5228 (but even if it
weren't, I'd make it work that way anyway, since I'd want it to be
useful rather than fail).  The list-id header field has a well defined
format that conforms to the way my implementation extracts addreses, and
so I feel happy using the address test against List-ID.  e.g.:

   if anyof(
      address :is :all ["to", "cc"] "[email-masked]",
      address :is :all "list-id" "ietf-mta-filters.imc.org" )
     { ... }

(barring typos and errors ..)

mm

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2008 22:58:45 +0200

Mark E. Mallett writes:
> Yes, Arnt's test should not attempt to combine "list-id" with "to" and 
> "cc". I suspect that was just a quick mistake.

I wrote the example in the natural, readable way on purpose.

> But there's a good point in there (not about converting the format, 
> but about the lack of an '@').  My implementation does not require an 
> '@' in any address in an address test.  If an '@' is missing, 
> :localpart and :domain do not return anything, but :all returns the 
> entire string.  I believe this is correct behavior according to 
> RFC5228 (but even if it weren't, I'd make it work that way anyway, 
> since I'd want it to be useful rather than fail).  The list-id header 
> field has a well defined format that conforms to the way my 
> implementation extracts addreses, and so I feel happy using the 
> address test against List-ID.  e.g.:

So you're saying it should be okay to use the address test for 
non-address-fields, except that :user and :domain might not work at 
all, and :all might work differently from the way it usually works?

Arnt

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 22 Oct 2008 17:33:32 -0400

> I wrote the example in the natural, readable way on purpose.

Sorry, I was just making a charitable guess.  Natural, readable examples
are good, but not when they don't work.  It certainly begged the point
that Kjetil Torgrim Homme raised, i.e., that it would require some
magical conversion of the address string in order to make any sense.

> all, and :all might work differently from the way it usually works?

No, I'm saying that the address test should work the same way all the
time when it finds an address.  I don't get what you are saying there.

mm

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Oct 2008 12:05:51 +0200

Mark E. Mallett writes:
> Sorry, I was just making a charitable guess. Natural, readable 
> examples are good, but not when they don't work. It certainly begged 
> the point that Kjetil Torgrim Homme raised, i.e., that it would 
> require some magical conversion of the address string in order to 
> make any sense.

Exactly.

The complexity isn't going away. You can accept difficult conversions in 
the code, or you can push it into the user's sieve script and the 
documentation of what address does. See below.

(Actually I don't think that particular conversion would be so difficult 
— I've done much worse ones. Turn <a.b.c.d> into ["[email-masked]", "[email-masked]", 
"a.b.c@d"] and Sieve's usual matching rules will do the rest. But other 
ones could be arbitrarily difficult.)

> time when it finds an address. I don't get what you are saying there.

Let me try it differently.

The address test is defined to test an address against another address, 
or against a part of another address. The domain part, the user part, 
or optionally the subaddress part.

The address test could also test

  - addresses against a non-address
  - non-addresses against an address (what my example did)
  - non-addresses against a non-address (what yours did)

But I think that doing that leads to unacceptable complexity and/or 
unreliability.

Complexity: If you want to guarantee a particular behaviour for e.g. 
non-addresses against non-addresses, then the documentation's 
explanation of what the address test does becomes too difficult for 
mere mortals. Testing becomes a pain, too.

Or unreliability: You mentioned that in your code, :all worked on 
list-id, right? Suppose one of your colleagues revises your address 
parser so that it canonicalises "From: Cron Daemon <root>" into "From: 
Cron Daemon <[email-masked]>". Then using address :all on list-id 
stops working and address :localpart starts working.

Arnt