Subject: I-D ACTION:draft-degener-sieve-body-00.txt

From: Jutta Degener <[email-masked]>
Date: Thu, 25 Jul 2002 15:38:01 -0700

[ This is a much revised version in response to the calls for more
  text processing before a match on this list.  --Jutta]

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: Sieve -- 'body' extension
	Author(s)	: J. Degener
	Filename	: draft-degener-sieve-body-00.txt
	Pages		: 
	Date		: 26-Jun-02
	
This document defines a new primitive for the 'sieve' language
that tests for the occurrence of one or more strings in the body
of an e-mail message.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-degener-sieve-body-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-degener-sieve-body-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-degener-sieve-body-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 26 Jul 2002 15:41:16 +0200

> 	Title		: Sieve -- 'body' extension

I like this version much better, but still some small
comments/nits/thoughts:

0) No way to split based on MIME structure or MIME headers.  Compare
   the complex MIME-parsing features of the IMAP FETCH BODY[] command.
   It may take the sieve body extension too far to dvelve into a
   full-blown MIME parser, but it might be useful some day.  E.g.,
   split all OpenPGP multipart/encrypted mail into one folder, all
   S/MIME multipart/encrypted mail into another folder.

Two issues regarding the text:

,----
|    MIME parts encoded in a content transfer encoding must be decoded,
|    and text MIME parts in charsets other than UTF-8 MUST be converted
|    to UTF-8 prior to the match.
`----

1) Converting into UTF-8 is non-trivial.  Standards wise it is not
even defined what this means, since I don't know of any standards body
that publish official transcoding tables.  It is possible for
transcoding tables "out there" to even conflict (e.g. CP437 used in a
german environment may transcode differently than CP437 used in a
greek environment).  I suggest at least noting that this is
non-trivial, but that such efforts should however be made, and the
exact implementation is left to local policy.

2) CTE "decoding" is a bit loose.  I suggest specifying that
   implementations must support some set of common CTE's, such as
   base64, qp, 8bit.  It may make sense to discuss what should happen
   if CTE is syntactically incorrect, or contains a value which the
   implementation does not support (e.g., "x-yenc").

3) What about multipart/partial?

4) (Revealing my ignorance:) Are sieve scripts binary clean?  If I
   want to implement something like the unix "file" command in a sieve
   script to split some data into special folders depending on magic
   values in the file, I might need to express binary data.

---

From: [email-masked]
Date: Fri, 26 Jul 2002 10:28:14 -0700 (PDT)

>    S/MIME multipart/encrypted mail into another folder.

This is starting to get complicated, and complexity can be expensive...

> exact implementation is left to local policy.

This sounds like a reasonable thing to do, but there's also the issue of how
unknown charsets are handled. It may be that searching unconverted text is a
reasonable fallback or it may not. I'd suggest a compromise: A search involving
8bit characters on text with an unknown charset always fails, while a search
involving only the ASCII subset proceeds on the unconverted text. I base this
on the observation that ASCII-compatible charsets are pretty common. (Although
perhaps a further heuristic that any unknown charset that has 2022 in the name
shouldn't receive such treatment would also be useful...)

>    implementation does not support (e.g., "x-yenc").

I agree that this should be discussed. I'm not completely sure how to handle
it, but I'm leaning towards saying that the search should simply fail.

> 3) What about multipart/partial?

Hmm. Tricky. I think any search involving non-ASCII should fail. I'm less
clear on whether or not an ASCII search should proceed.

Searches of encrypted data are also likely to produce less than useful
results.

> 4) (Revealing my ignorance:) Are sieve scripts binary clean?

No. Sieves are defined to be in UTF-8, with the restrictions that implies.

>    If I want to implement something like the unix "file" command in a sieve

I think searching for patterns in binary data goes beyond the scope of this
particular extension. In the specific case of a magic number test, I'd suggest
that rather than attempting to code such tests into sieve that this be done by
having a "magic" test that applies a set of system-provided magic number tests
to the file and returns a result that the sieve can then check.

				Ned

---

From: "Nigel Swinson" <[email-masked]>
Date: Mon, 29 Jul 2002 15:43:19 +0100

> I like this version much better,

Me too :o)

> but still some small

Here's a very small nit:

Syntax in 4.0 reads:
   Syntax:
                 ":raw"
                / ":content" <content-types: string-list>
                / ":text"

But section header 4.2 reads:
    4.2 Body Transform ":content_type"

I presume that it's meant to be "content_type" in both places...

Default transformation should be ":text".  If in our implementation we don't
see the :raw argument, then it will be a compile time error.  When we create
the script element we'd create with the :raw tag argument.

>    S/MIME multipart/encrypted mail into another folder.

Yeah it seems the way we have it now it's great for looking for stuff inside
the text of body parts, but we're missing a way to test against the mime
headers of body parts, and the existance of body parts (although we could
just about do it by doing a :raw search).  Should we try to do this in the
same test, or should we have a separate test for these kind of searches?

In some ways we want to have recursive tests, like size, header and exists
that would operate on all body parts rather than just the main message.
Perhaps a test like this

    if header :recursive :contains "Content-Type" "text" {}

To look for messages that contain at least one text body part.

    if allof (header :recursive :contains "Content-Type" "text", size :over
"2K") {}

All messages that contain a text body part and the complete message is over
2K.

    if allof (header :recursive :contains "Content-Type" "text", size
:recursive :over "2K") {}

All messages that contain a text body part that is bigger than 2K

    if header :recursive :contains "Content-Type" "text" {
        if size :recursive :over "2K" {}
    }

All messages that contain a text body part, and a body part that is bigger
than 2K.

Perhaps a count test that would count the number of body parts of a
particular type would be useful too?

    if count :contains "Content-Type" "text" :over 2 {}

Perhaps something along these lines would complement the proposed body test
and together they would give us most (all?) the matching we'd need?

Nigel