Subject: Matching NUL characters

From: [email-masked]
Date: 3 Apr 2003 21:58:00 -0000

Hello,

as defined by RFC 3028, Sieve scripts may not contain NUL characters.
MIME encoded headers may contain them (encoded), though, and there is
currently no way to match them with Sieve.  For that reason I suggest
to change paragraph 2.4.2 of the RFC to define \0 being a representation
of the NUL character.

I know, that's quite a change, but it's quite a bug, too. ;-) Btw, both
Mutt and Outlook Express don't show anything after a MIME encoded NUL
characters, so they simply decode it and, given C strings, terminate the
string that way, because nobody ever thought about how to handle them.
If a sieve implementation is not subject to this bug, then most likely
it could support \0 pretty easy.  Otherwise some work is required anyway
to fix the bug.

While extending the meaning of \, I suggest to use the Java \u notation
to specify unicode characters.  Sieve scripts are written in UTF-8 for
good reasons, but being able to use non-ASCII characters without having
to use UTF-8 would be great.  That's just an idea, though, and not really
required like \0 is.

I am currently writing my own sieve implementation as extension to Exim.
Hacking MIME decoding, I discovered the problem above and Tim Showalter
suggested I join the list.  It's not the first issue I had with the RFC
so far, but the first that asks for a change in the language.

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: 04 Apr 2003 10:11:52 +0200

[[email-masked]]:
>
>   as defined by RFC 3028, Sieve scripts may not contain NUL
>   characters.  MIME encoded headers may contain them (encoded),
>   though, and there is currently no way to match them with Sieve.
>   For that reason I suggest to change paragraph 2.4.2 of the RFC to
>   define \0 being a representation of the NUL character.
>   
>   I know, that's quite a change, but it's quite a bug, too. ;-) Btw,
>   both Mutt and Outlook Express don't show anything after a MIME
>   encoded NUL characters, so they simply decode it and, given C
>   strings, terminate the string that way, because nobody ever
>   thought about how to handle them.

since the wording is quite explicit in the Sieve standard, I'd rather
say that this was intentional.  it complicates implementation to
support the NUL character, for little benefit.

>   If a sieve implementation is not subject to this bug, then most
>   likely it could support \0 pretty easy.  Otherwise some work is
>   required anyway to fix the bug.

I suggest you propose an extension.  an implementation that follows
the current RFC should not be rendered incompatible.

>   While extending the meaning of \, I suggest to use the Java \u
>   notation to specify unicode characters.  Sieve scripts are written
>   in UTF-8 for good reasons, but being able to use non-ASCII
>   characters without having to use UTF-8 would be great.  That's
>   just an idea, though, and not really required like \0 is.

in that case I suggest an arbitrary length of hexadecimal digits,
e.g., \u0 == \u00000000.  otherwise we'll get a mess when characters
outside the basic plane are needed.  (encode to UTF-16 yourself, let
the Sieve implementation convert that to UTF-8 -- ugly!)  note that
you can terminate the hex sequence with a backslash, this never gives
ambiguity since the only escaped characters are backslash, doublequote
and "u".  e.g.,

  \u0\foo => NUL f o o
  
if you want the "\0" escape, you can't express U+ef followed by zero.

  \uef\0 => U+ef NUL

"\u0" is so short "\0" is superfluous.  the number of escapes whould
be kept to a minimum.

-- 
Kjetil T.

---

From: Chris Newman <[email-masked]>
Date: Fri, 04 Apr 2003 20:11:16 -0800

Anyone who uses a NUL of any flavor in this context deserves to lose. 
Implementations should be free to drop the NUL or truncate the line at a NUL, 
regardless of whether it is encoded.

RFC 2822 deprecated the NUL character, and I think we can and should deprecate 
an encoded NUL when RFC 2047 is revised -- one of the important steps moving 
forward on the standards track is to deprecate features that aren't useful and 
cause interoperability problems.  Encoded NUL certainly meets that test.

So I am opposed to adding _any_ additional complexity to Sieve in order to 
support NUL.

                - Chris