Subject: Proposal for escaping on non-UTF-8 sequences in Sieve

From: Alexey Melnikov <[email-masked]>
Date: Sun, 17 Sep 2006 23:39:53 +0100

Here is a strawman syntax for new quoted strings that would only allow
for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
sequences:

   new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE

   new-quoted-text    = *(utf8-quoted-safe / quoted-special /
                          quoted-non-utf8)

   not-qspecial       = %x01-09 / %x0B-0C / %x0E-21 / %x23-5B / %x5D-7F /
                        UTF8-2 / UTF8-3 / UTF8-4
                          ; a single Unicode character other than NUL,
                          ; CR, or LF, represented in UTF-8

   old-quoted-string  = DQUOTE quoted-text DQUOTE

   quoted-non-utf8    = "\" "x" 2*HEXDIG
                          ; represents a hex encoded octet

   quoted-string      = old-quoted-string / new-quoted-string
                          ; "new-quoted-string" is not allowed as
                          ; a parameter to "require" action.
                          ; "new-quoted-string" is only allowed after
                          ; 'require "utf8-strings";'

   utf8-quoted-safe   = CRLF / not-qspecial
                          ; either a CRLF pair, OR a single octet other
                          ; than NUL, CR, LF, double-quote, or backslash

As quoted strings can already contain multiple lines, I don't think we
need to introduce a new variant of the "multi-line" string. Please let
me know if you disagree.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 18 Sep 2006 11:31:07 +0200

Alexey Melnikov writes:
> ...

The capability name needs changing.

>                          ; 'require "utf8-strings";'

You're actually talking about non-utf8-compliant-strings. "utf8-strings" 
is a poor name for that.

I find it very difficult to summon any enthusiasm for this extension. 
Please include at least three realistic examples in the draft to show 
that it has value.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 18 Sep 2006 11:38:28 +0200

Alexey reminded me of some archived posts I'd forgotten. Please ignore 
my previous message.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 18 Sep 2006 10:39:45 +0100

Arnt Gulbrandsen wrote:

> You're actually talking about non-utf8-compliant-strings. 
> "utf8-strings" is a poor name for that.

Ok.

> Please include at least three realistic examples in the draft to show 

I will refer you to the following messages in the archive:
<http://www.imc.org/ietf-mta-filters/mail-archive/msg03105.html>
<http://www.imc.org/ietf-mta-filters/mail-archive/msg03147.html>

---

From: Michael Haardt <[email-masked]>
Date: Mon, 18 Sep 2006 16:51:49 +0200

>   new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE

You introduce a new lexical token to Sieve, thus changing the Sieve
syntax.  I don't think the base spec allows that.  Sieve has a very
small syntax, moving most stuff to the semantical layer for being very
extensible, pretty much like LISP does.

The variables extension, for example, introduces the concept of expanding
strings, but only at the semantic level.  Syntactically, strings still
look the same.

Michael

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 18 Sep 2006 09:59:20 -0400

Hi Arnt,

> You're actually talking about non-utf8-compliant-strings. "utf8-strings"

Not quite - the new mechanism allows for non-utf8 strings to be 'encoded' 
as a utf-8 string via an escaping mechanism. As such strings (and indeed 
the entire script) is still utf-8 (except where current implementations 
already violate that). Perhaps a better better capability name is 
'string-escape'.

-- 
Cyrus Daboo

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 18 Sep 2006 16:08:55 +0100

Michael Haardt wrote:

>syntax.
Yes.

>I don't think the base spec allows that.
>
IMHO, if it explicitly prohibit such change, we need to change the base 
spec.

>Sieve has a very
The problem is that RFC 3028 said that '\ <octet>' for any <octet> other 
than <\> or <"> SHOULD be interpreted as <octet>.
This effectively means that we can't fix quoted strings. So I had to 
introduced a new quoted string.

Do you have any better suggestions?

---

From: Ned Freed <[email-masked]>
Date: Mon, 18 Sep 2006 09:44:20 -0700 (PDT)

> >You introduce a new lexical token to Sieve, thus changing the Sieve

I have to agree with Michael - changing the basic syntax of sieve is
unacceptable.

> IMHO, if it explicitly prohibit such change, we need to change the base

One of the key strengths of sieve is that the core syntax is both simple
and immutable. This makes it possible to perform syntax checks even if
you don't understand all the extensions a given script uses. Breaking 
this would IMO be hugely damaging.

> introduced a new quoted string.

First of all, having an extension that changes the interpretation of, say, \x
followed by two hex digits would IMO be far less damaging than an extension
that changes the core syntax.

> Do you have any better suggestions?

There are all sorts of ways to do it. Here's an obvious one: If the
octet-value extension is enabled, any occurances of ${x} where x takes
the form of a space-separated list of decimal values is replaced with
a sequence of octets corresponding to each value. Hex is a bit more
difficult since the sequences could be confused with a variable names, so
if you want values in hex the simplest thing would be to invent a different
escape convention, e.g. something like $%x%.

				Ned

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 18 Sep 2006 11:06:30 -0400

Hi Michael,

> strings, but only at the semantic level.  Syntactically, strings still

How about if 'require string-escape' is present in the script, then ALL 
strings following that will be of the new 'escaped' form. i.e. a global 
switch covering all strings used in the script.

-- 
Cyrus Daboo

---

From: Ned Freed <[email-masked]>
Date: Mon, 18 Sep 2006 11:18:15 -0700 (PDT)

> Please include at least three realistic examples in the draft to show

The obvious one is that I, and I suspect lots of other people, have to deal
with unencoded 8bit in various charsets in message headers. I've tried the
"reject all mail containing raw 8bit in the headers" approach and find it
catches far too much legitimate (but malformed) mail - when I had such a policy
in place here at home my wife found it impossible to correspond with several
people she works with.

Interestingy, I actually find it's better to use this as a whilelisting
mechanism than a blacklisting mechanism, that is, I sometimes set up rules
that look for bogus 8bit sequences found only in legitimate mail and use
that to keep it out of the spam folder.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Mon, 18 Sep 2006 22:20:50 +0200

On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
> a sequence of octets corresponding to each value.

What a brilliant idea! How about using the same syntax variables do,
and embedding arbitrary data written as a function?

  "${hex:deadbeef}"
  "${dec:222 173 190 239}"

Exim string operators work that way.  I didn't follow the variables
extension too close, because I consider it to be too expensive for my
needs.  If there are string operators already, adapt the above as needed,
but it would be both extensible and easier to read than backslash quoting.
If needed, someone could add "${from_base64:}".

If used without variables, the function would not expand its arguments
first, thus evaluate in linear time.  If used with variables, its argument
would be expanded first, like strings are expanded.

Michael

---

From: Philip Guenther <[email-masked]>
Date: Mon, 18 Sep 2006 15:12:48 -0600

> and embedding arbitrary data written as a function?

You mean something like option (3) from my message of March 2005?
    http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html

Philip Guenther

("Over the years, I've developed my sense of deja vu so acutely that now I 
can remember things that *have* happened before ...")

---

From: Aaron Stone <[email-masked]>
Date: Mon, 18 Sep 2006 14:44:25 -0700

>     http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html

Didn't we end up zapping namespaces from the variables spec?

Aaron

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 19 Sep 2006 09:18:47 +0100

>Didn't we end up zapping namespaces from the variables spec?
No, an extension can define a namespace for variables.

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 24 Sep 2006 22:06:44 +0100

Ned Freed wrote:

>> [...]

Ok, good point.

> that changes the core syntax.

Fair enough.

> escape convention, e.g. something like $%x%.

After thinking more about this (and reviewing list of options provided 
by Philip), I would like to propose a variables-like syntax close to 
what you suggested above:
 $%<hex>
controlled by a new Sieve capability "string-escape".
This would give existing implementations an ability to provide the 
escaping functionality in both quoted and multiline strings, without the 
need to implement variables.
Any objections?
If people don't like this option, my next favorite would be to change 
the interpretation of \x.

Alexey