Subject: non-UTF-8 sequences in Sieve scripts

From: Alexey Melnikov <[email-masked]>
Date: Fri, 15 Sep 2006 00:34:16 +0100

Hi everyone,

I think there is rough consensus to have an extension that would 
introduce new strings with proper escaping mechanism. The new strings 
will allow for UTF-8 (as is) + non-UTF-8 can be escaped. As nobody 
volunteered to do this, I will write a short proposal on this.

But I would like to get general feeling of the Working Group about 
non-UTF-8 sequences in existing 3028bis strings.
draft-ietf-sieve-3028bis-09.txt doesn't prohibit them. However, what 
should our long term solution be? I see two choices:

1). draft-ietf-sieve-3028bis stays as it is now, i.e. it allows for 
non-UTF-8 sequences, but recommends UTF-8. Eventually implementations 
will implement new strings with proper escaping mechanism, but existing 
strings will stay as described till the end of time.

2). draft-ietf-sieve-3028bis gets updated to only describe UTF-8 
sequences with no escaping (*). The draft can describe existing 
situation with non-UTF-8 sequences, or it can remain completely silent 
on the issue. This wouldn't affect existing implementations, as long as 
non-UTF-8 sequences in scripts are not explicitly prohibited. The draft 
will also explicitly state that future revisions of this document would 
not allow for non-UTF-8 sequences.

=====
So basically 2) would state the desire to deprecate use of unescaped 
non-UTF-8 sequences in scripts, while 1) would allow for them indefinitely.

Do people see other choices?

Please reply with your preferences till September 29th.

Thanks!
Alexey

(*) - "MUST accept UTF-8 sequences, MAY accept non-UTF-8 sequences"

---

From: Ned Freed <[email-masked]>
Date: Fri, 15 Sep 2006 07:01:40 -0700 (PDT)

> Do people see other choices?

First of all, I think we need to describe whatever escaping mechanism we decide
on, preferably in the base document. I don't think it is reasonable to suggest
abandoning a capability, let alone forbidding it, unless we have the
alternative in place.

Once we have the alternative defined I think the right approach is somewhere
between (1) and (2). (2) goes too far IMO by saying that deprecation will
happen in the future. I think the correct course is to say that the
ability to use non-UTF-8 _may_ be abandoned in the future and to plan
accordingly.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 15 Sep 2006 15:22:06 +0100

> abandoning a capability, let alone forbidding it, unless we have the

Right. I will post a proposal next week.
I am not sure I want to have it in the base spec, as it will mean 
delaying its publications, but we can argue this point separately later.

> ability to use non-UTF-8 _may_ be abandoned in the future and to plan

Right. In general case it is not possible to put requirements on future 
specs.

Ok, if (2) is replaced with what you've proposed, which choice would you 
prefer?

---

From: Ned Freed <[email-masked]>
Date: Fri, 15 Sep 2006 08:12:57 -0700 (PDT)

> Ok, if (2) is replaced with what you've proposed, which choice would you

Definitely (2). People need to be warned that this may change in the future.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Mon, 18 Sep 2006 12:06:43 +0200

> strings will stay as described till the end of time.

I dislike allowing non-UTF-8 sequences very strongly, because it forbids
to convert a Sieve filter to other character sets without scanning it
for strings.

> not allow for non-UTF-8 sequences.

I prefer that the base spec continues to only describe UTF-8, staying
silent on implementations that take the freedom to violate RFC 3028,
which says in 2.1:

   The language is represented in UTF-8, as specified in [UTF-8].

Whoever decides to leave UTF-8 knows the risks of doing so.
We decided "envelope-auth" should be introduced as extension, although
implementations simply offered it before.  I see a similar situation now:
Some implementations offer a super set of RFC 3028.  To me, that is not a
reason to extend the base spec, but a reason to introduce an extension.
That way, we can be sure all Sieve implementations will conform to the
new spec.

On the other side, I don't think we gain something by saying: "You MUST
not use anything but UTF-8, or you MUST NOT stick the Sieve logo on your
implementation." ;-)

An extension to introduce arbitrary octet sequences encoded as UTF-8
(or ASCII) character sequences is fine with me, but I doubt its use
for the base specification, as it is most useful when working with
raw data, which the base spec does not offer.

Michael

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 18 Sep 2006 10:05:19 -0400

Hi Michael,

>    The language is represented in UTF-8, as specified in [UTF-8].

But it also says:

2.4.2.4. MIME Parts

   In a few places, [MIME] body parts are represented as strings.  These
   parts include MIME headers and the body.  This provides a way of
   embedding typed data within a Sieve script so that, among other
   things, character sets other than UTF-8 can be used for output
   messages.

i.e. there is ambiguity here, but the bottom line is non-utf-8 data can be 
used in current sieve scripts without violating the spec. That is the 
problem we want to solve, but we cannot outlaw non-utf-8 now without making 
many implementations non-compliant. What we want to do in the long run is 
deprecate use of non-utf-8 strings in favor of escaped utf-8 strings, which 
is what Alexey's strawman is leading to.

-- 
Cyrus Daboo

---

From: Michael Haardt <[email-masked]>
Date: Mon, 18 Sep 2006 16:33:03 +0200

> is what Alexey's strawman is leading to.

I don't see the ambiguity.  MIME encodes the data in US-ASCII, and the
script only contains the MIME-encoded data.  Sieve knows nothing about
the MIME-encoding of content passed e.g. to vacation: Its _type_ may be
a Klingon sound file, not even text data, but MIME-encoded Sieve only
sees US-ASCII.

We are looking at non-UTF-8 data that Sieve sees as such.

I don't mean to outlaw non-UTF-8 in Sieve scripts, because I see no
purpose in doing so.  But I don't want to legalise it either.  If many
(how many?) implementations decided to implement a superset of RFC 3028,
that's their decision.  I prefer to stay silent about that, and use a
regular extension to offer that functionality.

Michael

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 18 Sep 2006 11:07:27 -0400

Hi Michael,

> a Klingon sound file, not even text data, but MIME-encoded Sieve only

I cannot see any restriction that would prevent use of 
Content-Transfer-Encoding: 8-bit, in which case the MIME parts are not 
necessarily US-ASCII and not necessarily UTF-8, e.g. a text/plain part with 
charset=iso8859-1.

-- 
Cyrus Daboo

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 18 Sep 2006 16:11:05 +0100

Michael Haardt wrote:

>script only contains the MIME-encoded data.
>
What about 8bit and binary content transfer encodings?

>Sieve knows nothing about
You opinion is noted. So basically you want a variant of option (2) from 
my poll.

---

From: Michael Haardt <[email-masked]>
Date: Mon, 18 Sep 2006 21:57:51 +0200

> necessarily US-ASCII and not necessarily UTF-8, e.g. a text/plain part with 

The restriction is that the script is written in UTF-8.  MIME is
specifically made to transport arbitrary typed data through an environment
that allows only 7-bit US-ASCII, so no problem there.

With regard to Sieve, think of UTF-8 as your transport.  You can
encode arbitrary data, but you can not encode it arbitrarily, but
like 7-bit-only mail gateways, that's no problem for MIME.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 30 Sep 2006 22:25:22 +0100

>> >> non-UTF-8 sequences in scripts, while 1) would allow for them
[...]

>> > Once we have the alternative defined I think the right approach is
>> > somewhere between (1) and (2). (2) goes too far IMO by saying that 
>> deprecation will

Thinking more about this. I think replacing existing ABNF in 3028bis to 
only allow for valid UTF-8 sequences would be a drastic change at this 
point, as it will pretty much declare all existing implementations non 
compliant. I've already suggested some warning text about phasing out 
non-UTF-8 sequences and you agreed with my text. I think the suggested 
text is sufficient for this round of work on 3028bis.

Comments?

Date: Sat, 30 Sep 2006 22:25:22 +0100
From: Alexey Melnikov <[email-masked]>
To: Ned Freed <[email-masked]>
CC: MTA filtering mailing list <[email-masked]>
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

>> >> non-UTF-8 sequences in scripts, while 1) would allow for them
[...]

>> > Once we have the alternative defined I think the right approach is
>> > somewhere between (1) and (2). (2) goes too far IMO by saying that 
>> deprecation will

Thinking more about this. I think replacing existing ABNF in 3028bis to 
only allow for valid UTF-8 sequences would be a drastic change at this 
point, as it will pretty much declare all existing implementations non 
compliant. I've already suggested some warning text about phasing out 
non-UTF-8 sequences and you agreed with my text. I think the suggested 
text is sufficient for this round of work on 3028bis.

Comments?

Cc: [email-masked]
To: Alexey Melnikov <[email-masked]>
Date: Sat, 30 Sep 2006 13:50:25 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: Text suggesting that non-UTF-8 sequenced may be abandoned in the future
Sender: [email-masked]

> Hi everyone,
> 3028bis, section 2.1. (Form of the Language) says:

>  >   With the exceptions of strings and comments, the language is limited
>  >   to US-ASCII characters.  Strings and comments may contain octets
>  >   outside the US-ASCII range.  Specifically, they will normally be in
>  >   UTF-8, as specified in [UTF-8].  NUL (US-ASCII 0) is never permitted
>  >   in scripts, while CR and LF can only appear as the CRLF line ending.

> As per recent discussion on the mailing list I suggest adding the
> following text to the end of this paragraph:

> Note that Sieve WG is working on an extension that would allow for 7bit
> escaping
> of non-UTF-8 data. When and if this extension is deployed, the ability
> to use
> non-UTF-8 data unescaped in Sieve scripts may be abandoned.

> Comments?

Seems OK to me.

				Ned

Date: Sat, 30 Sep 2006 21:18:38 +0100
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
Subject: Text suggesting that non-UTF-8 sequenced may be abandoned in the future
Sender: [email-masked]

Hi everyone,
3028bis, section 2.1. (Form of the Language) says:

 >   With the exceptions of strings and comments, the language is limited
 >   to US-ASCII characters.  Strings and comments may contain octets
 >   outside the US-ASCII range.  Specifically, they will normally be in
 >   UTF-8, as specified in [UTF-8].  NUL (US-ASCII 0) is never permitted
 >   in scripts, while CR and LF can only appear as the CRLF line ending.

As per recent discussion on the mailing list I suggest adding the 
following text to the end of this paragraph:

Note that Sieve WG is working on an extension that would allow for 7bit 
escaping
of non-UTF-8 data. When and if this extension is deployed, the ability 
to use
non-UTF-8 data unescaped in Sieve scripts may be abandoned.

Comments?

Alexey

Date: Wed, 27 Sep 2006 11:47:55 -0700
From: Matthew Elvey <[email-masked]>
To: [email-masked]
CC: Alexey Melnikov <[email-masked]>
Subject: draft-ietf-sieve-3028bis-09
Sender: [email-masked]

Minor.

Would it make sense to add this or something like it to the security
section?
   Security issues associated with UTF-8 are fully
   discussed in the security consideration section of [RFC3629]. This
   document is believed not to introduce any additional security
   considerations in this general area.

What's Cyrus' new email?

Date: Sun, 24 Sep 2006 22:06:44 +0100
From: Alexey Melnikov <[email-masked]>
To: Ned Freed <[email-masked]>
CC: Michael Haardt <[email-masked]>, [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Ned Freed wrote:

>> [...]
>
> I have to agree with Michael - changing the basic syntax of sieve is
> unacceptable.
>
>> >I don't think the base spec allows that.
>> >
>> IMHO, if it explicitly prohibit such change, we need to change the base
>> spec.
>
> One of the key strengths of sieve is that the core syntax is both simple
> and immutable. This makes it possible to perform syntax checks even if
> you don't understand all the extensions a given script uses. Breaking 
> this would IMO be hugely damaging.

Ok, good point.

>> >Sieve has a very
>> >small syntax, moving most stuff to the semantical layer for being very
>> >extensible, pretty much like LISP does.
>> >
>> >The variables extension, for example, introduces the concept of 
>> expanding
>> >strings, but only at the semantic level.  Syntactically, strings still
>> >look the same.
>
>> The problem is that RFC 3028 said that '\ <octet>' for any <octet> other
>> than <\> or <"> SHOULD be interpreted as <octet>.
>> This effectively means that we can't fix quoted strings. So I had to
>> introduced a new quoted string.
>
> First of all, having an extension that changes the interpretation of, 
> say, \x
> followed by two hex digits would IMO be far less damaging than an 
> extension
> that changes the core syntax.

Fair enough.

>> Do you have any better suggestions?
>
> There are all sorts of ways to do it. Here's an obvious one: If the
> octet-value extension is enabled, any occurances of ${x} where x takes
> the form of a space-separated list of decimal values is replaced with
> a sequence of octets corresponding to each value. Hex is a bit more
> difficult since the sequences could be confused with a variable names, so
> if you want values in hex the simplest thing would be to invent a 
> different
> escape convention, e.g. something like $%x%.

After thinking more about this (and reviewing list of options provided 
by Philip), I would like to propose a variables-like syntax close to 
what you suggested above:
 $%<hex>
controlled by a new Sieve capability "string-escape".
This would give existing implementations an ability to provide the 
escaping functionality in both quoted and multiline strings, without the 
need to implement variables.
Any objections?
If people don't like this option, my next favorite would be to change 
the interpretation of \x.

Alexey

Date: Tue, 19 Sep 2006 09:18:47 +0100
From: Alexey Melnikov <[email-masked]>
To: Aaron Stone <[email-masked]>
CC: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Aaron Stone wrote:

>On Mon, 2006-09-18 at 15:12 -0600, Philip Guenther wrote:
>  
>
>>On Mon, 18 Sep 2006, Michael Haardt wrote:
>>    
>>
>>>On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
>>>      
>>>
>>>>There are all sorts of ways to do it. Here's an obvious one: If the
>>>>octet-value extension is enabled, any occurances of ${x} where x takes
>>>>the form of a space-separated list of decimal values is replaced with
>>>>a sequence of octets corresponding to each value.
>>>>        
>>>>
>>>What a brilliant idea! How about using the same syntax variables do,
>>>and embedding arbitrary data written as a function?
>>>      
>>>
>>You mean something like option (3) from my message of March 2005?
>>    http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html
>>    
>>
>Didn't we end up zapping namespaces from the variables spec?
>  
>
No, an extension can define a namespace for variables.

Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
From: Aaron Stone <[email-masked]>
To: Philip Guenther <[email-masked]>
Cc: [email-masked]
Date: Mon, 18 Sep 2006 14:44:25 -0700
Sender: [email-masked]

On Mon, 2006-09-18 at 15:12 -0600, Philip Guenther wrote:
> On Mon, 18 Sep 2006, Michael Haardt wrote:
> > On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
> >> There are all sorts of ways to do it. Here's an obvious one: If the
> >> octet-value extension is enabled, any occurances of ${x} where x takes
> >> the form of a space-separated list of decimal values is replaced with
> >> a sequence of octets corresponding to each value.
> >
> > What a brilliant idea! How about using the same syntax variables do,
> > and embedding arbitrary data written as a function?
> 
> You mean something like option (3) from my message of March 2005?
>     http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html

Didn't we end up zapping namespaces from the variables spec?

Aaron

Date: Mon, 18 Sep 2006 15:12:48 -0600
From: Philip Guenther <[email-masked]>
To: Michael Haardt <[email-masked]>
cc: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

On Mon, 18 Sep 2006, Michael Haardt wrote:
> On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
>> There are all sorts of ways to do it. Here's an obvious one: If the
>> octet-value extension is enabled, any occurances of ${x} where x takes
>> the form of a space-separated list of decimal values is replaced with
>> a sequence of octets corresponding to each value.
>
> What a brilliant idea! How about using the same syntax variables do,
> and embedding arbitrary data written as a function?

You mean something like option (3) from my message of March 2005?
    http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html

Philip Guenther

("Over the years, I've developed my sense of deja vu so acutely that now I 
can remember things that *have* happened before ...")

From: The IESG <[email-masked]>
To: IETF-Announce <[email-masked]>
Cc: Internet Architecture Board <[email-masked]>, RFC Editor <[email-masked]>, sieve mailing list <[email-masked]>, sieve chair <[email-masked]>, sieve chair <[email-masked]>
Subject: Protocol Action: 'SIEVE Email Filtering: Spamtest and  Virustest Extensions' to Proposed Standard 
Date: Mon, 18 Sep 2006 16:39:31 -0400
Sender: [email-masked]

The IESG has approved the following document:

- 'SIEVE Email Filtering: Spamtest and Virustest Extensions '
   <draft-ietf-sieve-spamtestbis-05.txt> as a Proposed Standard

This document is the product of the Sieve Mail Filtering Language 
Working Group. 

The IESG contact persons are Lisa Dusseault and Ted Hardie.

A URL of this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-sieve-spamtestbis-05.txt

Technical Summary
 
This is a revision of RFC 3685. This document adds 3 new tests to the
SIEVE email filtering language: "spamtest", "spamtestplus" and
"virustest". Those tests permit users to use simple, portable commands
for spam and virus tests on email messages.  

This document addresses issues raised by implementors. The document
contains straightforward clarifications, updates to references and
boilerplate. In addition to that it introduces a new extension
"spamtestplus", which is basically the same as "spamtest", but allows to
test scores in percents, while the original "spamtest" test only used the
1 to 10 scale. The introduction of new scale is the result of customer
feedback: users feel more comfortable using percents. 
 
Working Group Summary
 
The SIEVE WG has reviewed the draft and discussed it at a few meetings.
Last-call (and post last-call) reviews were performed by the following
people:
- Ned Freed
- Mark E. Mallett
- Arnt Gulbrandsen
- Aaron Stone

The WG discussed the issue of obsoleting the "spamtest" extension and
decided to keep it in the document, as currently there are several
deployments of "spamtest" and no deployments of "spamtestplus".
The WG will reevaluate this decision when the time comes to move the
document to Draft.

Protocol Quality
 
Lisa Dusseault reviewed this document for the IESG.

Note to RFC Editor
 
1.  Note that this draft obsoletes RFC3685.

2.  The reference to "I-D.newman-i18n-comparator" should be moved to the
"Normative" reference section.

3.  Please remove the version-by-version changes and add the summary of
changes here:

ADD:
   Appendix B. Important changes since RFC3685.

   Listed below are the major changes from the previous specification
   [RFC3685], which this one supersedes.

   1. A ":percent" argument has been added to the "spamtest" test adding
a new 0-100 numerical range for test results.

   2. A "spamtestplus" requires item has been added to indicate the
   presence of this extension in scripts.

   3. The "count" match type from [I-D.ietf-sieve-3431bis] can now be
used to determine whether a message was tested or not.

   4. Clarified that "test not done" also means "SIEVE system could not
   determine if a test was done".

Date: Mon, 18 Sep 2006 22:20:50 +0200
From: Michael Haardt <[email-masked]>
To: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
> There are all sorts of ways to do it. Here's an obvious one: If the
> octet-value extension is enabled, any occurances of ${x} where x takes
> the form of a space-separated list of decimal values is replaced with
> a sequence of octets corresponding to each value.

What a brilliant idea! How about using the same syntax variables do,
and embedding arbitrary data written as a function?

  "${hex:deadbeef}"
  "${dec:222 173 190 239}"

Exim string operators work that way.  I didn't follow the variables
extension too close, because I consider it to be too expensive for my
needs.  If there are string operators already, adapt the above as needed,
but it would be both extensible and easier to read than backslash quoting.
If needed, someone could add "${from_base64:}".

If used without variables, the function would not expand its arguments
first, thus evaluate in linear time.  If used with variables, its argument
would be expanded first, like strings are expanded.

Michael

Date: Mon, 18 Sep 2006 21:57:51 +0200
From: Michael Haardt <[email-masked]>
To: [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Michael

Cc: [email-masked], Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 18 Sep 2006 11:18:15 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

> Alexey Melnikov writes:
> > Here is a strawman syntax for new quoted strings that would only allow
> > for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
> > sequences:
> > ...

> The capability name needs changing.

> >   quoted-string      = old-quoted-string / new-quoted-string
> >                          ; "new-quoted-string" is not allowed as
> >                          ; a parameter to "require" action.
> >                          ; "new-quoted-string" is only allowed after
> >                          ; 'require "utf8-strings";'

> You're actually talking about non-utf8-compliant-strings. "utf8-strings"
> is a poor name for that.

> I find it very difficult to summon any enthusiasm for this extension.
> Please include at least three realistic examples in the draft to show
> that it has value.

The obvious one is that I, and I suspect lots of other people, have to deal
with unencoded 8bit in various charsets in message headers. I've tried the
"reject all mail containing raw 8bit in the headers" approach and find it
catches far too much legitimate (but malformed) mail - when I had such a policy
in place here at home my wife found it impossible to correspond with several
people she works with.

Interestingy, I actually find it's better to use this as a whilelisting
mechanism than a blacklisting mechanism, that is, I sometimes set up rules
that look for bogus 8bit sequences found only in legitimate mail and use
that to keep it out of the spam folder.

				Ned

Cc: Michael Haardt <[email-masked]>, [email-masked]
To: Alexey Melnikov <[email-masked]>
Date: Mon, 18 Sep 2006 09:44:20 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

> >On Sun, Sep 17, 2006 at 11:39:53PM +0100, Alexey Melnikov wrote:
> >
> >
> >>Here is a strawman syntax for new quoted strings that would only allow
> >>for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
> >>sequences:
> >>
> >>  new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE
> >>
> >>
> >You introduce a new lexical token to Sieve, thus changing the Sieve
> >syntax.
> >
> Yes.

I have to agree with Michael - changing the basic syntax of sieve is
unacceptable.

> >I don't think the base spec allows that.
> >
> IMHO, if it explicitly prohibit such change, we need to change the base
> spec.

One of the key strengths of sieve is that the core syntax is both simple
and immutable. This makes it possible to perform syntax checks even if
you don't understand all the extensions a given script uses. Breaking 
this would IMO be hugely damaging.

> >Sieve has a very
> >small syntax, moving most stuff to the semantical layer for being very
> >extensible, pretty much like LISP does.
> >
> >The variables extension, for example, introduces the concept of expanding
> >strings, but only at the semantic level.  Syntactically, strings still
> >look the same.
> >
> >
> The problem is that RFC 3028 said that '\ <octet>' for any <octet> other
> than <\> or <"> SHOULD be interpreted as <octet>.
> This effectively means that we can't fix quoted strings. So I had to
> introduced a new quoted string.

First of all, having an extension that changes the interpretation of, say, \x
followed by two hex digits would IMO be far less damaging than an extension
that changes the core syntax.

> Do you have any better suggestions?

There are all sorts of ways to do it. Here's an obvious one: If the
octet-value extension is enabled, any occurances of ${x} where x takes
the form of a space-separated list of decimal values is replaced with
a sequence of octets corresponding to each value. Hex is a bit more
difficult since the sequences could be confused with a variable names, so
if you want values in hex the simplest thing would be to invent a different
escape convention, e.g. something like $%x%.

				Ned

Date: Mon, 18 Sep 2006 16:11:05 +0100
From: Alexey Melnikov <[email-masked]>
To: Michael Haardt <[email-masked]>
CC: [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Date: Mon, 18 Sep 2006 16:08:55 +0100
From: Alexey Melnikov <[email-masked]>
To: Michael Haardt <[email-masked]>
CC: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Michael Haardt wrote:

>On Sun, Sep 17, 2006 at 11:39:53PM +0100, Alexey Melnikov wrote:
>  
>
>>Here is a strawman syntax for new quoted strings that would only allow
>>for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
>>sequences:
>>
>>  new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE
>>    
>>
>You introduce a new lexical token to Sieve, thus changing the Sieve
>syntax.
>
Yes.

>I don't think the base spec allows that.
>
IMHO, if it explicitly prohibit such change, we need to change the base 
spec.

>Sieve has a very
>small syntax, moving most stuff to the semantical layer for being very
>extensible, pretty much like LISP does.
>
>The variables extension, for example, introduces the concept of expanding
>strings, but only at the semantic level.  Syntactically, strings still
>look the same.
>  
>
The problem is that RFC 3028 said that '\ <octet>' for any <octet> other 
than <\> or <"> SHOULD be interpreted as <octet>.
This effectively means that we can't fix quoted strings. So I had to 
introduced a new quoted string.

Do you have any better suggestions?

Date: Mon, 18 Sep 2006 11:07:27 -0400
From: Cyrus Daboo <[email-masked]>
To: Michael Haardt <[email-masked]>, [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Hi Michael,

-- 
Cyrus Daboo

Date: Mon, 18 Sep 2006 11:06:30 -0400
From: Cyrus Daboo <[email-masked]>
To: Michael Haardt <[email-masked]>, [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Hi Michael,

--On September 18, 2006 4:51:49 PM +0200 Michael Haardt 
<[email-masked]> wrote:

>> Here is a strawman syntax for new quoted strings that would only allow
>> for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
>> sequences:
>>
>>   new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE
>
> You introduce a new lexical token to Sieve, thus changing the Sieve
> syntax.  I don't think the base spec allows that.  Sieve has a very
> small syntax, moving most stuff to the semantical layer for being very
> extensible, pretty much like LISP does.
>
> The variables extension, for example, introduces the concept of expanding
> strings, but only at the semantic level.  Syntactically, strings still
> look the same.

How about if 'require string-escape' is present in the script, then ALL 
strings following that will be of the new 'escaped' form. i.e. a global 
switch covering all strings used in the script.

-- 
Cyrus Daboo

Date: Mon, 18 Sep 2006 16:51:49 +0200
From: Michael Haardt <[email-masked]>
To: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

On Sun, Sep 17, 2006 at 11:39:53PM +0100, Alexey Melnikov wrote:
> Here is a strawman syntax for new quoted strings that would only allow
> for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
> sequences:
> 
>   new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE

You introduce a new lexical token to Sieve, thus changing the Sieve
syntax.  I don't think the base spec allows that.  Sieve has a very
small syntax, moving most stuff to the semantical layer for being very
extensible, pretty much like LISP does.

The variables extension, for example, introduces the concept of expanding
strings, but only at the semantic level.  Syntactically, strings still
look the same.

Michael

Date: Mon, 18 Sep 2006 16:33:03 +0200
From: Michael Haardt <[email-masked]>
To: [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Michael

Date: Mon, 18 Sep 2006 10:15:32 -0400
From: Cyrus Daboo <[email-masked]>
To: IMAP Extensions WG <[email-masked]>, SIEVE <[email-masked]>
Subject: CalDAV uses collations - please review
Sender: [email-masked]

Hi folks,
Following IESG review, we got push-back on the text comparison operations 
used in CalDAV (Calendaring Extensions to WebDAV). As a result we have 
added support for collations as defined in 
<http://www.ietf.org/internet-drafts/draft-newman-i18n-comparator-14.txt>. 
We would appreciate it if the collation experts in the IMAP and SIEVE 
communities could review the changes and use of collations in this 
specification to ensure we are doing the right thing.

The link to the new draft is here:

<http://www.ietf.org/internet-drafts/draft-dusseault-caldav-15.txt>

An HTML diff from the previous version is here:

<http://ietf.osafoundation.org/caldav/draft-dusseault-caldav-15-from-14.diff.html>

The key section is 7.5 that describes use of collations in CalDAV.

Discussion of CalDAV is taking place at 
<mailto:[email-masked]> so please post any comments there or 
send directly to me or the other co-authors, thanks.

-- 
Cyrus Daboo

Date: Mon, 18 Sep 2006 10:05:19 -0400
From: Cyrus Daboo <[email-masked]>
To: Michael Haardt <[email-masked]>, [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Hi Michael,

But it also says:

2.4.2.4. MIME Parts

-- 
Cyrus Daboo

Date: Mon, 18 Sep 2006 09:59:20 -0400
From: Cyrus Daboo <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
cc: Alexey Melnikov <[email-masked]>
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Hi Arnt,

--On September 18, 2006 11:31:07 AM +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

>>   quoted-string      = old-quoted-string / new-quoted-string
>>                          ; "new-quoted-string" is not allowed as
>>                          ; a parameter to "require" action.
>>                          ; "new-quoted-string" is only allowed after
>>                          ; 'require "utf8-strings";'
>
> You're actually talking about non-utf8-compliant-strings. "utf8-strings"
> is a poor name for that.

Not quite - the new mechanism allows for non-utf8 strings to be 'encoded' 
as a utf-8 string via an escaping mechanism. As such strings (and indeed 
the entire script) is still utf-8 (except where current implementations 
already violate that). Perhaps a better better capability name is 
'string-escape'.

-- 
Cyrus Daboo

Date: Mon, 18 Sep 2006 12:06:43 +0200
From: Michael Haardt <[email-masked]>
To: [email-masked]
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Michael

Date: Mon, 18 Sep 2006 10:39:45 +0100
From: Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Alexey Melnikov writes:
>
>> Here is a strawman syntax for new quoted strings that would only allow
>> for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
>> sequences:
>> ...
>
> The capability name needs changing.
>
>>   quoted-string      = old-quoted-string / new-quoted-string
>>                          ; "new-quoted-string" is not allowed as
>>                          ; a parameter to "require" action.
>>                          ; "new-quoted-string" is only allowed after
>>                          ; 'require "utf8-strings";'
>
> You're actually talking about non-utf8-compliant-strings. 
> "utf8-strings" is a poor name for that.

Ok.

> I find it very difficult to summon any enthusiasm for this extension. 
> Please include at least three realistic examples in the draft to show 
> that it has value.

I will refer you to the following messages in the archive:
<http://www.imc.org/ietf-mta-filters/mail-archive/msg03105.html>
<http://www.imc.org/ietf-mta-filters/mail-archive/msg03147.html>

Date: Mon, 18 Sep 2006 11:38:28 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Alexey reminded me of some archived posts I'd forgotten. Please ignore 
my previous message.

Date: Mon, 18 Sep 2006 11:31:07 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve
Cc: Alexey Melnikov <[email-masked]>
Sender: [email-masked]

Alexey Melnikov writes:
> Here is a strawman syntax for new quoted strings that would only allow
> for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
> sequences:
> ...

The capability name needs changing.

>   quoted-string      = old-quoted-string / new-quoted-string
>                          ; "new-quoted-string" is not allowed as
>                          ; a parameter to "require" action.
>                          ; "new-quoted-string" is only allowed after
>                          ; 'require "utf8-strings";'

You're actually talking about non-utf8-compliant-strings. "utf8-strings" 
is a poor name for that.

I find it very difficult to summon any enthusiasm for this extension. 
Please include at least three realistic examples in the draft to show 
that it has value.

Arnt

Date: Sun, 17 Sep 2006 23:39:53 +0100
From: Alexey Melnikov <[email-masked]>
To: MTA filtering mailing list <[email-masked]>
Subject: Proposal for escaping on non-UTF-8 sequences in Sieve
Sender: [email-masked]

Here is a strawman syntax for new quoted strings that would only allow
for valid UTF-8 sequences, but would also allow for escaped non-UTF-8
sequences:

   new-quoted-string  = "~" DQUOTE new-quoted-text DQUOTE

   new-quoted-text    = *(utf8-quoted-safe / quoted-special /
                          quoted-non-utf8)

   not-qspecial       = %x01-09 / %x0B-0C / %x0E-21 / %x23-5B / %x5D-7F /
                        UTF8-2 / UTF8-3 / UTF8-4
                          ; a single Unicode character other than NUL,
                          ; CR, or LF, represented in UTF-8

   old-quoted-string  = DQUOTE quoted-text DQUOTE

   quoted-non-utf8    = "\" "x" 2*HEXDIG
                          ; represents a hex encoded octet

   quoted-string      = old-quoted-string / new-quoted-string
                          ; "new-quoted-string" is not allowed as
                          ; a parameter to "require" action.
                          ; "new-quoted-string" is only allowed after
                          ; 'require "utf8-strings";'

   utf8-quoted-safe   = CRLF / not-qspecial
                          ; either a CRLF pair, OR a single octet other
                          ; than NUL, CR, LF, double-quote, or backslash

As quoted strings can already contain multiple lines, I don't think we
need to introduce a new variant of the "multi-line" string. Please let
me know if you disagree.

Cc: Ned Freed <[email-masked]>, MTA filtering mailing list <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Date: Fri, 15 Sep 2006 08:12:57 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

				Ned

Date: Fri, 15 Sep 2006 15:22:06 +0100
From: Alexey Melnikov <[email-masked]>
To: Ned Freed <[email-masked]>
CC: MTA filtering mailing list <[email-masked]>
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Cc: MTA filtering mailing list <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Date: Fri, 15 Sep 2006 07:01:40 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

				Ned

Date: Fri, 15 Sep 2006 00:34:16 +0100
From: Alexey Melnikov <[email-masked]>
To: MTA filtering mailing list <[email-masked]>
Subject: non-UTF-8 sequences in Sieve scripts
Sender: [email-masked]

Hi everyone,

=====

Thanks!
Alexey

Date: Thu, 14 Sep 2006 21:02:21 +0100
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
Subject: Working Group Last Call on draft-ietf-sieve-3028bis-09.txt
Sender: [email-masked]

Hi everybody,
It has been a year since Working Group Last Call on 
draft-ietf-sieve-3028bis. There was a significant number of changes to 
the document since then, so Cyrus and I decided that the current version 
(draft-ietf-sieve-3028bis-09.txt) has to be last called again.

A two week working group last call of this document starts tomorrow on 
September 15th 2006 and ends on September 29th 2006 at 6 pm EST.

Please review this document and send issues to the list or directly to 
the editors (CCing chairs).

NB If you do review this document, but have no issues or comments, 
please send both co-chairs (or the list) an email to indicate you have 
looked at it. This will allow the chairs to measure the scope of reviews 
of this document to aid in our determination on whether to send it to 
the IESG. Thanks.

--
Alexey Melnikov
On behalf of SIEVE WG Chairs