Subject: status of 3028bis

From: Philip Guenther <[email-masked]>
Date: Tue, 18 Oct 2005 14:06:00 -0700

For those wondering what's up with the Sieve base spec, it's currently
held up waiting for the comparator draft (draft-newman-i18n-comparator-04)
to get nailed down a bit.  In particular, there's some uncertainty
over whether comparators take octets or characters as input and how
i;octet is defined and used in the definition of other comparators.
THe resolution of those issues are likely to require changes to
3028bis, so it would *really* help if more people reviewed the draft
and the discussion thereof.

The comparator draft is discussed on <[email-masked]>,
which is archived at
	http://lists.w3.org/Archives/Public/public-ietf-collation/

This draft is a direct dependency of 3028bis, the IMAP i18n draft
(draft-ietf-imapext-i18n-05), and is also holding up the completion
of the Sieve 'body' extension, so we should all be interested in
finishing it.

Philip Guenther

---

From: Michael Haardt <[email-masked]>
Date: Wed, 19 Oct 2005 12:48:58 +0200

On Tue, Oct 18, 2005 at 02:06:00PM -0700, Philip Guenther wrote:
> In particular, there's some uncertainty

We had that discussion before and you summarised it very well.  I
agree with it, but want to add reasons:

Summary:

In Sieve, comparators act on characters, not octets.  Looking only at the
base specification, "i;octet" is badly named and means: "Compare unicode
character by unicode character".  At least I think that's what we agreed
on and why the text said UTF-8 before, and unicode now.  The change from
UTF-8 to unicode was made, because although Sieve is written in UTF-8,
implementations should not be forced to work in UTF-8 inside, but use
whatever unicode encoding they like.

Reason:

Indeed, we lack the other two levels: There is no way to work on raw
headers, but at least we agreed that if we want to do that, we would
invent a new test like "rawheader".  And there is no way to work on
the MIME decoded data without character set conversion to unicode.
We have no access to the MIME character set specification, either.

If "i;octet" were to match arbitrary binary data, thus omitting the
character set translation, it would be mostly useless, as there are
ISO-8859-15 strings that are invalid UTF-8 strings and Sieve scripts
are expressed in UTF-8.  Sieve offers no binary data literals.  It is
bad enough that the decoded header data may contain NUL characters and
Sieve can not express those (\0 does not have a special meaning), but at
least we can match any unicode character else.  Since headers contain
only characters of a specific character set, and rarely just random
binary data, it makes sense "i;octet" does indeed work on decoded and
translated characters, despite its name.

If it were for me, I would: Introduce a new comparator instead,
e.g. "i;character", and change "i;octet" to match against a string
representation of binary data, e.g. hex characters.  And introduce
\0 to mean NUL at the same time. :-)

But Sieve is widespread by now and that would break many scripts.
Not even I dare to suggest going that way.  That's why I suggest:

Let "i;octet" work on characters and document the name is a misnomer.

IF someone really feels the need to specify and implement them: Introduce
Sieve extensions for a "i;binary" comparator that compares against a
string representation of binary data.  Introduce new tests "rawheader"
and "decodedheader".

I am not aware how "i;octet" is used at other places, so the above may
only fit well to Sieve.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 20 Oct 2005 14:59:04 +0100

>>i;octet is defined and used in the definition of other comparators.
[...]

>I am not aware how "i;octet" is used at other places, so the above may
Your suggestion make sense for header fields. But what about body 
extension when matching against application/* MIME part?

---

From: Michael Haardt <[email-masked]>
Date: Thu, 20 Oct 2005 16:23:31 +0200

> extension when matching against application/* MIME part?

I don't think the MIME type should make a difference on what a test or
comparator does, or how the literal used for comparison is represented.
Things get too confusing otherwise.

That having said, using "i;octet" could not be used reasonably on decoded
MIME parts that can not be translated to unicode.  Trying to do so should
yield the same behaviour as it does on headers that can not be translated
to unicode, e.g. because their character set is not known.

The imaginary "i;binary" comparator would be the proper solution to
compare with an octet sequence, working on the decoded part, if we tried
to keep a minimum of logic.  And I really hope the extension does not
specify what exactly "i;octet" does to let me get away with saying: I am
not breaking it.

Honestly, I never looked at the body extension, because it is way
too expensive for my systems.  Skimming it, it does not talk about
comparators and assumes they work as defined by external references,
which is the way it should be.

Michael

---

From: Dave Cridland <[email-masked]>
Date: Thu, 20 Oct 2005 19:44:05 +0100

> extension when matching against application/* MIME part?
We need to restrict this discussion to just the one mailing list, 
really, but I've posted a message saying that actually, the reverse 
is true - comparators match on octet strings, and happen to have a 
decode built in - hence i;octet doesn't decode, and i;ascii-* both 
decode using ASCII.

The notion that comparators work on character strings is a notion 
that comes pre-flawed - ACAP does not operate on character strings, 
but octet strings, which might on a good day happen to be UTF-8 
encoded text, but might be anything.

I've also suggested that where all the protocol has is a character 
string, then the semantics of a comparator must behave as though the 
string were encoded using UTF-8 (possibly by actually doing so).

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Oct 2005 12:27:43 -0700 (PDT)

> decode built in - hence i;octet doesn't decode, and i;ascii-* both

Wow, I'm sure glad you said this, because this has always been my
interpretation as well, and I was getting the idea I was all alone in seeing it
this way. It definitely is how our implementation works too.

> encoded text, but might be anything.

Yep.

> string were encoded using UTF-8 (possibly by actually doing so).

But not necessarily by doing so. In many cases it is more convenient to decode
to a fixed-length representation like UTF-32. The behavior is what matters, not
the implementation details.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Fri, 21 Oct 2005 00:03:11 +0200

> decode built in - hence i;octet doesn't decode, and i;ascii-* both 

What exactly do you mean by "decode"? Removing the MIME encoding or
converting the character set?

> encoded text, but might be anything.

That explains why we have that mess.  Over here, users certainly expect
"en;ascii-case" to match characters, and will be confused if the first
test is true and the second is not, and yet more, if both are false:

Subject: =?utf8?q?A=c3=a4?=

:comparator "en;ascii-casemap" :matches "a?"
:comparator "i;octet" :matches "A?"

If "i;octet" operates on octets, we can't talk of unicode, but need
to talk about UTF8 for comparisons, and users will ask instantly:
How can I match characters case sensitive? The base spec makes me think
"i;octet" is just that, and operating on characters, despite the name.
Section 2.7.1, Match Type, does not mention octets anywhere.

> string were encoded using UTF-8 (possibly by actually doing so).

Are you saying that even using "en;ascii-casemap", the wildcard "?"
does not match a single character outside US-ASCII?

Alexey: No matter how this turns out, could we add the above example
including the result to the base spec?

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 21 Oct 2005 02:38:29 +0200

> converting the character set?

I'm not Dave, but I would mean "decode UTF-8 octet sequences to Unicode
characters".  RFC 2047 and RFC 2231 decoding will always be done.  BTW,
the draft does not mention RFC 2231 at all, I think that should be more
explicit, since it has explicit references to RFC 2047.

> :comparator "en;ascii-casemap" :matches "a?"

no.  if you change the pattern to "a?*", and have variables, ${1} would
hold U+00C3 (not U+00E4, the decoded value).

> :comparator "i;octet" :matches "A?"

no.  same as above.

> "i;octet" is just that, and operating on characters, despite the name.

well, to me it is quite obvious that when it says "octet", it doesn't
support multibyte encodings, but rather operates on raw byte values.
RFC 2244 is quite explicit:

     For collation, the i;octet comparator interprets the value of
     an attribute as a series of unsigned octets with ordinal
     values from 0 to 255.

the base spec does not support caseful matching where "?" makes sense in
non-ASCII.  in fact, the latest comparator draft I can find doesn't
provide one either.  the algorithm for en;ascii-casemap is quite
amusing:

   The "en;ascii-casemap" collation is a simple collation intended for
   use with English language text in pure US-ASCII.  It provides
   equality, substring and ordering functions.  The algorithm first
   applies a canonicalization algorithm to both input strings which
   subtracts 32 (0x20) from all octet values between 97 (0x61) and 122
   (0x7A) inclusive.  The result of the collation is then the same as
   the result of the "i;octet" collation for the canonicalized strings.

(the algorithm in RFC 2244 is essentially the same.)

this was surprising and interesting to me, since it means that with
"abc" :matches "ab?", ${1} will hold the uppercase "C"!  I wonder how
many users would expect that one, or how many implementations get it
right.

> does not match a single character outside US-ASCII?

since the spec defines it in terms of i;octet, the "?" wildcard is
essentially broken.  it gets really interesting with "*", though, since
you will probably get doubly encoded UTF-8 :-(

this is truly a mess.  we need a comparator which makes sense, a
comparator which operates case-fully on Unicode characters, but without
the difficult bits (e.g. normalising combining characters).

-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Fri, 21 Oct 2005 12:29:13 +0200

> many users would expect that one, or how many implementations get it

Oh my god.  Basically, this means a Sieve implementation does not
know anything about its input being UTF8.  All it does is converting
headers to UTF8, but other than that, it works on octets, not characters.

If so, we must not talk about unicode, but UTF8, and s/character/octet/g.
And I have to change my implementation, which works on characters,
crippling it.

Do all implementations else work on octets instead of characters?

> you will probably get doubly encoded UTF-8 :-(

Worse, matched octets can be invalid (incomplete) UTF8 octet-sequences.
You gave an example how variables extract a single octet of a sequence,
and you can't even control if ? matches an incomplete UTF8 octet-sequence
or a US-ASCII character: You can not use the matched part to modify or
generate mails or you risk to ruin the result.

> the difficult bits (e.g. normalising combining characters).

Indeed!

Michael

---

From: Dave Cridland <[email-masked]>
Date: Sat, 22 Oct 2005 12:41:33 +0100

> headers to UTF8, but other than that, it works on octets, not 
Yep. But don't be too alarmed, this only breaks :matches.

Incidentally, I really suspect that :matches in this case should be 
returning "c", not "C", and it depends heavily on a lot of 
specification that I suspect doesn't exist.

> Do all implementations else work on octets instead of characters?
It actually doesn't matter. All the comparators so far defined at RFC 
level (ie, by ACAP), operate on octets, not characters, but assuming 
your implementation (modulo :matches) operates such that *if* all 
comapators operated on UTF-8 encoded octet strings rather than 
Unicode character strings, then this is easily rectifiable.

:matches is the problem.

> generate mails or you risk to ruin the result.
Yes, true. Fun, isn't it?

> > the difficult bits (e.g. normalising combining characters).
This is a very, very quick proposal that I've had in the back of my 
mind, but the use of :matches convinces me that this is the way to go.

The current comparator draft changed the name of comparators to 
collations - I thought that was wrong, because I'd have liked to see 
comparators that performed pattern matching, which precludes their 
ability to collate.

I'd like to propose:

1) A family of comparators for both UTF-8 and Octet matching, the 
matches themselves being Globs (as Sieve :matches), and regular 
expressions. (perhaps Basic and Extended). This is off the top of my 
head, and it's a Saturday, so I can't recall the conventions for 
comparator naming, but I'll use the example of "i;utf-8;glob".

This comparator would perform the "EQUAL" operation such that if the 
left hand side were matched by the pattern contained in the 
right-hand side, it returned true, otherwise false.

"SUBSTRING" would match if the pattern occured anywhere in the 
string, and "PREFIX" if it occured at the beginning of the string.

We need at least a UTF-8 glob to replace :matches, and a case-folding 
variant, too.

2) A new API requirement that "submatches" may be extracted from the 
result of a successful EQUAL, PREFIX, or SUBSTRING match.

3) In Sieve, :matches becomes deprecated, and for backwards 
compatibility, it should be rewritten such that:

a) The comparator is changed from "i;octet" to "i;utf-8;glob", and 
from "i;ascii-casemap" to "en;utf-8;casemap;glob".

b) The operation is changed from :matches to :is

The net result is not only that :matches now works as people expect, 
but that regular expression searching and glob pattern matching 
become available in ACAP and IMAP, too.

So I can do something like EQUAL "addressbook.Email" 
"en;utf-8;casemap;regex" "dave(\\+[^@]*)?@cridland.net(\0[a-z]*)?" 
when searching my addressbook, which'd keep me happy.

Dave.

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Oct 2005 14:02:38 -0700 (PDT)

> know anything about its input being UTF8.

Quite correct. IMO this is a feature, not a bug.

> All it does is converting

Well, it has always been possible to specify a comparator that operates
on characters rather than octets. I've had the hooks in place for this
in our implementation waiting for the comparator stuff to be completed.

> And I have to change my implementation, which works on characters,

There are advantages as well as drawbacks. The ability to match illegal gunk is
something I've used more than once, and I know that some of our cusstomers have
used it too.

> Do all implementations else work on octets instead of characters?

Ours does.

> Worse, matched octets can be invalid (incomplete) UTF8 octet-sequences.

The underlying structure of UTF-8 itself tends to minimize the liklihood of
this happening. As long as you're dealing with well formed UTF-8 inputs there
is no way for part of one character to match part of another. (This is one of
the real advantages of UTF-8 over, say, UTF-16 - you can perform byte by byte
comparisons and most things end up working properly. :matches and ? are an
obvious exception, of course.)

Of course once you leave the realm of well formed utf-8 this no longer
applies.

> generate mails or you risk to ruin the result.

Assuming:

(1) An octet-based comparator.
(2) A single ? used in isolation with no adjacent *s or ?s.
(3) Well formed UTF-8 as input.

The somewhat surprising result is that ? can only match an ASCII character. Of
course something like ???? can get really interesting and match anything that
encodes down to four octets. 

> > the difficult bits (e.g. normalising combining characters).

Well, we have quite a large user based on this stuff and most of these
issues have proved to be theoretical. Variables may change this, however.
A construct like:

    require "variables";
    if header :matches "subject" "*" {set "subject" "${1}"}
    else {set "subject" ""}

ends up storing the subject in all caps, which likely isn't what was intended.
For this to work properly you need to say:

    require "variables";
    if header :comparator "i;octet" :matches "subject" "*" {set "subject" "${1}"}
    else {set "subject" ""}

I believe that some of the examples in the variables draft make have similar
mistakes in them.

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Sun, 23 Oct 2005 23:29:08 +0100

> Quite correct. IMO this is a feature, not a bug.
Hmmm... I've been reading about :matches, and I'm not so sure there 
isn't a bug here.

:matches has a problem - its specification talks about characters, 
and in particular, "?" matches 'a single character'.

This is all very nice, but comparators don't operate on characters 
(or if they do, they get to define what a character *is*), so exactly 
what this means is vague. It really doesn't help that neither ACAP 
nor the new "collations" draft mention any kind of matching operation 
- its sole definition is in one paragraph in RFC3028 or the new draft.

> encodes down to four octets. 
I think you intended to say that "?" can only match a character if it 
is within ASCII - or more generally, if it happens to encode to a 
single octet in UTF-8. But it'll match any octet, of course, whatever 
character it might happen to be part of the encoding for.

> > > the difficult bits (e.g. normalising combining characters).
I'm certainly in full agreement there.

> ends up storing the subject in all caps, which likely isn't what 

I think that's a matter of interpretation.

Variables says, in section 3.2, that the list variables expand to 
what the wildcard matched.

I see nothing saying that this must be in the internal transformation 
of the string by a comparator (if such a thing exists), nor that it 
should be those matching portions of the original string, but my gut 
feeling is that a comparator should be essentially a black box - that 
is, the internal transformations of the comparator shouldn't be 
visible to the script.

This is especially true since some comparators don't transform to a 
string internally - not that we should be trying matches with 
"i;ascii-numeric" (and why is this remaining "i;" if we have to 
change "i;ascii-casemap"? ASCII digits aren't available everywhere. 
Perhaps I'm being naïve again.)

Moreover:

require "variables";
if header "subject" :comparator "i;ascii-casemap" :matches "[foo] *" {
	set "subject" "${1}";
} else {
	set "subject" "${1}";
}

would be a whole load more useful if ${subject} was a substring of 
the original subject, rather than an upper-case variant of it.

> I believe that some of the examples in the variables draft make 

Different interpretations, rather than mistakes.

Dave.

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Oct 2005 15:47:53 -0700 (PDT)

> Hmmm... I've been reading about :matches, and I'm not so sure there

A specification bug, perhaps.

> - its sole definition is in one paragraph in RFC3028 or the new draft.

Some comparators do octets, others may group those octets in some way,
including but not limited to grouping them into things we call characters.
So it's a single something, with "something" being defined by the underlying
comparator.

I don't know the best way to write this, but that's in effect what it needs to
say.

> character it might happen to be part of the encoding for.

Yes, but this fails to take the context into account. There are three subcases
where ? appears at the beginning, middle, or end of the pattern.

If ? appears at the beginning and the target begins with a non-ASCII character,
the ? will match the first octet of that character. But since this is well
formed UTF-8 there has to be a second byte and that byte cannot possibly
match the next thing in the pattern. So the the match always fails unless
the first character of the target is ASCII.

I won't bother to walk through the remaining cases, but they all end up the
same way: The ? may match a non-ASCII character in isolation, but unless it is
adjacent to some other wildcard there is no way the entire pattern can
subsequently match.

> > Well, we have quite a large user based on this stuff and most of
> > these issues have proved to be theoretical. Variables may change this,
> > however.
> > A construct like:
> >
> >     require "variables";

> what the wildcard matched.

But what the wildcard matched was transformed text. Try as I might, I cannot
read what's there the way you do.

> is, the internal transformations of the comparator shouldn't be

Although it will be some work for me to change my implementation to match
this, I would be happy to make such a change because I think the resulting
behavior is better. However, if this is what people want it needs to be
made explicit in the variables specification. I think your reading of
what's there now is a huge stretch.

> Perhaps I'm being naïve again.)

IMO we have much too much deployed code to change the names of any of these
things. At the very most we can add new names and make the new names the
preferred names. The old names have to remain and be recognized indefinitely.

> the original subject, rather than an upper-case variant of it.

Yes, that's the case that argues strongly for your approach. It is also worth
nothing that those who want to force things into a particular case can do so
with the various modifiers to set.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 24 Oct 2005 02:12:54 +0200

> character it might happen to be part of the encoding for.

yes, but since the argument to :matches has implicit anchors, another
wildcard needs to follow the ?.  e.g., with "foo?", only US-ASCII can be
matched, since all UTF-8 sequences are multi-octet.

> is, the internal transformations of the comparator shouldn't be 

yes, the behaviour of wildcard matching is under-specified (you might
say "unspecified"), and trying to extrapolate it from the matching
algorithm is probably wrong, especially since no one will want that
behaviour.
-- 
Kjetil T.