Subject: Re: [Imap-protocol] Private FETCH items
From: Bron Gondwana <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: Bron Gondwana <[email-masked]>, Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
Date: Fri, 28 Dec 2007 14:13:50 +1100
List-Id: <ietf-imapext.imc.org>

On Thu, Dec 27, 2007 at 03:16:07PM -0700, Lyndon Nerenberg wrote:
>
>
> On 2007-Dec-27, at 15:09 , Bron Gondwana wrote:
>
>> You're lucky if you've never seen file corruption due to bitrot on large
>> drive arrays.  We have about 30TB online at the moment and that's
>> growing fairly rapidly.  The research out there suggests we're likely to
>> have a block level corruption every couple of months with the current
>> level of affordable component reliability.
>
> But that's a local hardware issue, and should be handled locally to the 
> host. I.e. have the IMAP server checksum the files when writing to disk, 
> and verify that checksum when reading the data. Doing random checks via the 
> IMAP protocol won't prevent you from offering corrupted data to clients. 
> You have the right solution, but you're applying it at the wrong layer.

Certainly - though once you put replication in the mix then there needs
to be some way to double check things all the way down.  I agree that 
it doesn't need to be exposed all the way out to IMAP.  I just put it
there for convenience.

That said, if the client wants to confirm that it has the same data as
the server, then there needs to be some way for the client to extract
a digest from the server that it can compare to its local copy.  It's
an implementation detail whether this gets calculated on the fly or
is pulled from a cache.

It would be nice to be able to tell the client which digest is already
cached/pre-calculated by the server to save overloading.

Bron.

Date: Fri, 28 Dec 2007 09:35:47 +1100
From: Bron Gondwana <[email-masked]>
To: Ken Murchison <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Fri, Aug 24, 2007 at 04:42:18PM -0400, Ken Murchison wrote:
>
> Arnt Gulbrandsen wrote:
>> Ken Murchison writes:
>>> Arnt Gulbrandsen wrote:
>>>> What does the GUID do?
>>>
>>> Uniquely identifies a message so that if the same message is present in 
>>> more than one mailbox, its contents only gets replicated once.
>> I'm afraid I phrased the question badly. What does the code do when asked 
>> for a GUID?
>
> Pulls the message GUID out of the index file for the mailbox.

This is currently 20 bytes containing the SHA1 of the rfc822 message
bytes, and has been for a while.  It used to be 12 bytes, either
(0x01, hostid, process id, time) or later I added
(0x02, first 11 bytes of md5 digest)

More recently David Carter rewrote it as a 20 bytes value containing
the full sha1, which is not only cryptographically better but it's
complete!  The partial MD5 was a hack to avoid changing the index
file format while still allowing some level of integrity checking.
I knew it was a nasty compromise - but hey, life is built on nasty
compromises.

Bron.

Date: Fri, 28 Dec 2007 09:30:22 +1100
From: Bron Gondwana <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: [email-masked], IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Mon, Aug 27, 2007 at 11:31:20AM +0100, Tony Finch wrote:
> On Thu, 23 Aug 2007, Ken Murchison wrote:
> >
> > I probably wasn't clear in the use case.  Cyrus now has the ability to
> > replicate a mailstore in near realtime.  Several sites (including CMU in the
> > near future) are using this to have redundant mailstores in case of a hardware
> > failure.  Rather than create a new tool or protocol to verify the consistency
> > of the mailstore pairs, extending IMAP with a couple of new FETCH items seems
> > like a reasonable thing to do.
> 
> I should note that the MD5 replica consistency check has been very useful
> in finding lurking bugs in the code, both the original version and across
> versions as the code has evolved. It's also good for discovering bit-
> smashing hardware failures that haven't (yet) affected lower levels of the
> os/driver/raid stack. You can only be sure of data integrity if you test
> it end-to-end.

Thanks :)  My sentiments exactly.

We have layers of integrity checks and scripts watching other scripts to
maintain our replication system.  Even without the replication, it's
good to record the digest of a delivered message into an index file as
it's delivered and be able to compare that to the spooled file itself
at any later time as an integrity check.

Just this week I've been working on an auditlog patch to cyrus:

http://cyrus.brong.fastmail.fm/patches/cyrus-auditlog-2.3.11.diff

This spits out the GUID (sha1) at APPEND time as well as any time a
COPY is performed on the message.  In the same message is the mailbox
name, mailbox unique id (Cyrus internal) and UID.  This is duplicated
on the replica, so by combining those logs you can see that the same
message exists on both ends.

Our backup system fetches index files, and then uses the sha1 to
compare the message pulled from the spool and ensure it's backing
up the message it expects to be backing up.

Integrity checking is the responsibility of every layer, at least
for the semantic things that only that layer understands.

Bron.

Date: Fri, 28 Dec 2007 09:23:09 +1100
From: Bron Gondwana <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, Ken Murchison <[email-masked]>, [email-masked], IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Wed, Aug 22, 2007 at 02:14:14PM -0700, Mark Crispin wrote:
> On Wed, 22 Aug 2007, Lyndon Nerenberg wrote:
>> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
>> representation is, and comparing file sizes is NOT the same as comparing 
>> message content.
>
> I agree.  The c-client library (the beating heart of UW imapd, Pine, etc.) 
> has a concept of an internal size, but that depends upon the mail store and 
> thus is only used by mail store drivers.  It never gets as far as imapd (or 
> other application), much less exported in the protocol.
>
> Let us be clear here; any .FILESIZE value in any server which supports 
> multiple mail store formats could (and WOULD) vary depending upon the 
> mailbox format with no obvious indication to the user as to why.  This, by 
> itself, should be enough cause to reject it.

That's fine with me.  .FILESIZE is really part of an "internal integrity
check" and was only exposed via IMAP because there's no easy way to
compare \Seen flags in Cyrus other than logging in as the user.  Once
you've done that it's easier to have everything else you need to compare
available through the same connection.

>> .MD5 is useful in the general sense as a fast way to compare messages for 
>> equality, although I think it would be better served in that context if 
>> you could MD5 the body content without the 822 headers to be able to 
>> handle Received: differences for the same message delivered to two or more 
>> folders via different [SL]MTP transactions.  It would probably make more 
>> sense to be able to generically MD5 by MIME body part.
>
> IMHO, this functionality belongs as part of CONVERT.  An MD5 checksum is, 
> in effect, a conversion.

So long as it's specified in a way that makes it easy for the server to
cache so that requests for the value are cheap, and it's a digest that
can be calculated at the other end as an integrity check, I'm pretty
happy.  I'd also like all implementation to be using the SAME value so
it's not just unique within a particular implementation but can be used
to track a series of bytes regardless of who is storing them.

>> What's a GUID?
>
> Presumably this is a global UID which uniquely and permanently identifies 
> the message on the server.
>
> I think that IMAP should eventually have a GUID, but it needs careful 
> design and review.  I don't think that we should take the first patch that 
> comes along.

Excellent.  That's what I'm here for.  The intention with the first
patch was purely to have an internal integrity check.  This time I want
everyone else to be doing it as well.  Now that I've found the right
forum I'd like to start work on a proper design, and I'd love input on
it.

>> But overall I guess I have to question why it's necessary to burden the 
>> protocol with this just because somebody wrote some broken code.  I have 
>> done an awful lot of server migrations over the years and I have never yet 
>> come up against the problem this is trying to detect. I realize you intend 
>> this as private data, but once it's in the wild, it will get used, for 
>> better and worse.
>
> I agree with this sentiment.

Definitely.  I'm pretty sure I tagged that Cyrus patch Ken is talking
about "Not a candidate for upstream without further review" when I was
submitting the series of patches it was in.  I've got a few patches
like that floating around.

Thanks,

Bron.

Cc: Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
From: Lyndon Nerenberg <[email-masked]>
To: Bron Gondwana <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Date: Thu, 27 Dec 2007 15:16:07 -0700
Sender: [email-masked]

On 2007-Dec-27, at 15:09 , Bron Gondwana wrote:

> You're lucky if you've never seen file corruption due to bitrot on  
> large
> drive arrays.  We have about 30TB online at the moment and that's
> growing fairly rapidly.  The research out there suggests we're  
> likely to
> have a block level corruption every couple of months with the current
> level of affordable component reliability.

But that's a local hardware issue, and should be handled locally to  
the host. I.e. have the IMAP server checksum the files when writing to  
disk, and verify that checksum when reading the data. Doing random  
checks via the IMAP protocol won't prevent you from offering corrupted  
data to clients. You have the right solution, but you're applying it  
at the wrong layer.

--lyndon

Date: Fri, 28 Dec 2007 09:16:22 +1100
From: Bron Gondwana <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, [email-masked], [email-masked], Ken Murchison <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Wed, Aug 22, 2007 at 10:13:42PM +0100, Dave Cridland wrote:
> Note that I've proposed "CRC" instead of explicitly naming MD5, partly in 
> case people are tempted to use it in clients, and partly to stop the flurry 
> of emails you'll otherwise get telling you that MD5 is insecure.

Two things here.

1) I consider auditability to be very important.  I want to
   be able to cheaply fetch some form of digest (preferably
   cheaply for both the server and the network) that allows
   me to check that the local message is the same.

More on this - it allows detection of both copies and folder
renames to occur without fetching the full content of the
message again.  Even without detecting two different deliveries
with different Received headers, this is a massive win.  Indeed,
it's 90+% of the duplication I would expect to see.  Multiple
deliveries with different received headers is much rarer than
re-filing a message to a different folder (at least in our users'
usage)

2) I'd prefer this to be usable across multiple different server
   implementations.  My current goal (and the reason I'm re-visiting
   this rather than just maintaining yet another patch against Cyrus.
   I already have about 20, what's one more!) is to build an
   intelligent IMAP proxy/synchroniser.  Having to write separate
   logic for every IMAP server out there would suck.

Assuming this makes it as far as a draft, I would be happy to work
on adding support to a few of the most popular open-source servers
and possibly a client or two to bootstrap support.  I think it's
worth the effort.

Bron.

Date: Fri, 28 Dec 2007 09:09:34 +1100
From: Bron Gondwana <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

Ducking in to follow up on this thread as the author of the
patch Ken is talking about.  I work at FastMail.FM, and have
been doing an awful lot of work patching Cyrus recently.

I have signed up to the imapext mailing list now (should have
before!) - I can't see [email-masked] on the public
lists page, so have not signed up there.  I'd be happy to join
if that's allowed though.

On Wed, Aug 22, 2007 at 12:27:15PM -0700, Lyndon Nerenberg wrote:
> On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:
>
>> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that the 
>> former uses the cached size in the message index file, and the latter 
>> stat()s for the file size at the time of the request.
>
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

I added these options originally to support our replication integrity
checking script.  We run Cyrus replication, and it had quite a few
bugs at first.  We still run it as an advance warning if something goes
wrong.

I wanted some method to access the underlying file size and digest to
compare, and I agree that FILESIZE is not ideal and shouldn't be
exposed publically.  It's useful not to have to use a separate protocol
or direct filesystem lookup for us though.

We find that with multiple terabytes of disk and non-ZFS filesystems
(ZFS didn't even exist when I started work on this) there are occasional
underlying errors.  There needs to be integrity checking up at the layer
which understands the semantic meaning of the files.

> .MD5 is useful in the general sense as a fast way to compare messages for 
> equality, although I think it would be better served in that context if you 
> could MD5 the body content without the 822 headers to be able to handle 
> Received: differences for the same message delivered to two or more folders 
> via different [SL]MTP transactions.  It would probably make more sense to 
> be able to generically MD5 by MIME body part.

There are two different values we have now, 'GUID' and 'RFC822.SHA1'.
The first is indexed and fast, the second is calculated by reading the
bytes directly from disk.  We check that for 1/1000 messages at the
moment in a standard replication scan.  Enough to find a systematic
issue fairly quickly without bringing our system IO to its knees.

I have recently re-written the GUID part as "DIGEST.SHA1" and added
a CAPABILITY line "DIGEST=SHA1" with a view to making it extensible
to other message digests in future.  Cyrus now stores the GUID as a
SHA1.
> What's a GUID?

Globally Unique ID.

> But overall I guess I have to question why it's necessary to burden the 
> protocol with this just because somebody wrote some broken code.  I have 
> done an awful lot of server migrations over the years and I have never yet 
> come up against the problem this is trying to detect. I realize you intend 
> this as private data, but once it's in the wild, it will get used, for 
> better and worse.

You're lucky if you've never seen file corruption due to bitrot on large
drive arrays.  We have about 30TB online at the moment and that's
growing fairly rapidly.  The research out there suggests we're likely to
have a block level corruption every couple of months with the current
level of affordable component reliability.

Bron ( now to reply to some other messages )

Date: Mon, 24 Dec 2007 00:24:52 +0300
From: Alexey Melnikov <[email-masked]>
To: Pete Resnick <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Pete Resnick wrote:
> On 12/16/07 at 11:45 PM +0000, Alexey Melnikov wrote:
>> but I agree it was not entirely clear and I reworded the text along 
>> the lines of your proposal:
> I'm going to go ahead and write up -14 for IETF Last Call and send it 
> to Lisa. We'll consider Timo's comments and your clarification part of 
> that last call and make the correction there.
This is fine with me.

Date: Thu, 20 Dec 2007 15:19:42 -0600
To: Alexey Melnikov <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Timo Sirainen <[email-masked]>, ietf-imapext <[email-masked]>
Sender: [email-masked]

On 12/16/07 at 11:45 PM +0000, Alexey Melnikov wrote:

>but I agree it was not entirely clear and I reworded the text along 
>the lines of your proposal:

I'm going to go ahead and write up -14 for IETF Last Call and send it 
to Lisa. We'll consider Timo's comments and your clarification part 
of that last call and make the correction there.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sun, 16 Dec 2007 23:45:54 +0000
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen wrote:

>On Sun, 2007-12-16 at 15:30 +0200, Timo Sirainen wrote:
>  
>
>>I just realized it's possible for collations to return "undefined" even
>>if all the input characters are valid.
>>    
>>
>(Continuing to talk to myself..)
>
>Am I correct about this? To me it looks like RFC 4790 defines "valid
>string" as "passes validity test", which may not be as simple as "all
>characters have valid code points".
>  
>
Correct.

>If this is correct, then the i18n draft needs changing, since currently
>it only specifies what to do with charset decoding failures. 4.6.c
>should then be something like:
>
>"All strings are collated using the active comparator. If the comparator
>returns "undefined", the string is moved to a second list. The second
>list is sorted using i;octet and appended after the first sorted list."
>  
>
This was implied by the following text:

  The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate the 
following
  together: strings encoded using unknown or invalid character 
encodings, strings
  in unrecognized charsets, and invalid input (as defined by the active 
collation).

but I agree it was not entirely clear and I reworded the text along the 
lines of your proposal:

    All strings that were successfully converted by step (b) are
    separated from all strings that failed step (b). Strings in
    each group are collated independently.  All strings successfully
    converted by step (b) are then validated by the active
    comparator. Strings that pass validation are collated using the
    active comparator. All strings that either fail step (b) or fail
    the active collation's validity operation are collated (after
    applying step (a)) using the i;octet comparator (see [RFC4790]).
    The resulting sorted list is produced by appending all collated
    "failed" strings after all strings collated using the active
    comparator.

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 15:54:31 +0200
Sender: [email-masked]

--=-BlSSngy9BBzqQkYjbPDT
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 15:30 +0200, Timo Sirainen wrote:
> I just realized it's possible for collations to return "undefined" even
> if all the input characters are valid.

(Continuing to talk to myself..)

Am I correct about this? To me it looks like RFC 4790 defines "valid
string" as "passes validity test", which may not be as simple as "all
characters have valid code points".

If this is correct, then the i18n draft needs changing, since currently
it only specifies what to do with charset decoding failures. 4.6.c
should then be something like:

"All strings are collated using the active comparator. If the comparator
returns "undefined", the string is moved to a second list. The second
list is sorted using i;octet and appended after the first sorted list."

--=-BlSSngy9BBzqQkYjbPDT
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-BlSSngy9BBzqQkYjbPDT--

Date: Sun, 16 Dec 2007 14:51:35 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> On Sun, 2007-12-16 at 13:18 +0100, Arnt Gulbrandsen wrote:
>>  > Anyway having to specifically track a list of valid and non-valid
>>  > strings and sort them separately would require adding a lot of new
>>  > code, all just to comply with a specification that works worse (from
>>  > user's POV) than what I could have implemented much more easily.
>>
>>  As I see it, you're adding support for all codepoints and all character
>>  sets,
>
> When it gets to collation, yes, but is this really allowed for 
> implementing SORT?

Why not?

>>  so the exception we're discussing no longer applies to your code.
>
> Why would anyone do anything different then?

Beats me.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 15:30:23 +0200
Sender: [email-masked]

--=-3jgGaz40hdiWy2jWA9s9
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I just realized it's possible for collations to return "undefined" even
if all the input characters are valid. So I guess I'll have to redefine
my request a bit.

Change:

4.6.

(b) The decoded text from (a) MUST be converted to the charset expected
by the active comparator.

to something like:

(b) The decoded text from (a) MUST be converted to the charset expected
by the active comparator. The server MAY/SHOULD handle failures (e.g.
the text is in an unknown charset, contains a sequence which is not
valid according in that charset, etc.) by replacing the failed part with
a unicode "replacement character" #FFFD (or equivalent in other
charsets).

--=-3jgGaz40hdiWy2jWA9s9
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-3jgGaz40hdiWy2jWA9s9--

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 14:47:44 +0200
Sender: [email-masked]

--=-oD5TbLhPGjl8mMfeyHnP
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 13:18 +0100, Arnt Gulbrandsen wrote:
> > Anyway having to specifically track a list of valid and non-valid=20
> > strings and sort them separately would require adding a lot of new=20
> > code, all just to comply with a specification that works worse (from=20
> > user's POV) than what I could have implemented much more easily.
>=20
> As I see it, you're adding support for all codepoints and all character=20
> sets,=20

When it gets to collation, yes, but is this really allowed for
implementing SORT?

> so the exception we're discussing no longer applies to your code.

Why would anyone do anything different then? I see only benefits doing
it this way.

--=-oD5TbLhPGjl8mMfeyHnP
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-oD5TbLhPGjl8mMfeyHnP--

Date: Sun, 16 Dec 2007 13:18:57 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen writes:
> On Fri, 2007-12-14 at 04:46 +0000, Alexey Melnikov wrote:
>>  >Would be better if this rule applied only at the position where the
>>  >unknown data was found.
>>  IMAP doesn't feed partial strings into collations. Describing this 
>>  would be a nightmare and I am not sure it is worth the effort.
>
> Since no-one's talking, I'll continue..
>
> The way I was going to implement it was to just replace 
> invalid/unknown input with a special unicode character and have it 
> get sorted in whatever way it normally gets sorted. This would all be 
> done before the string is fed to collation.

I do that sometimes. And sometimes I map to some private use codepoints.

> I'm not yet sure what the special character would be, but IIRC there 
> were some control characters in unicode already that user wasn't 
> allowed to write. Or maybe it could be just some non-coding 
> character, like the highest valid UTF-8 character. That would make it 
> automatically get sorted to the end.

You can use U+FFFD, which means "a character I couldn't represent while 
translating to unicode", or U+0F0000 to U+10FFFD, which are reserved 
for private use.

> Anyway having to specifically track a list of valid and non-valid 
> strings and sort them separately would require adding a lot of new 
> code, all just to comply with a specification that works worse (from 
> user's POV) than what I could have implemented much more easily.

As I see it, you're adding support for all codepoints and all character 
sets, so the exception we're discussing no longer applies to your code.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 14:00:18 +0200
Sender: [email-masked]

--=-FNHi/+1E/hFbVXeRRGp8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 13:43 +0200, Timo Sirainen wrote:
> I'm not yet sure what the special character would be, but IIRC there
> were some control characters in unicode already that user wasn't allowed
> to write. Or maybe it could be just some non-coding character, like the
> highest valid UTF-8 character. That would make it automatically get
> sorted to the end.

#fffd I suppose.=20
http://www.fileformat.info/info/unicode/char/fffd/index.htm

--=-FNHi/+1E/hFbVXeRRGp8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-FNHi/+1E/hFbVXeRRGp8--

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 13:43:39 +0200
Sender: [email-masked]

--=-747T55IOzcaOcTQMdE66
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-12-14 at 04:46 +0000, Alexey Melnikov wrote:
> >Would be better if this rule applied only at the position where the
> >unknown data was found.
> IMAP doesn't feed partial strings into collations. Describing this would=20
> be a nightmare and I am not sure it is worth the effort.

Since no-one's talking, I'll continue..

The way I was going to implement it was to just replace invalid/unknown
input with a special unicode character and have it get sorted in
whatever way it normally gets sorted. This would all be done before the
string is fed to collation.

I'm not yet sure what the special character would be, but IIRC there
were some control characters in unicode already that user wasn't allowed
to write. Or maybe it could be just some non-coding character, like the
highest valid UTF-8 character. That would make it automatically get
sorted to the end.

Anyway having to specifically track a list of valid and non-valid
strings and sort them separately would require adding a lot of new code,
all just to comply with a specification that works worse (from user's
POV) than what I could have implemented much more easily.

--=-747T55IOzcaOcTQMdE66
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-747T55IOzcaOcTQMdE66--

Cc: ietf-imapext <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Date: Fri, 14 Dec 2007 15:33:30 +0200
To: Alexey Melnikov <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-288678712
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On 14.12.2007, at 6.46, Alexey Melnikov wrote:

> Timo Sirainen wrote:
>
>> On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
>>
>>> It is currently the belief of the chair and the editor that this  
>>> document reasonably addresses all outstanding issues. Consider  
>>> this a 1 week last call for the WG. I'll be out of town until  
>>> Monday, Dec. 17. If you are not heard of by then, this document  
>>> will be written up for IETF Last Call.
>>>
>>
>> Looks pretty good, although I'm not sure if it's such a good idea to
>> have it specify how to handle invalid input / unknown charsets.
>>
> I think we need some degree of predictability here. Sieve defines  
> something similar.

Why? For Sieve I can understand because there a wrong decision could  
in worst case cause a message to be discarded completely.

Unknown charset handling especially isn't predictable between servers  
and clients. For example a client can't reliably update its internal  
SORT view for new messages exactly the same as the server would have  
done it, if they support different charsets.

I'd like it more if both unknown charset and invalid input handling  
was just left up to implementation to handle the best way it knows how.

>> Would be better if this rule applied only at the position where the
>> unknown data was found.
>>
> IMAP doesn't feed partial strings into collations. Describing this  
> would be a nightmare and I am not sure it is worth the effort.

Which is why I'd vote to just replace all of it with being  
"implementation specific"..

I doubt anyone really cares how invalid input is handled, unless they  
see that sorting clearly goes wrong, which is what the current draft  
requires.

--Apple-Mail-1-288678712
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-288678712--

Date: Fri, 14 Dec 2007 14:22:37 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Alexey Melnikov writes:
> IMAP doesn't feed partial strings into collations. Describing this 
> would be a nightmare and I am not sure it is worth the effort.

I strongly agree. Having IMAP-I18N reach through 4790 and specify how 
collations behave would be bad.

Also it's not a big problem. The worst that can happen is that for e.g. 
THREAD ORDEREDSUBJECT, a thread can be sorted in a possibly bad 
position if its subject is encoded in a character set the server 
doesn't support. That is a very mild consequence of an unlikely 
occurence. IMO it's not enough to justify a layering violation.

Arnt

Date: Fri, 14 Dec 2007 04:46:04 +0000
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen wrote:

>On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
>  
>
>>It is currently the belief of the chair and the editor that this 
>>document reasonably addresses all outstanding issues. Consider this a 
>>1 week last call for the WG. I'll be out of town until Monday, Dec. 
>>17. If you are not heard of by then, this document will be written up 
>>for IETF Last Call.
>>    
>>
>
>Looks pretty good, although I'm not sure if it's such a good idea to
>have it specify how to handle invalid input / unknown charsets.
>  
>
I think we need some degree of predictability here. Sieve defines 
something similar.

>Especially for SORT:
>
>    The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate
>    the following together: strings encoded using unknown or invalid
>    character encodings, strings in unrecognized charsets, and invalid
>    input (as defined by the active collation).
>
>Does that mean if I have messages sorted by subject:
>
>abc
>bcd =?unknown-charset?q?1234?=
>cde
>
>These should instead be sorted by:
>
>abc
>cde
>bcd =?unknown-charset?q?1234?=
>  
>
Yes.

>Would be better if this rule applied only at the position where the
>unknown data was found.
>  
>
IMAP doesn't feed partial strings into collations. Describing this would 
be a nightmare and I am not sure it is worth the effort.

Date: Mon, 10 Dec 2007 13:14:02 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen writes:
> Would be better if this rule applied only at the position where the 
> unknown data was found.

I think that's in scope for RFC 4790, not this document. And I don't 
remember what 4790 says, and I haven't the time to look it up now.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Mon, 10 Dec 2007 13:33:35 +0200
Sender: [email-masked]

--=-9GzIi25CgvpgLNpbfW2u
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
> It is currently the belief of the chair and the editor that this=20
> document reasonably addresses all outstanding issues. Consider this a=20
> 1 week last call for the WG. I'll be out of town until Monday, Dec.=20
> 17. If you are not heard of by then, this document will be written up=20
> for IETF Last Call.

Looks pretty good, although I'm not sure if it's such a good idea to
have it specify how to handle invalid input / unknown charsets.
Especially for SORT:

    The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate
    the following together: strings encoded using unknown or invalid
    character encodings, strings in unrecognized charsets, and invalid
    input (as defined by the active collation).

Does that mean if I have messages sorted by subject:

abc
bcd =3D?unknown-charset?q?1234?=3D
cde

These should instead be sorted by:

abc
cde
bcd =3D?unknown-charset?q?1234?=3D

Would be better if this rule applied only at the position where the
unknown data was found.

--=-9GzIi25CgvpgLNpbfW2u
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-9GzIi25CgvpgLNpbfW2u--

Importance: High
Date: Sun, 9 Dec 2007 19:25:07 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

It is currently the belief of the chair and the editor that this 
document reasonably addresses all outstanding issues. Consider this a 
1 week last call for the WG. I'll be out of town until Monday, Dec. 
17. If you are not heard of by then, this document will be written up 
for IETF Last Call.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sat, 08 Dec 2007 22:17:56 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: Updated IMAP I18N document
Sender: [email-masked]

Mark Crispin wrote:

> I am uneasy about having COMPARATOR and UNICASEMAP as separate 
> capabilities, especially since COMPARATOR implies but does not require 
> UNICASEMAP.  This means that a client that only cares about UNICASEMAP 
> capability must test for both requirements.
>
> As an alternative, how about I18NLEVEL=<version #>, where version 1 
> means UNICASEMAP capability and version 2 means COMPARATOR?  This 
> implies that any subsequent additions will be additive with all 
> previous ones being mandatory-to-implement; but I think that is the 
> right thing in this case.

Based on various discussions in Vancouver it seems that people are Ok 
with this change.
If you object, please speak up now.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D Action:draft-ietf-imapext-i18n-14.txt 
Date: Sat, 08 Dec 2007 17:40:02 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title           : Internet Message Access Protocol Internationalization
	Author(s)       : C. Newman, et al.
	Filename        : draft-ietf-imapext-i18n-14.txt
	Pages           : 18
	Date            : 2007-12-08

Internet Message Access Protocol (IMAP) version 4rev1 has basic
 support for non-ASCII characters in mailbox names and search
 substrings.  It also supports non-ASCII message headers and content
 encoded as specified by Multipurpose Internet Mail Extensions
 (MIME).  This specification defines a collection of IMAP extensions
 which improve international support including comparator negotiation
 for search, sort and thread, language negotiation for international
 error text, and translations for namespace prefixes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-14.txt

To remove yourself from the I-D Announcement list, send a message to
[email-masked] with the word unsubscribe in the body of 
the message.
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then
	"get draft-ietf-imapext-i18n-14.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-i18n-14.txt".

NOTE:   The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:     <[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-i18n-14.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-i18n-14.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:     <2007-12-08173840.I-D\@ietf.org>

--OtherAccess--

--NextPart--

Date: Mon, 03 Dec 2007 17:32:00 +0000
From: Alexey Melnikov <[email-masked]>
To: Lemonade WG <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Friends of IMAP dinner meeting time
Sender: [email-masked]

Let's meet at 8pm today near the IETF message board.

Date: Mon, 03 Dec 2007 14:20:01 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Discussion of IMAP I18N in Vancouver
Sender: [email-masked]

We will meet on Thursday, 10am near the IETF message board.

Date: Mon, 03 Dec 2007 10:55:02 -0500
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Cyrus Daboo <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: draft-daboo-imap-annotatemore-11
Sender: [email-masked]

I'm in the process of migrating my older ANNOTATEMORE implementation to 
the new METADATA syntax and I noticed that draft -11 doesn't contain 
ABNF for the LIST-EXTENDED METADATA return option or the METADATA 
extended list data item (response).

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Sat, 01 Dec 2007 15:02:40 -0800
From: Peter Coates <[email-masked]>
Subject: RE: Ad-hoc meeting to discuss IMAP Internationalization
To: "'Alexey Melnikov'" <[email-masked]>, "'IMAP Extensions WG'" <[email-masked]>
Sender: [email-masked]

I'm only going to be there the Wednesday and Thursday.  Either of those
times work for me.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Alexey Melnikov
Sent: December 1, 2007 14:36
To: IMAP Extensions WG
Subject: Ad-hoc meeting to discuss IMAP Internationalization

With permissions of IMAPEXT and Lemonade chairs, I've put IMAP I18N 
document on the Lemonade WG agenda on Thursday, December 6th.
But I was thinking it might be better to spend some time before that, in 
order to make some progress on the document and save some time during 
the Lemonade WG meeting. I would also like to discuss future potential 
work on IMAP internationalization, because I would prefer to get 
draft-ietf-imapext-i18n done pretty much as is soon (within 2 months) 
and defer any further work till another document.

So, is there any interest to meet up during one of the following slots 
in Vancouver:

Wednesday, December 5th, 15:10-16:10
Thursday, December 6th, 09:00-11:30

?

Date: Sat, 01 Dec 2007 22:35:30 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Ad-hoc meeting to discuss IMAP Internationalization
Sender: [email-masked]

With permissions of IMAPEXT and Lemonade chairs, I've put IMAP I18N 
document on the Lemonade WG agenda on Thursday, December 6th.
But I was thinking it might be better to spend some time before that, in 
order to make some progress on the document and save some time during 
the Lemonade WG meeting. I would also like to discuss future potential 
work on IMAP internationalization, because I would prefer to get 
draft-ietf-imapext-i18n done pretty much as is soon (within 2 months) 
and defer any further work till another document.

So, is there any interest to meet up during one of the following slots 
in Vancouver:

Wednesday, December 5th, 15:10-16:10
Thursday, December 6th, 09:00-11:30

?

Date: Fri, 28 Dec 2007 14:13:50 +1100
From: Bron Gondwana <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: Bron Gondwana <[email-masked]>, Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Thu, Dec 27, 2007 at 03:16:07PM -0700, Lyndon Nerenberg wrote:
>
>
> On 2007-Dec-27, at 15:09 , Bron Gondwana wrote:
>
>> You're lucky if you've never seen file corruption due to bitrot on large
>> drive arrays.  We have about 30TB online at the moment and that's
>> growing fairly rapidly.  The research out there suggests we're likely to
>> have a block level corruption every couple of months with the current
>> level of affordable component reliability.
>
> But that's a local hardware issue, and should be handled locally to the 
> host. I.e. have the IMAP server checksum the files when writing to disk, 
> and verify that checksum when reading the data. Doing random checks via the 
> IMAP protocol won't prevent you from offering corrupted data to clients. 
> You have the right solution, but you're applying it at the wrong layer.

Certainly - though once you put replication in the mix then there needs
to be some way to double check things all the way down.  I agree that 
it doesn't need to be exposed all the way out to IMAP.  I just put it
there for convenience.

That said, if the client wants to confirm that it has the same data as
the server, then there needs to be some way for the client to extract
a digest from the server that it can compare to its local copy.  It's
an implementation detail whether this gets calculated on the fly or
is pulled from a cache.

It would be nice to be able to tell the client which digest is already
cached/pre-calculated by the server to save overloading.

Bron.

Date: Fri, 28 Dec 2007 09:35:47 +1100
From: Bron Gondwana <[email-masked]>
To: Ken Murchison <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Fri, Aug 24, 2007 at 04:42:18PM -0400, Ken Murchison wrote:
>
> Arnt Gulbrandsen wrote:
>> Ken Murchison writes:
>>> Arnt Gulbrandsen wrote:
>>>> What does the GUID do?
>>>
>>> Uniquely identifies a message so that if the same message is present in 
>>> more than one mailbox, its contents only gets replicated once.
>> I'm afraid I phrased the question badly. What does the code do when asked 
>> for a GUID?
>
> Pulls the message GUID out of the index file for the mailbox.

This is currently 20 bytes containing the SHA1 of the rfc822 message
bytes, and has been for a while.  It used to be 12 bytes, either
(0x01, hostid, process id, time) or later I added
(0x02, first 11 bytes of md5 digest)

More recently David Carter rewrote it as a 20 bytes value containing
the full sha1, which is not only cryptographically better but it's
complete!  The partial MD5 was a hack to avoid changing the index
file format while still allowing some level of integrity checking.
I knew it was a nasty compromise - but hey, life is built on nasty
compromises.

Bron.

Date: Fri, 28 Dec 2007 09:30:22 +1100
From: Bron Gondwana <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: [email-masked], IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Mon, Aug 27, 2007 at 11:31:20AM +0100, Tony Finch wrote:
> On Thu, 23 Aug 2007, Ken Murchison wrote:
> >
> > I probably wasn't clear in the use case.  Cyrus now has the ability to
> > replicate a mailstore in near realtime.  Several sites (including CMU in the
> > near future) are using this to have redundant mailstores in case of a hardware
> > failure.  Rather than create a new tool or protocol to verify the consistency
> > of the mailstore pairs, extending IMAP with a couple of new FETCH items seems
> > like a reasonable thing to do.
> 
> I should note that the MD5 replica consistency check has been very useful
> in finding lurking bugs in the code, both the original version and across
> versions as the code has evolved. It's also good for discovering bit-
> smashing hardware failures that haven't (yet) affected lower levels of the
> os/driver/raid stack. You can only be sure of data integrity if you test
> it end-to-end.

Thanks :)  My sentiments exactly.

We have layers of integrity checks and scripts watching other scripts to
maintain our replication system.  Even without the replication, it's
good to record the digest of a delivered message into an index file as
it's delivered and be able to compare that to the spooled file itself
at any later time as an integrity check.

Just this week I've been working on an auditlog patch to cyrus:

http://cyrus.brong.fastmail.fm/patches/cyrus-auditlog-2.3.11.diff

This spits out the GUID (sha1) at APPEND time as well as any time a
COPY is performed on the message.  In the same message is the mailbox
name, mailbox unique id (Cyrus internal) and UID.  This is duplicated
on the replica, so by combining those logs you can see that the same
message exists on both ends.

Our backup system fetches index files, and then uses the sha1 to
compare the message pulled from the spool and ensure it's backing
up the message it expects to be backing up.

Integrity checking is the responsibility of every layer, at least
for the semantic things that only that layer understands.

Bron.

Date: Fri, 28 Dec 2007 09:23:09 +1100
From: Bron Gondwana <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, Ken Murchison <[email-masked]>, [email-masked], IMAP Protocol Interest List <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Wed, Aug 22, 2007 at 02:14:14PM -0700, Mark Crispin wrote:
> On Wed, 22 Aug 2007, Lyndon Nerenberg wrote:
>> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
>> representation is, and comparing file sizes is NOT the same as comparing 
>> message content.
>
> I agree.  The c-client library (the beating heart of UW imapd, Pine, etc.) 
> has a concept of an internal size, but that depends upon the mail store and 
> thus is only used by mail store drivers.  It never gets as far as imapd (or 
> other application), much less exported in the protocol.
>
> Let us be clear here; any .FILESIZE value in any server which supports 
> multiple mail store formats could (and WOULD) vary depending upon the 
> mailbox format with no obvious indication to the user as to why.  This, by 
> itself, should be enough cause to reject it.

That's fine with me.  .FILESIZE is really part of an "internal integrity
check" and was only exposed via IMAP because there's no easy way to
compare \Seen flags in Cyrus other than logging in as the user.  Once
you've done that it's easier to have everything else you need to compare
available through the same connection.

>> .MD5 is useful in the general sense as a fast way to compare messages for 
>> equality, although I think it would be better served in that context if 
>> you could MD5 the body content without the 822 headers to be able to 
>> handle Received: differences for the same message delivered to two or more 
>> folders via different [SL]MTP transactions.  It would probably make more 
>> sense to be able to generically MD5 by MIME body part.
>
> IMHO, this functionality belongs as part of CONVERT.  An MD5 checksum is, 
> in effect, a conversion.

So long as it's specified in a way that makes it easy for the server to
cache so that requests for the value are cheap, and it's a digest that
can be calculated at the other end as an integrity check, I'm pretty
happy.  I'd also like all implementation to be using the SAME value so
it's not just unique within a particular implementation but can be used
to track a series of bytes regardless of who is storing them.

>> What's a GUID?
>
> Presumably this is a global UID which uniquely and permanently identifies 
> the message on the server.
>
> I think that IMAP should eventually have a GUID, but it needs careful 
> design and review.  I don't think that we should take the first patch that 
> comes along.

Excellent.  That's what I'm here for.  The intention with the first
patch was purely to have an internal integrity check.  This time I want
everyone else to be doing it as well.  Now that I've found the right
forum I'd like to start work on a proper design, and I'd love input on
it.

>> But overall I guess I have to question why it's necessary to burden the 
>> protocol with this just because somebody wrote some broken code.  I have 
>> done an awful lot of server migrations over the years and I have never yet 
>> come up against the problem this is trying to detect. I realize you intend 
>> this as private data, but once it's in the wild, it will get used, for 
>> better and worse.
>
> I agree with this sentiment.

Definitely.  I'm pretty sure I tagged that Cyrus patch Ken is talking
about "Not a candidate for upstream without further review" when I was
submitting the series of patches it was in.  I've got a few patches
like that floating around.

Thanks,

Bron.

Cc: Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
From: Lyndon Nerenberg <[email-masked]>
To: Bron Gondwana <[email-masked]>
Subject: Re: [Imap-protocol] Private FETCH items
Date: Thu, 27 Dec 2007 15:16:07 -0700
Sender: [email-masked]

On 2007-Dec-27, at 15:09 , Bron Gondwana wrote:

> You're lucky if you've never seen file corruption due to bitrot on  
> large
> drive arrays.  We have about 30TB online at the moment and that's
> growing fairly rapidly.  The research out there suggests we're  
> likely to
> have a block level corruption every couple of months with the current
> level of affordable component reliability.

But that's a local hardware issue, and should be handled locally to  
the host. I.e. have the IMAP server checksum the files when writing to  
disk, and verify that checksum when reading the data. Doing random  
checks via the IMAP protocol won't prevent you from offering corrupted  
data to clients. You have the right solution, but you're applying it  
at the wrong layer.

--lyndon

Date: Fri, 28 Dec 2007 09:16:22 +1100
From: Bron Gondwana <[email-masked]>
To: Dave Cridland <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, [email-masked], [email-masked], Ken Murchison <[email-masked]>, [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

On Wed, Aug 22, 2007 at 10:13:42PM +0100, Dave Cridland wrote:
> Note that I've proposed "CRC" instead of explicitly naming MD5, partly in 
> case people are tempted to use it in clients, and partly to stop the flurry 
> of emails you'll otherwise get telling you that MD5 is insecure.

Two things here.

1) I consider auditability to be very important.  I want to
   be able to cheaply fetch some form of digest (preferably
   cheaply for both the server and the network) that allows
   me to check that the local message is the same.

More on this - it allows detection of both copies and folder
renames to occur without fetching the full content of the
message again.  Even without detecting two different deliveries
with different Received headers, this is a massive win.  Indeed,
it's 90+% of the duplication I would expect to see.  Multiple
deliveries with different received headers is much rarer than
re-filing a message to a different folder (at least in our users'
usage)

2) I'd prefer this to be usable across multiple different server
   implementations.  My current goal (and the reason I'm re-visiting
   this rather than just maintaining yet another patch against Cyrus.
   I already have about 20, what's one more!) is to build an
   intelligent IMAP proxy/synchroniser.  Having to write separate
   logic for every IMAP server out there would suck.

Assuming this makes it as far as a draft, I would be happy to work
on adding support to a few of the most popular open-source servers
and possibly a client or two to bootstrap support.  I think it's
worth the effort.

Bron.

Date: Fri, 28 Dec 2007 09:09:34 +1100
From: Bron Gondwana <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Cc: Ken Murchison <[email-masked]>, IMAP Protocol Interest List <[email-masked]>, [email-masked], [email-masked]
Subject: Re: [Imap-protocol] Private FETCH items
Organization: brong.net
Sender: [email-masked]

Ducking in to follow up on this thread as the author of the
patch Ken is talking about.  I work at FastMail.FM, and have
been doing an awful lot of work patching Cyrus recently.

I have signed up to the imapext mailing list now (should have
before!) - I can't see [email-masked] on the public
lists page, so have not signed up there.  I'd be happy to join
if that's allowed though.

On Wed, Aug 22, 2007 at 12:27:15PM -0700, Lyndon Nerenberg wrote:
> On Aug 22, 2007, at 12:05 PM, Ken Murchison wrote:
>
>> FYI, the difference between RFC822.SIZE and RFC822.FILESIZE is that the 
>> former uses the cached size in the message index file, and the latter 
>> stat()s for the file size at the time of the request.
>
> .FILESIZE is just  asking for trouble.  Nobody cares what the on-disk 
> representation is, and comparing file sizes is NOT the same as comparing 
> message content.

I added these options originally to support our replication integrity
checking script.  We run Cyrus replication, and it had quite a few
bugs at first.  We still run it as an advance warning if something goes
wrong.

I wanted some method to access the underlying file size and digest to
compare, and I agree that FILESIZE is not ideal and shouldn't be
exposed publically.  It's useful not to have to use a separate protocol
or direct filesystem lookup for us though.

We find that with multiple terabytes of disk and non-ZFS filesystems
(ZFS didn't even exist when I started work on this) there are occasional
underlying errors.  There needs to be integrity checking up at the layer
which understands the semantic meaning of the files.

> .MD5 is useful in the general sense as a fast way to compare messages for 
> equality, although I think it would be better served in that context if you 
> could MD5 the body content without the 822 headers to be able to handle 
> Received: differences for the same message delivered to two or more folders 
> via different [SL]MTP transactions.  It would probably make more sense to 
> be able to generically MD5 by MIME body part.

There are two different values we have now, 'GUID' and 'RFC822.SHA1'.
The first is indexed and fast, the second is calculated by reading the
bytes directly from disk.  We check that for 1/1000 messages at the
moment in a standard replication scan.  Enough to find a systematic
issue fairly quickly without bringing our system IO to its knees.

I have recently re-written the GUID part as "DIGEST.SHA1" and added
a CAPABILITY line "DIGEST=SHA1" with a view to making it extensible
to other message digests in future.  Cyrus now stores the GUID as a
SHA1.
> What's a GUID?

Globally Unique ID.

> But overall I guess I have to question why it's necessary to burden the 
> protocol with this just because somebody wrote some broken code.  I have 
> done an awful lot of server migrations over the years and I have never yet 
> come up against the problem this is trying to detect. I realize you intend 
> this as private data, but once it's in the wild, it will get used, for 
> better and worse.

You're lucky if you've never seen file corruption due to bitrot on large
drive arrays.  We have about 30TB online at the moment and that's
growing fairly rapidly.  The research out there suggests we're likely to
have a block level corruption every couple of months with the current
level of affordable component reliability.

Bron ( now to reply to some other messages )

Date: Mon, 24 Dec 2007 00:24:52 +0300
From: Alexey Melnikov <[email-masked]>
To: Pete Resnick <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Pete Resnick wrote:
> On 12/16/07 at 11:45 PM +0000, Alexey Melnikov wrote:
>> but I agree it was not entirely clear and I reworded the text along 
>> the lines of your proposal:
> I'm going to go ahead and write up -14 for IETF Last Call and send it 
> to Lisa. We'll consider Timo's comments and your clarification part of 
> that last call and make the correction there.
This is fine with me.

Date: Thu, 20 Dec 2007 15:19:42 -0600
To: Alexey Melnikov <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Timo Sirainen <[email-masked]>, ietf-imapext <[email-masked]>
Sender: [email-masked]

On 12/16/07 at 11:45 PM +0000, Alexey Melnikov wrote:

>but I agree it was not entirely clear and I reworded the text along 
>the lines of your proposal:

I'm going to go ahead and write up -14 for IETF Last Call and send it 
to Lisa. We'll consider Timo's comments and your clarification part 
of that last call and make the correction there.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sun, 16 Dec 2007 23:45:54 +0000
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen wrote:

>On Sun, 2007-12-16 at 15:30 +0200, Timo Sirainen wrote:
>  
>
>>I just realized it's possible for collations to return "undefined" even
>>if all the input characters are valid.
>>    
>>
>(Continuing to talk to myself..)
>
>Am I correct about this? To me it looks like RFC 4790 defines "valid
>string" as "passes validity test", which may not be as simple as "all
>characters have valid code points".
>  
>
Correct.

>If this is correct, then the i18n draft needs changing, since currently
>it only specifies what to do with charset decoding failures. 4.6.c
>should then be something like:
>
>"All strings are collated using the active comparator. If the comparator
>returns "undefined", the string is moved to a second list. The second
>list is sorted using i;octet and appended after the first sorted list."
>  
>
This was implied by the following text:

  The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate the 
following
  together: strings encoded using unknown or invalid character 
encodings, strings
  in unrecognized charsets, and invalid input (as defined by the active 
collation).

but I agree it was not entirely clear and I reworded the text along the 
lines of your proposal:

    All strings that were successfully converted by step (b) are
    separated from all strings that failed step (b). Strings in
    each group are collated independently.  All strings successfully
    converted by step (b) are then validated by the active
    comparator. Strings that pass validation are collated using the
    active comparator. All strings that either fail step (b) or fail
    the active collation's validity operation are collated (after
    applying step (a)) using the i;octet comparator (see [RFC4790]).
    The resulting sorted list is produced by appending all collated
    "failed" strings after all strings collated using the active
    comparator.

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 15:54:31 +0200
Sender: [email-masked]

--=-BlSSngy9BBzqQkYjbPDT
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 15:30 +0200, Timo Sirainen wrote:
> I just realized it's possible for collations to return "undefined" even
> if all the input characters are valid.

(Continuing to talk to myself..)

Am I correct about this? To me it looks like RFC 4790 defines "valid
string" as "passes validity test", which may not be as simple as "all
characters have valid code points".

If this is correct, then the i18n draft needs changing, since currently
it only specifies what to do with charset decoding failures. 4.6.c
should then be something like:

"All strings are collated using the active comparator. If the comparator
returns "undefined", the string is moved to a second list. The second
list is sorted using i;octet and appended after the first sorted list."

--=-BlSSngy9BBzqQkYjbPDT
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-BlSSngy9BBzqQkYjbPDT--

Date: Sun, 16 Dec 2007 14:51:35 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Timo Sirainen <[email-masked]>
Sender: [email-masked]

Timo Sirainen writes:
> On Sun, 2007-12-16 at 13:18 +0100, Arnt Gulbrandsen wrote:
>>  > Anyway having to specifically track a list of valid and non-valid
>>  > strings and sort them separately would require adding a lot of new
>>  > code, all just to comply with a specification that works worse (from
>>  > user's POV) than what I could have implemented much more easily.
>>
>>  As I see it, you're adding support for all codepoints and all character
>>  sets,
>
> When it gets to collation, yes, but is this really allowed for 
> implementing SORT?

Why not?

>>  so the exception we're discussing no longer applies to your code.
>
> Why would anyone do anything different then?

Beats me.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 15:30:23 +0200
Sender: [email-masked]

--=-3jgGaz40hdiWy2jWA9s9
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I just realized it's possible for collations to return "undefined" even
if all the input characters are valid. So I guess I'll have to redefine
my request a bit.

Change:

4.6.

(b) The decoded text from (a) MUST be converted to the charset expected
by the active comparator.

to something like:

(b) The decoded text from (a) MUST be converted to the charset expected
by the active comparator. The server MAY/SHOULD handle failures (e.g.
the text is in an unknown charset, contains a sequence which is not
valid according in that charset, etc.) by replacing the failed part with
a unicode "replacement character" #FFFD (or equivalent in other
charsets).

--=-3jgGaz40hdiWy2jWA9s9
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-3jgGaz40hdiWy2jWA9s9--

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 14:47:44 +0200
Sender: [email-masked]

--=-oD5TbLhPGjl8mMfeyHnP
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 13:18 +0100, Arnt Gulbrandsen wrote:
> > Anyway having to specifically track a list of valid and non-valid=20
> > strings and sort them separately would require adding a lot of new=20
> > code, all just to comply with a specification that works worse (from=20
> > user's POV) than what I could have implemented much more easily.
>=20
> As I see it, you're adding support for all codepoints and all character=20
> sets,=20

When it gets to collation, yes, but is this really allowed for
implementing SORT?

> so the exception we're discussing no longer applies to your code.

Why would anyone do anything different then? I see only benefits doing
it this way.

--=-oD5TbLhPGjl8mMfeyHnP
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-oD5TbLhPGjl8mMfeyHnP--

Date: Sun, 16 Dec 2007 13:18:57 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen writes:
> On Fri, 2007-12-14 at 04:46 +0000, Alexey Melnikov wrote:
>>  >Would be better if this rule applied only at the position where the
>>  >unknown data was found.
>>  IMAP doesn't feed partial strings into collations. Describing this 
>>  would be a nightmare and I am not sure it is worth the effort.
>
> Since no-one's talking, I'll continue..
>
> The way I was going to implement it was to just replace 
> invalid/unknown input with a special unicode character and have it 
> get sorted in whatever way it normally gets sorted. This would all be 
> done before the string is fed to collation.

I do that sometimes. And sometimes I map to some private use codepoints.

> I'm not yet sure what the special character would be, but IIRC there 
> were some control characters in unicode already that user wasn't 
> allowed to write. Or maybe it could be just some non-coding 
> character, like the highest valid UTF-8 character. That would make it 
> automatically get sorted to the end.

You can use U+FFFD, which means "a character I couldn't represent while 
translating to unicode", or U+0F0000 to U+10FFFD, which are reserved 
for private use.

> Anyway having to specifically track a list of valid and non-valid 
> strings and sort them separately would require adding a lot of new 
> code, all just to comply with a specification that works worse (from 
> user's POV) than what I could have implemented much more easily.

As I see it, you're adding support for all codepoints and all character 
sets, so the exception we're discussing no longer applies to your code.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 14:00:18 +0200
Sender: [email-masked]

--=-FNHi/+1E/hFbVXeRRGp8
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-16 at 13:43 +0200, Timo Sirainen wrote:
> I'm not yet sure what the special character would be, but IIRC there
> were some control characters in unicode already that user wasn't allowed
> to write. Or maybe it could be just some non-coding character, like the
> highest valid UTF-8 character. That would make it automatically get
> sorted to the end.

#fffd I suppose.=20
http://www.fileformat.info/info/unicode/char/fffd/index.htm

--=-FNHi/+1E/hFbVXeRRGp8
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-FNHi/+1E/hFbVXeRRGp8--

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Sun, 16 Dec 2007 13:43:39 +0200
Sender: [email-masked]

--=-747T55IOzcaOcTQMdE66
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2007-12-14 at 04:46 +0000, Alexey Melnikov wrote:
> >Would be better if this rule applied only at the position where the
> >unknown data was found.
> IMAP doesn't feed partial strings into collations. Describing this would=20
> be a nightmare and I am not sure it is worth the effort.

Since no-one's talking, I'll continue..

The way I was going to implement it was to just replace invalid/unknown
input with a special unicode character and have it get sorted in
whatever way it normally gets sorted. This would all be done before the
string is fed to collation.

I'm not yet sure what the special character would be, but IIRC there
were some control characters in unicode already that user wasn't allowed
to write. Or maybe it could be just some non-coding character, like the
highest valid UTF-8 character. That would make it automatically get
sorted to the end.

Anyway having to specifically track a list of valid and non-valid
strings and sort them separately would require adding a lot of new code,
all just to comply with a specification that works worse (from user's
POV) than what I could have implemented much more easily.

--=-747T55IOzcaOcTQMdE66
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-747T55IOzcaOcTQMdE66--

Cc: ietf-imapext <[email-masked]>
From: Timo Sirainen <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Date: Fri, 14 Dec 2007 15:33:30 +0200
To: Alexey Melnikov <[email-masked]>
Sender: [email-masked]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-1-288678712
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On 14.12.2007, at 6.46, Alexey Melnikov wrote:

> Timo Sirainen wrote:
>
>> On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
>>
>>> It is currently the belief of the chair and the editor that this  
>>> document reasonably addresses all outstanding issues. Consider  
>>> this a 1 week last call for the WG. I'll be out of town until  
>>> Monday, Dec. 17. If you are not heard of by then, this document  
>>> will be written up for IETF Last Call.
>>>
>>
>> Looks pretty good, although I'm not sure if it's such a good idea to
>> have it specify how to handle invalid input / unknown charsets.
>>
> I think we need some degree of predictability here. Sieve defines  
> something similar.

Why? For Sieve I can understand because there a wrong decision could  
in worst case cause a message to be discarded completely.

Unknown charset handling especially isn't predictable between servers  
and clients. For example a client can't reliably update its internal  
SORT view for new messages exactly the same as the server would have  
done it, if they support different charsets.

I'd like it more if both unknown charset and invalid input handling  
was just left up to implementation to handle the best way it knows how.

>> Would be better if this rule applied only at the position where the
>> unknown data was found.
>>
> IMAP doesn't feed partial strings into collations. Describing this  
> would be a nightmare and I am not sure it is worth the effort.

Which is why I'd vote to just replace all of it with being  
"implementation specific"..

I doubt anyone really cares how invalid input is handled, unless they  
see that sorting clearly goes wrong, which is what the current draft  
requires.

--Apple-Mail-1-288678712
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

--Apple-Mail-1-288678712--

Date: Fri, 14 Dec 2007 14:22:37 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Cc: Alexey Melnikov <[email-masked]>, Timo Sirainen <[email-masked]>
Sender: [email-masked]

Alexey Melnikov writes:
> IMAP doesn't feed partial strings into collations. Describing this 
> would be a nightmare and I am not sure it is worth the effort.

I strongly agree. Having IMAP-I18N reach through 4790 and specify how 
collations behave would be bad.

Also it's not a big problem. The worst that can happen is that for e.g. 
THREAD ORDEREDSUBJECT, a thread can be sorted in a possibly bad 
position if its subject is encoded in a character set the server 
doesn't support. That is a very mild consequence of an unlikely 
occurence. IMO it's not enough to justify a layering violation.

Arnt

Date: Fri, 14 Dec 2007 04:46:04 +0000
From: Alexey Melnikov <[email-masked]>
To: Timo Sirainen <[email-masked]>
CC: ietf-imapext <[email-masked]>
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen wrote:

>On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
>  
>
>>It is currently the belief of the chair and the editor that this 
>>document reasonably addresses all outstanding issues. Consider this a 
>>1 week last call for the WG. I'll be out of town until Monday, Dec. 
>>17. If you are not heard of by then, this document will be written up 
>>for IETF Last Call.
>>    
>>
>
>Looks pretty good, although I'm not sure if it's such a good idea to
>have it specify how to handle invalid input / unknown charsets.
>  
>
I think we need some degree of predictability here. Sieve defines 
something similar.

>Especially for SORT:
>
>    The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate
>    the following together: strings encoded using unknown or invalid
>    character encodings, strings in unrecognized charsets, and invalid
>    input (as defined by the active collation).
>
>Does that mean if I have messages sorted by subject:
>
>abc
>bcd =?unknown-charset?q?1234?=
>cde
>
>These should instead be sorted by:
>
>abc
>cde
>bcd =?unknown-charset?q?1234?=
>  
>
Yes.

>Would be better if this rule applied only at the position where the
>unknown data was found.
>  
>
IMAP doesn't feed partial strings into collations. Describing this would 
be a nightmare and I am not sure it is worth the effort.

Date: Mon, 10 Dec 2007 13:14:02 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

Timo Sirainen writes:
> Would be better if this rule applied only at the position where the 
> unknown data was found.

I think that's in scope for RFC 4790, not this document. And I don't 
remember what 4790 says, and I haven't the time to look it up now.

Arnt

Subject: Re: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
From: Timo Sirainen <[email-masked]>
To: ietf-imapext <[email-masked]>
Date: Mon, 10 Dec 2007 13:33:35 +0200
Sender: [email-masked]

--=-9GzIi25CgvpgLNpbfW2u
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Sun, 2007-12-09 at 19:25 -0600, Pete Resnick wrote:
> It is currently the belief of the chair and the editor that this=20
> document reasonably addresses all outstanding issues. Consider this a=20
> 1 week last call for the WG. I'll be out of town until Monday, Dec.=20
> 17. If you are not heard of by then, this document will be written up=20
> for IETF Last Call.

Looks pretty good, although I'm not sure if it's such a good idea to
have it specify how to handle invalid input / unknown charsets.
Especially for SORT:

    The ordering operation (e.g., IMAP SORT and THREAD) SHOULD collate
    the following together: strings encoded using unknown or invalid
    character encodings, strings in unrecognized charsets, and invalid
    input (as defined by the active collation).

Does that mean if I have messages sorted by subject:

abc
bcd =3D?unknown-charset?q?1234?=3D
cde

These should instead be sorted by:

abc
cde
bcd =3D?unknown-charset?q?1234?=3D

Would be better if this rule applied only at the position where the
unknown data was found.

--=-9GzIi25CgvpgLNpbfW2u
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-9GzIi25CgvpgLNpbfW2u--

Importance: High
Date: Sun, 9 Dec 2007 19:25:07 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: QUICK WG Last Call: draft-ietf-imapext-i18n-14.txt
Sender: [email-masked]

It is currently the belief of the chair and the editor that this 
document reasonably addresses all outstanding issues. Consider this a 
1 week last call for the WG. I'll be out of town until Monday, Dec. 
17. If you are not heard of by then, this document will be written up 
for IETF Last Call.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sat, 08 Dec 2007 22:17:56 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Re: Updated IMAP I18N document
Sender: [email-masked]

Mark Crispin wrote:

> I am uneasy about having COMPARATOR and UNICASEMAP as separate 
> capabilities, especially since COMPARATOR implies but does not require 
> UNICASEMAP.  This means that a client that only cares about UNICASEMAP 
> capability must test for both requirements.
>
> As an alternative, how about I18NLEVEL=<version #>, where version 1 
> means UNICASEMAP capability and version 2 means COMPARATOR?  This 
> implies that any subsequent additions will be additive with all 
> previous ones being mandatory-to-implement; but I think that is the 
> right thing in this case.

Based on various discussions in Vancouver it seems that people are Ok 
with this change.
If you object, please speak up now.

To: [email-masked]
Cc: [email-masked]
From: [email-masked]
Subject: I-D Action:draft-ietf-imapext-i18n-14.txt 
Date: Sat, 08 Dec 2007 17:40:02 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title           : Internet Message Access Protocol Internationalization
	Author(s)       : C. Newman, et al.
	Filename        : draft-ietf-imapext-i18n-14.txt
	Pages           : 18
	Date            : 2007-12-08

Internet Message Access Protocol (IMAP) version 4rev1 has basic
 support for non-ASCII characters in mailbox names and search
 substrings.  It also supports non-ASCII message headers and content
 encoded as specified by Multipurpose Internet Mail Extensions
 (MIME).  This specification defines a collection of IMAP extensions
 which improve international support including comparator negotiation
 for search, sort and thread, language negotiation for international
 error text, and translations for namespace prefixes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-i18n-14.txt

To remove yourself from the I-D Announcement list, send a message to
[email-masked] with the word unsubscribe in the body of 
the message.
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then
	"get draft-ietf-imapext-i18n-14.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-i18n-14.txt".

NOTE:   The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:     <[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-i18n-14.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-i18n-14.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:     <2007-12-08173840.I-D\@ietf.org>

--OtherAccess--

--NextPart--

Date: Mon, 03 Dec 2007 17:32:00 +0000
From: Alexey Melnikov <[email-masked]>
To: Lemonade WG <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Friends of IMAP dinner meeting time
Sender: [email-masked]

Let's meet at 8pm today near the IETF message board.

Date: Mon, 03 Dec 2007 14:20:01 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Discussion of IMAP I18N in Vancouver
Sender: [email-masked]

We will meet on Thursday, 10am near the IETF message board.

Date: Mon, 03 Dec 2007 10:55:02 -0500
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Cyrus Daboo <[email-masked]>
CC: IMAP Extensions WG <[email-masked]>
Subject: draft-daboo-imap-annotatemore-11
Sender: [email-masked]

I'm in the process of migrating my older ANNOTATEMORE implementation to 
the new METADATA syntax and I noticed that draft -11 doesn't contain 
ABNF for the LIST-EXTENDED METADATA return option or the METADATA 
extended list data item (response).

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Sat, 01 Dec 2007 15:02:40 -0800
From: Peter Coates <[email-masked]>
Subject: RE: Ad-hoc meeting to discuss IMAP Internationalization
To: "'Alexey Melnikov'" <[email-masked]>, "'IMAP Extensions WG'" <[email-masked]>
Sender: [email-masked]

I'm only going to be there the Wednesday and Thursday.  Either of those
times work for me.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Alexey Melnikov
Sent: December 1, 2007 14:36
To: IMAP Extensions WG
Subject: Ad-hoc meeting to discuss IMAP Internationalization

With permissions of IMAPEXT and Lemonade chairs, I've put IMAP I18N 
document on the Lemonade WG agenda on Thursday, December 6th.
But I was thinking it might be better to spend some time before that, in 
order to make some progress on the document and save some time during 
the Lemonade WG meeting. I would also like to discuss future potential 
work on IMAP internationalization, because I would prefer to get 
draft-ietf-imapext-i18n done pretty much as is soon (within 2 months) 
and defer any further work till another document.

So, is there any interest to meet up during one of the following slots 
in Vancouver:

Wednesday, December 5th, 15:10-16:10
Thursday, December 6th, 09:00-11:30

?

Date: Sat, 01 Dec 2007 22:35:30 +0000
From: Alexey Melnikov <[email-masked]>
To: IMAP Extensions WG <[email-masked]>
Subject: Ad-hoc meeting to discuss IMAP Internationalization
Sender: [email-masked]

With permissions of IMAPEXT and Lemonade chairs, I've put IMAP I18N 
document on the Lemonade WG agenda on Thursday, December 6th.
But I was thinking it might be better to spend some time before that, in 
order to make some progress on the document and save some time during 
the Lemonade WG meeting. I would also like to discuss future potential 
work on IMAP internationalization, because I would prefer to get 
draft-ietf-imapext-i18n done pretty much as is soon (within 2 months) 
and defer any further work till another document.

So, is there any interest to meet up during one of the following slots 
in Vancouver:

Wednesday, December 5th, 15:10-16:10
Thursday, December 6th, 09:00-11:30

?