Subject: Re: [Gen-art] [EAI] Gen-ART review of draft-ietf-eai-imap-utf8-07

From: John C Klensin <[email-masked]>
Date: Tue, 01 Sep 2009 14:26:34 -0400

David,

A comment on one issue -- speaking for myself, not the WG...

--On Monday, August 31, 2009 03:03 -0400 [email-masked]
wrote:

> I have been selected as the General Area Review Team (Gen-ART) 
> reviewer for this draft (for background on Gen-ART, please see 
> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html). 
>...
 
> Major issues:
>...
> Section 8 lists a number of 8859 character sets for which
> upconversion of MIME headers MUST be supported, and then says
> "Other widely deployed MIME charsets SHOULD be supported."
> How does an implementer figure out which character sets those
> would be?  As an alternative, I suggest saying something along
> the lines of: any server-supported character set that is a
> superset of ASCII should be supported for upconversion. That
> probably leads to fewer client surprises caused by UTF-8 not
> working as expected.

FWIW, "superset of ASCII" turns out to be a nightmare
definition.  There are languages out there that use Latin-based
scripts but don't use all ASCII characters.  While I'm not aware
of any of them that use coded character sets that don't include
all of the undecorated "Latin" characters that appear in ASCII,
that combination is certainly plausible and probably does occur
somewhere.  More important, the issues with local character
coding systems are far more severe with non-Latin-based writing
systems than they are with Latin-based ones, partially because
there are different models for coding (precomposed characters
versus combining ones, format and style effectors that may
actually change the characters versus greater numbers of
precomposed forms, etc.), rendering conversion to or from
Unicode a context-dependent translation activity rather than a
character-to-character mapping one.

On the other hand, if the specification says "this needs to be
Unicode, presented in UTF-8, on the wire" then servers pretty
much know what they use (or have to deliver to mail-reading
MUAs) locally and have to support and convert to and from,
without our doing anything more than reminding them that, if
they don't do those conversions, things won't work at all well.

So, while I agree with you that the text doesn't sound
sufficiently specific, I fear that the reality is that trying to
actually make it more specific will set us up for either
requiring support for conversions that are completely
unnecessary in context or for leaving out ones that are
important.  Indeed, I suspect that the listing of specific 8859
standards will turn out to be a bad idea in the long term, but
that is something the WG decided on and, for an Experimental
document, I think it is reasonable to leave it there and examine
operational experience when that is relevant.

best,
    john

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 01 Sep 2009 22:36:23 +0100

Hi  David,
Thank you for your detailed review.

[email-masked] wrote:

>Please resolve these comments along with any other Last Call
>comments you may receive.
>
>Document: draft-ietf-eai-imap-utf8-07
>Reviewer: David L. Black
>Review Date: August 31, 2009
>IETF LC End Date: August 31, 2009
>IESG Telechat Date: September 10, 2009
> 
>Summary:
>
>This draft is on the right track, but has open issues, described
>in the review.
>
>The draft appears to be in good shape, although one has to be an
>IMAP expert to understand all of its implications (and I am not
>an IMAP expert).  I found two open issues:
>
>- The header upconversion behavior specification for non-UTF-8
>	mailstores appears to be incomplete.
>- The recommendation to support MIME header upconversion for
>	"Other widely deployed MIME charsets" strikes me as
>	too vague to be useful guidance to implementers.
>
>Major issues:
>
>Section 3.2, upconversion behavior specification appears to be
>incomplete:
>
>   If the mailstore is not UTF-8
>   header native and the SELECT or EXAMINE command with UTF-8 header
>   modifier succeeds, then the server MUST return results as if the
>   mailstore was UTF-8 header native with upconversion requirements as
>   described in Section 8.  
>
>What happens if a header that is not upconverted is accessed with
>a UTF-8 comparison string (e.g., by SEARCH)?  I presume that no
>matches occur courtesy of the charset mismatch, but that needs to
>be explained, as it will be a surprise to users.
>
Actually no, I believe this is already fully specified in RFC 3501 (base 
IMAP spec), Section 6.4.4:

      The OPTIONAL [CHARSET] specification consists of the word
      "CHARSET" followed by a registered [CHARSET].  It indicates the
      [CHARSET] of the strings that appear in the search criteria.
      [MIME-IMB] content transfer encodings, and [MIME-HDRS] strings in
      [RFC-2822]/[MIME-IMB] headers, MUST be decoded before comparing
      text in a [CHARSET] other than US-ASCII.  US-ASCII MUST be
      supported; other [CHARSET]s MAY be supported.

So any of the recognized charsets will be decoded by the server and the 
right thing would happen.

>Section 8 lists a number of 8859 character sets for which upconversion
>of MIME headers MUST be supported, and then says "Other widely deployed
>MIME charsets SHOULD be supported."  How does an implementer figure
>out which character sets those would be?  As an alternative, I suggest
>saying something along the lines of: any server-supported character
>set that is a superset of ASCII should be supported for upconversion.
>That probably leads to fewer client surprises caused by UTF-8 not
>working as expected.
>  
>
I believe John Klensin has adequately responded to this point.

>Minor issues:
>
>Section 3.1, next to last paragraph needs a couple of RFC 2119
>keywords:
>
>   Mailbox
>   names must comply with the Net-Unicode Definition (section 2 of
>MUST >-->^^^^
>
>   [RFC5198]) with the specific exception that they may not contain
>MUST NOT >----------------------------------------->^^^^^^^
>
>   control characters (0000-001F, 0080-009F), delete (007F), line
>   separator (2028) or paragraph separator (2029).
>
Agreed. I've entered these as RFC Editor notes.

>The ABNF in this draft is extensions to ABNF specified elsewhere.
>I hope that the combined ABNF grammars have been run through an
>ABNF checker, but didn't see any mention of that in the IESG
>comment log.
>
Yes, both Harald Alvestrand and myself checked the ABNF.

>This would normally be covered by a shepherd's
>report, but I did not see one.
>
This is coming.

>Section 7 recommends that all IMAP clients be modified to display a
>clear error when the server advertises UTF8=ONLY.  What's the
>expected behavior of existing, unmodified clients?
>  
>
Such clients will not be using the UTF8 parameter to SELECT/EXAMINE 
(mailbox opening) commands, so they will fail to do anything useful. But 
this is to be expected.

>Nits/editorial comments:
>
>Section 2 ought to introduce what's being added to the protocol.
>Adaptations of the first two sentences in Section 10 (IANA
>Considerations) would suffice.
>
>While not strictly a security consideration, it would be useful for
>section 11 to point out the potential for user confusion caused by
>SEARCH command match strings that have different UTF-8 representations
>but display identically or similarly (strings that look like they
>should match don't).
>  
>
I will let editors respond/address these 2 comments.

> 
>------------------------------------------------------------------------
>----
>
>  ** There is 1 instance of too long lines in the document, the longest
>one
>     being 14 characters in excess of 72.
>
>
>  Miscellaneous warnings:
> 
>------------------------------------------------------------------------
>----
>
>  == The document seems to lack the recommended RFC 2119 boilerplate,
>even if
>     it appears to use RFC 2119 keywords -- however, there's a paragraph
>with
>     a matching beginning. Boilerplate error?
>
>     (The document does seem to have the reference to RFC 2119 which the
>     ID-Checklist requires).
>
>  Checking references for intended status: Experimental
> 
>------------------------------------------------------------------------
>----
>
>idnits 2.11.12 found a few things (I've deleted a couple of
>obviously incorrect "Missing Reference:" warnings):
>
>  Checking nits according to http://www.ietf.org/ID-Checklist.html:  == Unused Reference: 'RFC2045' is defined on line 475, but no explicit
>     reference was found in the text
>  
>
Not sure if this one is needed or not.

>  == Unused Reference: 'RFC2183' is defined on line 486, but no explicit
>     reference was found in the text
>  
>
I've entered an RFC Editor note that updates the document to reference 
this RFC.

>  ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)  
>
According to authors this reference is intentional.

---

From: <[email-masked]>
Date: Wed, 2 Sep 2009 12:46:04 -0400

John,

I freely admit to not being a character set expert.  The core
question from the review was:

> > would be? 

My attempt to improve on this seems to have missed the mark:

> FWIW, "superset of ASCII" turns out to be a nightmare definition.

I won't disagree with that, despite the number of 8859 character
sets that are listed in the draft.

Rather, what would you suggest to address the original concern? 

> they don't do those conversions, things won't work at all well.

Are you suggesting that the "SHOULD" for upconversion to UTF-8
ought to be on all character sets supported by the server, so
that a client can just deal with UTF-8?

Thanks,
--David

> -----Original Message-----
> From: John C Klensin [mailto:[email-masked]] 
> Sent: Tuesday, September 01, 2009 2:27 PM
> To: Black, David; [email-masked]; 
> [email-masked]; [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: Re: [EAI] Gen-ART review of draft-ietf-eai-imap-utf8-07
> 
> David,
>

---

From: Pete Resnick <[email-masked]>
Date: Thu, 3 Sep 2009 15:28:06 -0500

>>	too vague to be useful guidance to implementers.

I'll leave these as they are without further instruction.

>Agreed. I've entered these as RFC Editor notes.

Fixed.

>(mailbox opening) commands, so they will fail to do anything useful. 
>But this is to be expected.

No change.

>>Considerations) would suffice.

Done.

>>but display identically or similarly (strings that look like they

(*mumble*) This seems to me a job for RFC 3629, not this document. 
(And, BTW, it does so.)

>>  ** There is 1 instance of too long lines in the document

I'll find that.

>>  == The document seems to lack the recommended RFC 2119 boilerplate

Huh? It's in there. Or is this document so old that it doesn't use 
the precise recommended wording? I'll leave that for the RFC Editor.

>>  Checking nits according to http://www.ietf.org/ID-Checklist.html: 
>>== Unused Reference: 'RFC2045' is defined on line 475, but no 
>>explicit

Added a reference.

>I've entered an RFC Editor note that updates the document to 
>reference this RFC.

Added.

>According to authors this reference is intentional.

It is.

Thanks David (and Alexey).

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Joseph Yee <[email-masked]>
Date: Thu, 3 Sep 2009 18:10:14 -0400

On 3-Sep-09, at 4:28 PM, Pete Resnick wrote:

[snip]

> (*mumble*) This seems to me a job for RFC 3629, not this document.  

Agreed with Pete, the security concerns are user name and password.   
SEARCH is more of usability than security.  Maybe mention it in design  
rational instead? (Just suggesting, not saying it MUST or SHOULD)

Regards,
Joseph

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: <[email-masked]>
Date: Fri, 4 Sep 2009 02:14:48 -0400

Alexey,

...
> Actually no, I believe this is already fully specified in RFC 
> 3501 (base IMAP spec), Section 6.4.4:

I concur - this is not an issue.  Thanks for the pointer to the
relevant portion of RFC 3501.

...
> I believe John Klensin has adequately responded to this point.

John has thoroughly demolished the change I suggested, but I
don't think that demolition resolves this concern, as the original
text remains vague.

OTOH, this draft is intended to be an Experimental RFC - would
it be reasonable to say that determining which charsets should
be upconverted is part of the experiment?

> >but display identically or similarly (strings that look like they

That was a nit - leaving the Section 11 text unmodified is fine,
as Section 11 already references RFC 3629.

Thanks,
--David

> From: Alexey Melnikov [mailto:[email-masked]] 
> Sent: Tuesday, September 01, 2009 5:36 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]; 
> [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Gen-ART review of draft-ietf-eai-imap-utf8-07
> 
> Hi  David,
> upconversion
> >  Checking nits according to 
> http://www.ietf.org/ID-Checklist.html:  == Unused Reference: 
> 'RFC2045' is defined on line 475, but no explicit
> reference

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 06 Sep 2009 17:24:57 +0100

Hi David,

>be upconverted is part of the experiment?
I am quite skeptical that we can come to a consensus after running such 
experiment. The answer to the question is going to be very 
region/country specific.
This is not the first time we are discussing the rathole (the same 
applies to MIME, IMAP SEARCH, IMAP CONVERT extension) and this looks 
like a big rathole to me.

I think you mentioned earlier that maybe an implementation should 
support conversion to UTF-8 of all charsets it already supports in 
SEARCH. After thinking more about this I think this proposal is quite 
sensible.

Alternatively, the sentence saying:

  Other widely deployed MIME charsets SHOULD be supported

should just be removed, as I tend to agree that it doesn't provide much 
useful information.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 06 Sep 2009 20:07:22 +0200

Alexey Melnikov skrev:
> I am quite skeptical that we can come to a consensus after running 
> such experiment. The answer to the question is going to be very 
> region/country specific.
It makes quite a lot of sense to me that if an IMAP server supports a 
specific charset in some other context (such as IMAP SEARCH and 
friends), it should be consistent about supporting the same charset in 
UTF8 too.

Inconsistency here would be unreasonable for the user - and by pointing 
to that, we have "reduced to a previously unsolved problem".

Suggested language:

If the server supports other charsets in IMAP SEARCH or IMAP CONVERT 
(add references to taste), it SHOULD also support those charsets in this 
conversion.

              Harald

---

From: John C Klensin <[email-masked]>
Date: Sun, 06 Sep 2009 23:19:01 -0400

--On Sunday, September 06, 2009 8:07 PM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

> It makes quite a lot of sense to me that if an IMAP server
> supports a specific charset in some other context (such as
> IMAP SEARCH and friends), it should be consistent about
> supporting the same charset in UTF8 too.
>
> Inconsistency here would be unreasonable for the user - and by
> pointing to that, we have "reduced to a previously unsolved
> problem".
>
> Suggested language:
>
> If the server supports other charsets in IMAP SEARCH or IMAP
> CONVERT (add references to taste), it SHOULD also support
> those charsets in this conversion.

Wfm
    john

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 07 Sep 2009 10:59:59 +0100

John C Klensin wrote:

> --On Sunday, September 06, 2009 8:07 PM +0200 Harald Tveit Alvestrand 
> <[email-masked]> wrote:
>

Ok, I've added this as an RFC Editor note.

---

From: <[email-masked]>
Date: Mon, 7 Sep 2009 22:19:08 -0400

> Ok, I've added this as an RFC Editor note.

Also works for me.  Thanks for working through the details,
--David