Subject: [lemonade] AD review of draft-ietf-lemonade-convert-16.txt

From: Chris Newman <[email-masked]>
Date: Thu, 27 Mar 2008 12:42:17 -0700

I would like the following issue addressed before IETF last call:

---
Section 6
>   suitable target media type.  This document doesn't describe how the
>   server makes such a choice.  For example, the server can know
>   characteristics of the device through a device description mechanism,
>   or it can have a prioritized list of MIME types based on how
>   widespread they are, how difficult their rendering is, etc.  Note
>   that servers are REQUIRED to support "default conversion" requests.

This mandates support for default conversions without setting any
constraints or even clear expectations for what the default conversion
will do.  While I understand the need for default conversion (and
suspect it may be the common case use for this extension if this deploys
successfully), this gives the server too much room to do
something stupid (such as the identity conversion) and in the process
risks defeating the utility of this function through
non-interoperability.  This is not sufficient to create interoperable
implementations.

What's needed is a clear statement of the expectation for default 
conversions.
The mechanism the server uses can then be measured against the expectation 
to
determine the quality of the server implementation.  But without stating the
expectation, there are too many reasonable expectations and thus no way to
measure the quality of a server implementation.  Here are some examples of
expectations you could set:

* Reduce the size of converted parts to a minimum that remains useful for 
fast
  preview purposes

* Convert the media types, charsets, codecs and other customizable 
parameters to
  the most standard or widely deployed options available in that media 
category.
  Minimize quality loss in this conversion.

* Convert the media types, charsets, codecs and other customizable 
parameters to
  ones that maximize the rendering speed of the media on a display device 
appropriate
  for that media type.

* Convert the media types, charsets, codecs and other customizable 
parameters to
  ones that can be rendered by a device without the need for any patent 
licenses.

* Based on out-of-band information about the client device, convert the 
media to
  types the device is likely to support and display successfully while 
removing
  any unnecessary detail that exceeds the capabilities of the device (e.g. 
scaling
  images to just fit on the device's screen).  When out-of-band device 
information
  is unavailable, assume a default device with the following 
characteristics: ...
  The characteristics of the default device may be adjusted by server 
configuration.

I'd also feel more comfortable if there was an optional way for the client
to inform the server of its identity (e.g., RFC 2971 with the constraint
about server interpretation lifted for this one purpose and perhaps a
device-id tag added).
---

The following issues fall into the "last call comment" category and if you
want to enter IETF last call without addressing these, that's fine.  But I
recommend addressing some of these with a revision prior to IETF last call
to reduce last call controversy.

High-level comments:

* As a matter of technical architecture, I would prefer that we had a
generic data conversions service and hooked it into IMAP via URLAUTH,
rather than making IMAP itself more complex.  If the IESG is doing their
job correctly, this should be raised as an issue during IESG review and
I will need a response to provide the IESG in the event this question
comes up.

* The IETF has a standards track protocol (OPES, RFC 4037) designed so
that servers can call-out for conversions.  While there is no requirement
to implement IMAP Convert using OPES, it is one valid implementation
technique and an informative reference would be nice.  Further, if IMAP
convert is implemented via a protocol call-out mechanism, I would like
implementer feedback on whether OPES is a good fit for that usage.

* The Lemonade charter mandates that the CONNEG framework will used if
content negotiation is needed.  IMAP CONVERT performs content
negotiation using a different negotiation framework, but it leverages
the media-features registry from CONNEG.  After consultation with the
former area director responsible for the lemonade charter is my
understanding that the primary purpose of that line in the charter was
to prevent creation of a new media feature vocabulary.  As a result, my
judgement is that the document just barely passes the bar set by the
charter.  However, I am open to input on whether my interpretation of
the charter is correct.

Detailed comments:

Section 3
>   The CONVERT extension does not address conversion during streaming
>   of attachments.

This is an odd statement because there's no context to understand it
within the document (there is within the WG charter, but we can't assume
readers will have read the charter). If you're going to say this, say
why and provide an informative reference to the document or documents
that can provide this functionality (for example, a combination of IMAP
URLAUTH and extensions for RTSP session negotiation could provide
streaming functionality).

Section 5.1

The document does not state which IMAP states (Authenticated, Selected)
the conversions command is made available.  At a minimum, I see no reason
it should be available in unauthenticated state and it would be best from
a security perspective if it was not permitted during that state.

Also, the document does not include a section for the "CONVERSIONS"
untagged response as is traditional in IMAP documents.  Was this omission
intentional?

Section 6
I find the name "BODYPARTSTRUCTURE" misleading as the result does not in
fact reflect the structure of the body part.  I suggest
"CONVERTEDSTRUCTURE" as an alternative.

>   The UID CONVERT command is different from the CONVERT command in the
>   same way as the UID FETCH command is different from the FETCH
>   command:

Note RFC 3501 section 7.4.1 describes another difference between FETCH
and UID FETCH that applies to CONVERT and UID CONVERT.

>   Comments embedded like this SHOULD be preserved during
>   conversion, but MAY be removed entirely.

This "SHOULD" and "MAY" are contradictory.  I suggest re-wording.

Section 8.2:
>   No destination MIME type may be specified with BODY[HEADER],
>   BODY[section.HEADER], or BODY[section.MIME].  It is important the
>   encoded words be left as encoded words as to do otherwise may alter
>   the semantics of structured headers.

Has there been a discussion of support for message/global-headers as a
target media type here?  That might actually be an interesting
poor-man's approach to EAI compatibility.  I'm not saying it's something
that has to be done, but cross-WG issues do need to be evaluated during
IESG review so I'd like some discussion of this topic before IESG review.

>   returned for the exactly the same conversion in the same IMAP
                 XXX

>   A response to a CONVERT command containing the BODYPARTSTRUCTURE data
>   item MUST return a BODYPARTSTRUCTURE data item (in the CONVERTED
>   response) before any other data items for the same body part.

This is an odd requirement and to me violates the principle of least
astonishment.  I believe it's also the first time we've required an IMAP
server to re-order the response data items. I'd be a lot more
comfortable with a requirement that in the event BODYPARTSTRUCTURE is
listed prior to BINARY for the same part in the request that it MUST
also be listed first in the response.  That achieves the same goal
(clients can be sure to get it first if they want it first) without
mandating the server re-order response data items.

Section 8.3
>         request). The return value MUST be exact and MUST NOT change
>         during a duration of an IMAP "session". This allows a client

I'm not sure such a strict "MUST NOT change" is a good idea.  One case
where it may change is if the message is expunged from underneath the
client.  I think clients need the ability to handle a change to an error
(or 0/NIL) in the event the part is removed and saying so (preferably
with an example) is likely helpful to interoperability.

>   Note for client implementors: client authors are cautioned that this
>   might be an expensive operation for some server implementations.
>   Needlessly issuing this request could result in degraded performance
>   due to servers having to calculate the value every time the request
>   is issued.  For that reason it is not a good idea to needlessly
>   request BINARY.SIZE and use it in a UI to allow a user to chose
>   between multiple conversions.

While advice along these lines is needed, this seems a bit off to me as
it places constraints on UIs which we have no business making and
needlessly makes a value judgement about client implementation models.
Perhaps something more like this would be better:

   Note for client implementors: client authors are cautioned that this
   might be an expensive operation for some server implementations.
   Requesting BINARY.SIZE for a large number of converted body parts or
   for multiple conversions of the same part can result in slow
   performance and/or excessive server load and is discouraged.  Clients
   should consider implementation approaches that limit this request to
   only the most necessary cases and are encouraged to test the
   performance impact of BINARY.SIZE with multiple server implementations.

Section 8.4: AVAILABLECONVERSIONS

Unlike the "CONVERSIONS" command, this response item provides no way to
determine support for features beyond the media type itself.  I want to
verify that this is acceptable to the WG as it seems an unusual
restriction to me.  I also find myself somewhat dubious of the utility
of this feature.

Section 10
>          convert-err-descript = string
>               ; Human readable text explaining the conversion error

Please state the charset for this, and if its value may be modified by
the LANGUAGE command.

Section 12
1st paragraph -- another mitigation is for the client to authenticate
the server using a mutual authentication mechanism and integrity layer
and trust the server not to provide dangerous content.

3rd paragraph -- an appropriate mitigation is to log the client
authentication identity for append and/or convert operations so
administrative measures can be applied to the situation.

4th paragraph -- I would recommend deleting this.  Convert doesn't alter
the nature of the spam threat in any way.  If you want to note the spam
threat, include a reference to one or more of the BCPs on the topic.

		- Chris

_______________________________________________
lemonade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Neil Cook <[email-masked]>
Date: Fri, 28 Mar 2008 16:16:32 +0000

Chris,

with regard to your comments on streaming:

> streaming functionality).

The lemonade streaming draft does discuss conversion/transcoding  
during streaming of attachments. So that could be referenced here.

Neil

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 31 Mar 2008 18:21:00 +0100

Hi Chris,
Thank you for your detailed review. I mostly agree with the issues you 
raise.

I will answer your detailed comments in the email message, as they are 
easier to deal with. Stay tuned for a separate response on more generic 
issues.

>streaming functionality).
I've added an informative reference to 
draft-ietf-lemonade-streaming-04.txt and reworded the sentence slightly:

    Conversion of attachments during streaming is out of scope for the 
CONVERT
    extension and is described in a separate Lemonade WG document
    [lemonade-streaming].

>a security perspective if it was not permitted during that state.
Good point. I've added:
    The CONVERSIONS command is allowed in Authenticated and Selected 
IMAP states.

>untagged response as is traditional in IMAP documents.  Was this omission
No. While the source of the document was in plain text, I didn't want to 
add a new section, as it would have been difficult to renumber sections. 
That is much easier with XML, so I will add this.

>"CONVERTEDSTRUCTURE" as an alternative.
I can do search and replace in the document, but I need other people to 
provide feedback on this.

>and UID FETCH that applies to CONVERT and UID CONVERT.
Good catch. I've added:
    An EXPUNGE response MUST NOT be sent while responding to a CONVERT
    command.  This rule is necessary to prevent a loss of
    synchronization of message sequence numbers between client and
    server. Note that an EXPUNGE response MAY be sent during a UID 
CONVERT command.

>This "SHOULD" and "MAY" are contradictory.  I suggest re-wording.
The intend was to say that comments can be preserved at the place where 
they were found, they can be preserved, but moved, or they can be removed.
I thought this sentence was sufficiently clear, any suggestion how to 
improve it?

>target media type here?
Right. The CONVERT extension would support that, if we allow for target 
MIME type above. I am frankly scared of putting any text about 
message/global-headers into the document, considering that all EAI 
documents are going to be Experimental at first. But once EAI updates 
all documents to be on standard track, that would be an obvious thing to 
look at.

I was thinking about suggesting that the output for BODY[HEADER] should 
be headers in UTF-8 (with no encoded words), i.e. effectively they would 
be in message/global-headers format. But Peter convinced me that all 
existing email clients would need to support encoded words, so it is 
better to leave the converted headers with encoded UTF-8 words as well.

>That might actually be an interesting
I've deleted the first "the", if that is what you were referring to.

>server to re-order the response data items.
>
Correct.

>I'd be a lot more comfortable with a requirement that in the event BODYPARTSTRUCTURE is
>also be listed first in the response.
>
> That achieves the same goal
Clients can't have this guaranty with IMAP anyway, because pretty much 
all existing servers reorder [at least some] of FETCH data items.

This requirement is for clients benefit. I.e. the client can expect it 
to be the first data item (the second, if UID is also returned), so it 
can take advantage of this.
I wish we had something similar about UID data item in FETCH.

>client.
It was never my intent for the MUST NOT to prevent a message from being 
expunged. What about the following text clarifying this:

    If the message is expunged in another session, then the server
    MAY return the value 0 in response to the BINARY.SIZE request item
    later in the same session.
?

>I think clients need the ability to handle a change to an error
>
Ok, I think your text is better, so I've updated the document.

>determine support for features beyond the media type itself.
>
This was intentional. If the client wants to use conversion parameters, 
it is either going to use a fixed set of parameters, or it can discover 
which ones are available using the CONVERSIONS command. I didn't think 
it was worth adding extra complexity and increase response size for 
something which is unlikely to be used by the client.

>I want to verify that this is acceptable to the WG as it seems an unusual
The use case is as follows: the CONVERSIONS command for the source MIME 
type would return too many choices (for example for 
application/octet-stream). But for a specific body part conversion there 
is a smaller set of target MIME types, which can be presented to the 
user in a dialog, which allows the user to pick one of the supported 
MIME type viewers. Something like:

This body part can be converted to one of the following MIME types, 
please select your preferred viewer:
o image/jpeg (default viewer)
o application/pdf (Adobe Acrobat)
o text/plain (Notepad)

>Please state the charset for this, and if its value may be modified by
I've clarified that the default is US-ASCII, unless the LANGUAGE command 
is executed:

    ; The default charset is US-ASCII, unless
    ; the LANGUAGE command [IMAP-I18N] is called, when
    ; the charset changes to UTF-8.

>and trust the server not to provide dangerous content.
Right. I've added:
    Clients SHOULD use mutual SASL authentication and SASL/TLS integrity 
layer, to make sure
    they are talking to trusted servers.

>administrative measures can be applied to the situation.
Ok, I've added:

    In order to mitigate such attacks, servers SHOULD log the client
    authentication identity on APPEND and/or CONVERT operations in order
    to facilitate tracking of abusive clients.

SHOULD because many servers allow for protocol trace logging (by 
userid), which effectively does the same thing.

>the nature of the spam threat in any way.
>
Agreed.

>If you want to note the spam

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 31 Mar 2008 23:06:55 +0100

Hi Chris,

>non-interoperability.  This is not sufficient to create interoperable
I agree that default conversion is underspecified. The text was trying 
to hint what is acceptable, but it wasn't good enough.

>to determine the quality of the server implementation.  But without stating the
Some quick comments about your suggestions:

>fast preview purposes
While this is a worthy goal, this is can be controlled by a conversion 
parameter (e.g. max document size). So default conversion is not needed.

>parameters to the most standard or widely deployed options available in that media 
>category. Minimize quality loss in this conversion.
>
This is the goal, if device type is not known by the server.

>parameters to ones that maximize the rendering speed of the media on a display device 
>appropriate for that media type.
>
Not the goal (again, can be controlled by conversion parameters).

>parameters to ones that can be rendered by a device without the need for any patent 
>licenses.
>
Not the goal, this can be specified by conversion parameters and a 
specific target MIME type.

>media to types the device is likely to support and display successfully while 
>removing any unnecessary detail that exceeds the capabilities of the device (e.g. 
>scaling images to just fit on the device's screen).
>
This is the goal if the device type is known to the server.

>When out-of-band device 
>information is unavailable, assume a default device with the following 
>characteristics: ...
Combining my comments together, here is the updated description of 
default conversion:

    Instead of specifying the exact target MIME media type the client
    wants to convert to, the client MAY use a special marker NIL (also
    known as "default conversion") to request the server to pick a
    suitable target media type. This document doesn't describe how exactly
    the server makes such a choice, however some basic guidelines are
    described in this paragraph. If the server knows characteristics
    of the device using an in-band (such as device id specified
    in a conversion parameter) or an out-of-band mechanism, then
    it should convert the request body part to a media type the device
    is likely to support and display/play successfully. Unless specifically
    overridden by a conversion parameter, the server MAY also remove
    any unnecessary detail that exceeds the capabilities of the device
    (e.g. scaling images to just fit on the device's screen).
    In the absence of any in-band or out-of-band mechanism for
    determining device characteristics, the server should convert
    the request body part to the most standard or widely deployed
    media type available in that media category. In such case
    the server should minimize quality loss.
    Servers are REQUIRED to support "default conversion" requests.

Does this address your concern?

>about server interpretation lifted for this one purpose and perhaps a
Zoltan had a draft describing device ID. Once CONVERT is published, the 
device-id conversion parameter should be standardized.
I would feel uncomfortable with delaying publication of the CONVERT spec 
until this parameter is standardized.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 01 Apr 2008 00:25:30 +0100

Hi Chris,

>I will need a response to provide the IESG in the event this question
You and I discussed this briefly in jabber. Let me summarize our discussion:
1). An IMAP extension that provides body part conversion with support 
for partial download is easy for IMAP clients to implement.
Many IMAP clients already support downloading body parts in chunks 
(using partial FETCH), CONVERT provide a natural extension to that.

2). Based on feedback from client developers, the CONVERT extension 
supports BODYPARTSTRUCTURE (MIME structure of the converted body part) 
and BINARY.SIZE (size of the converted body part) CONVERT requests. Both 
as similar to functionality provided by IMAP FETCH command.
Any alternative solution has to provide the same functionality.

3). Mobile clients would be happier with having fewer TCP connections

And finally:

4). Lemonade WG is only chartered to extend SMTP and/or IMAP, so a 
separate protocol is out of scope for the WG.

>implementer feedback on whether OPES is a good fit for that usage.
I've skimmed through RFC 4037 and I think it can be a reasonable 
protocol for implementing conversions. It does have some tricky commands 
(like various optimizations) and might be overkill for some implementations.

I am not entirely sure where to add the informative reference to RFC 4037.

>negotiation using a different negotiation framework,
>
My understanding is that the content negotiation language defined in RFC 
2533 is much more complex, rarely used and thus would be overkill for 
CONVERT.

>but it leverages the media-features registry from CONNEG.  After consultation with the
>to prevent creation of a new media feature vocabulary.
>
I tend to agree with the former AD.

>As a result, my judgement is that the document just barely passes the bar set by the

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 01 Apr 2008 10:38:05 +0100

Alexey Melnikov wrote:

> No. While the source of the document was in plain text, I didn't want 
> to add a new section, as it would have been difficult to renumber 
> sections. That is much easier with XML, so I will add this. 

I've fixed that now.

> all existing servers reorder [at least some] of FETCH data items.

To clarify my earlier comment: many existing IMAP servers use bitmaps 
for requested FETCH items such as INTERNALDATE, UID and FLAGS. No matter 
what was their order in a FETCH command, any such server always return 
them in the fixed order in the corresponding FETCH responses.