Subject: [lemonade] strawman suggestion for convert

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 16 Jan 2008 14:35:27 +0100

All the suggestions I've seen center around providing enough flexibility 
to cater to whatever is needed. How about having too little flexibility 
instead? Solving the important 80%?

I've seen basically three uses cases:

1. Convert a bodypart of some sort to a JPEG formatted for the device's 
screen. (Sometimes this is a sequence of JPEGs, e.g. in the case of a 
multipage PDF.) The bodypart may be an image, a PDF or even an 
animation. Often it has MIME type application/octet-stream.

2. Convert a bodypart of some sort to to an MP3, chopping it off to 
limited length.

3. Convert a video or audio bodypart to flash in a given resolution and 
stream it.

Accordingly my strawman has three parts:

3. Disregard case 3 for CONVERT. We don't do streaming inside IMAP. We 
could "handle" it by mentioning that a streaming server can use 
URLAUTH/URLFETCH to obtain the bodypart.

1. Capability CONVERT=JPEG gives the ability to do 1. The client 
specifies JPEG, x resolution, y resolution, whether to make the result 
smaller or bigger if the aspect ratio is different, and a maximum 
number of images. The server converts to JPEG if it can, resizes, 
preserving aspect ratio, and sends the results.

2. Capability CONVERT=MP3 gives the ability to do 2. The client 
specifies a bitrate and length. (Or OGG, perhaps. MP3 is widely 
implemented, OGG is free of patents and there's good free code 
available.)

Arnt

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 16 Jan 2008 14:48:47 +0100

> I've seen basically three uses cases:

(Actually I've seen more, but these are the ones I take seriously. I 
don't care about converting HTML to test, because IMHO effectively all 
handhelds can handle HTML. I don't care about converting character 
encodings, because clients already do that. Their web browsers do it.)

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 16 Jan 2008 15:03:12 +0000

Arnt Gulbrandsen wrote:

> All the suggestions I've seen center around providing enough 
> flexibility to cater to whatever is needed. How about having too 
> little flexibility instead? Solving the important 80%?
>
> 1. Convert a bodypart of some sort to a JPEG formatted for the 
> device's screen. (Sometimes this is a sequence of JPEGs, e.g. in the 
> case of a multipage PDF.) The bodypart may be an image, a PDF or even 
> an animation. Often it has MIME type application/octet-stream.
>
> 3. Convert a video or audio bodypart to flash in a given resolution 
> and stream it.
>

Right. draft-ietf-lemonade-streaming-03.txt addresses this.

> implemented, OGG is free of patents and there's good free code 

The problem I have with this proposal is that any given CONVERT=<foo> 
presupposes a list of conversion parameters a compliant server/client 
needs to understand. Which also means that every such extension needs to 
be a separate RFC (which would require an editor and some of them will 
have to be included into Lemonade Profile Bis). What would happen if the 
list of conversion parameters needs to be extended?

And then I am also concerned that we will have an ever growing list of 
CONVERT-related capabilities in the CAPABILITY response. I would be glad 
to be proven wrong on this.

Combining my comments together: I am a firm believer that we need a 
generic conversion parameter discovery mechanism. I don't really care if 
it is METADATA-based or not, even though I have my preferences.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 16 Jan 2008 16:21:41 +0100

Alexey Melnikov writes:
> needs to understand. Which also means that every such extension needs 
> to be a separate RFC (which would require an editor and some of them 
> will have to be included into Lemonade Profile Bis). What would 
> happen if the list of conversion parameters needs to be extended?

Well, first of all, something will be replacing JPEG or MP3, and that's 
a rather bigger event than an RFC publication.

> CONVERT-related capabilities in the CAPABILITY response. I would be 
> glad to be proven wrong on this.

We'll have a new graphics target when a new format replaces JPEG as the 
widely-supported small format of choice, which I confidently predict 
will not happen in the next decade.

Audio doesn't seem to change quite so slowly these days. But if we 
choose the right format, I'm pretty sure we can get at least five years 
before there's strong pressure for another format.

And what else is there but audio and video?

> generic conversion parameter discovery mechanism. I don't really care 
> if it is METADATA-based or not, even though I have my preferences.

I don't see much enthusiasm for actually implementing the generic stuff.

I don't mind metadata as an extension, but it's a lot of baggage if the 
goals are converting whathaveyou to JPEG or MP3.

Arnt

---

From: <[email-masked]>
Date: Fri, 18 Jan 2008 16:07:31 +0200

Hi Arnt,
I think that what You are suggesting is very close to what we are doing.
The format is the easiest thing to add: I do not think it matters whether You write 'JPG', 'PNG' or 'MP3', 'OGG' after the 'CONVERT='.
I mean: when the protocol supports all the parameters for JPG and MP3, the exact same parameters can be re-used in any picture and audio format. (and adding support for video format will be just a combination of the two)

Streaming is different (however, I might want to transcode it [specify the picture size/audio quality] before I start streaming) - but I thought the idea was to get a link to the stream and transmit the stream via a dedicated transport (e.g. not IMAP).

Best regards: Zoltán Ördögh
E-mail: zoltan dot ordogh at nokia dot com
Phone: +358 50 386 0566

>-----Original Message-----
>From: ext Arnt Gulbrandsen [mailto:[email-masked]] 
>Sent: 16 January, 2008 15:35
>To: Lemonade WG
>Subject: [lemonade] strawman suggestion for convert
>
>flexibility to cater to whatever is needed. How about having 
>too little flexibility instead? Solving the important 80%?
>
>device's screen. (Sometimes this is a sequence of JPEGs, e.g. 
>in the case of a multipage PDF.) The bodypart may be an image, 
>a PDF or even an animation. Often it has MIME type 
>application/octet-stream.
>
>2. Convert a bodypart of some sort to to an MP3, chopping it 
>off to limited length.
>
>3. Convert a video or audio bodypart to flash in a given 
>resolution and stream it.
>
>3. Disregard case 3 for CONVERT. We don't do streaming inside 
>IMAP. We could "handle" it by mentioning that a streaming 
>server can use URLAUTH/URLFETCH to obtain the bodypart.
>
>1. Capability CONVERT=JPEG gives the ability to do 1. The 
>client specifies JPEG, x resolution, y resolution, whether to 
>make the result smaller or bigger if the aspect ratio is 
>different, and a maximum number of images. The server converts 
>to JPEG if it can, resizes, preserving aspect ratio, and sends 
>the results.
>
>2. Capability CONVERT=MP3 gives the ability to do 2. The 
>client specifies a bitrate and length. (Or OGG, perhaps. MP3 
>is widely implemented, OGG is free of patents and there's good 
>free code
>available.)
>
>Arnt
>
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 18 Jan 2008 16:48:19 +0100

> I think that what You are suggesting is very close to what we are doing.

Yes, very close, it's just that the last bit is the difficult one...

> The format is the easiest thing to add: I do not think it matters 
> whether You write 'JPG', 'PNG' or 'MP3', 'OGG' after the 'CONVERT='.
> I mean: when the protocol supports all the parameters for JPG and MP3, 
> the exact same parameters can be re-used in any picture and audio 
> format. (and adding support for video format will be just a 
> combination of the two)

Several implementers have expressed great discomfort at the discovery 
mechanisms suggested.

I wonder whether the best solution would be to say that autodiscovery is 
done using METADATA, that servers and clients may either do 
autodiscovery or implement and use only the MTI targets, and define a 
small, but useful set ot targets. JPEG and one audio format (OGG og 
MP3) IMO.

If we do that, a server which only implements the MTI conversions 
doesn't need METADATA, and a client which only uses the MTI targets 
doesn't have to perform target disovery.

Arnt

---

From: <[email-masked]>
Date: Mon, 21 Jan 2008 11:28:43 +0200

Arnt,

>Sent: 18 January, 2008 17:48
>To: [email-masked]
>Cc: Ordogh Zoltan (Nokia-D-MSW/Tampere)
>Subject: Re: [lemonade] strawman suggestion for convert
>
>Several implementers have expressed great discomfort at the 
>discovery mechanisms suggested.

I understand that the problem is the discovery mechanism itself and not
the parameters and formats (so, cutting back to JPG and MP3 only is not
a good solution to the problem).

I think everyone understands that METADATA will not work - it can only
solve half of the problems (when clients understand what they are
dealing with and they know what they want). (It is a different question
that METADATA is not favoured).
A client might have no clue what's inside the attachment (say, a PDF
file (Dave's example)) - so it seems obvious that there has to be some
sort of dynamic negotiation between the server and the client (to figure
out exactly how things will be converted). But: do we want to tackle
this problem? Isn't it easier to say "I want this attachment in a form
that I understand" - and based on some info about the client (that has
been disclosed previously), the server can try creating something that
the client will surely understand. If there is no good match, or the
server does not understand the content either, then the conversion
simply fails.

As for METADATA (in other words, the case when the client understand the
attachment and knows what the target should be) - do we really need
this? Isn't it easier to simply return the supported parameters whenever
a request fails? (Note that OMA MEM will never allow a request to fail.)

The current suggestions on the mailing list are:
 - Dave had a suggestion to tell the server what the client supports
(MEDIACAPS), so the server can choose the "most acceptable" format from
that list.
 - Dan had a suggestion to query  the server about certain source/target
types (CONVERSIONS).

After all this, here is my strawman proposal:
We have two cases (like Dave said):
1. Unpredictable (the client has no clue what it wants to convert and
what will it receive).
2. Predictable conversions (the client knows what it wants to convert to
what).

To tackle case 1., we do what Dave suggested (the client declares its
characteristics and codecs), and we allow "best match" conversion, when
the server picks anything from what the client can display.

To tackle case 2., we do nothing (just drop METADATA), we disclose the
list of all parametersin some specification (or, just refer OMA STI parm
list), and we allow the client to request any conversion (which is on
the list). The requests will be:
 - successful (great!) - we get the content we wanted.
 - unsuccessful I: some parameters have been ignored (like colour
depth), but the conversion has been done according to the requested
format - we get the content we wanted, but not 100% accurately.
 - unsuccessful II: not supported format (or conversion), the server
picks a new format according to the "best match" case - we do not get
the content we wanted, but we get something that we surely can handle.
 - unsuccessful III: mismatch (attept to convert to something funny,
like picture to audio), the same as "unsuccessful II".
 - unsuccessful IV: server does not understand the input, so it just
sends the content without conversion - we do not get any conversion, we
get the original.
 - fail I: bad request (syntax error or something) - client gets no
content
 - fail II: bad input (cannot convert, the file format is broken) -
client gets no content
 - fail III: contains virus (or, other screening issue) - client gets no
content
Finally, whenever the response code is "unsuccessful" or "fail", we can
return the list of supported conversion formats/parms (whichever makes
more sense in that context).

What do all You think?

---

From: Dave Cridland <[email-masked]>
Date: Mon, 21 Jan 2008 11:08:52 +0000

>  - fail III: contains virus (or, other screening issue) - client  

Well, there's no technical reason why a Windows virus can't be  
replaced with one suitable for Symbian, say. Also, should we be  
considering converting image spam to text for more convenient reading  
on mobile devices?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: <[email-masked]>
Date: Mon, 21 Jan 2008 13:37:49 +0200

Hi Dave,

>From: ext Dave Cridland [mailto:[email-masked]] 
>Sent: 21 January, 2008 13:09
>To: Ordogh Zoltan (Nokia-D-MSW/Tampere)
>Cc: Enhancements to Internet email to support diverse service 
>enivronments
>Subject: RE: [lemonade] strawman suggestion for convert
>
>>  - fail III: contains virus (or, other screening issue) - 
>client gets 
>> no content
>
>replaced with one suitable for Symbian, say. Also, should we 
>be considering converting image spam to text for more 
>convenient reading on mobile devices?

If the conversion fails, I guess the client should not get the mail either, or, as an alternatives:
 - a) the whole mail could be replaced with a text telling the reason - like "this mail contains harmful information for this client, therefore it cannot be retireved. however, you may try another client"
 - b) replace the suspicious bodypart and add a text to the mail saying that "a suspicious attachment has been replaced in this mail. if you know what you are doing, you can get it from this URL:"
But these are implementation issues.

The majority of the spam should be filtered when they are deposited, and the screening and virus scans should take place before deposit, too. But of course people always find a way to slip things through the safeguards. Today, all unfiltered spam is delivered to your mailbox - it might be a waste of time to convert spam, but since it has passed the filter, there is no way of telling that it is spam until the recipient has decided that it is (something that you consider spam might not be a spam for me), so the best thing we can do is to convert it and let the user judge it for himseld (he will update his filters if he thinks that it is spam).

As for the type of the virus scans, I think that this is an implementation issue and we should not mention anyhing specific (even though we want to scan for all possible viruses, including Linux, Mac and all other OSes as well).

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 21 Jan 2008 13:11:01 +0000

Hi Zoltan,
Some comments (as an individual and not as an editor of CONVERT) on a 
part of your message:

[email-masked] wrote:

>To tackle case 2., we do nothing (just drop METADATA),
>
I will send out a questionnaire on this shortly (later this week), so 
that we can hopefully get closure on this topic.
However I have some comments on a part of your proposal:

>we disclose the list of all parameters in some specification (or, just refer OMA STI parm
>list)
>
There are 2 problems with this:
1). Extensibility problem: This implies that the list of supported 
parameters and that it can only be updated/changed by revising or 
extending the CONVERT document (or another companion document that will 
define them). I don't think this is convenient in long term.
2). IANA registry problem: We can't just reference the OMA STI list, as 
the CONVERT document is now using the IANA registry established by RFC 
2506. We need to at least register OMA STI parameters in this registry.

>, and we allow the client to request any conversion (which is on
> - unsuccessful I: some parameters have been ignored
>
I would just note that the CONVERT requires that a parameter is always 
honored, unless the client explicitly allows to ignore it.
We've talked about "you can ignore conversion parameters" parameter 
before, but this is still something that needs to be standardized (IMHO, 
in a document separate from IMAP CONVERT) and registered with IANA.

>(like colour
Another note: the CONVERT extension as written would instead fail the 
conversion.

>more sense in that context).
I think your list of failure cases is useful, we just need to agree how 
they need to be handled.

---

From: <[email-masked]>
Date: Mon, 21 Jan 2008 15:55:51 +0200

Hi Alexey,
I added some comments below.

>From: ext Alexey Melnikov [mailto:[email-masked]] 
>Sent: 21 January, 2008 15:11
>Cc: [email-masked]
>Some comments (as an individual and not as an editor of 
>CONVERT) on a part of your message:
>
>I will send out a questionnaire on this shortly (later this 
>week), so that we can hopefully get closure on this topic.
>>we disclose the list of all parameters in some specification 
>(or, just 
>>refer OMA STI parm
>>list)
>>
>1). Extensibility problem: This implies that the list of 
>supported parameters and that it can only be updated/changed 
>by revising or extending the CONVERT document (or another 
>companion document that will define them). I don't think this 
>is convenient in long term.

It is quite clear that we do not want to update the specification every time someone proposes something new, or comes up with a new parameter.
I had the understanding that syntax will be generic to support any conversion parameters and Lemonade will specify only a limited set of parameters that will be mandatory. It seems logical to leave the non-mandatory parameters out completely, and leave a reference where they can be found. The IANA registry is a good candidate.

>2). IANA registry problem: We can't just reference the OMA STI 
>list, as the CONVERT document is now using the IANA registry 
>established by RFC 2506. We need to at least register OMA STI 
>parameters in this registry.

I do not disagree with this, but I recognize that there is work to be done here. I have no idea what needs to be and how - but I offer my assistance.

>>, and we allow the client to request any conversion (which is on the 
>>list). The requests will be:
>I would just note that the CONVERT requires that a parameter 
>is always honored, unless the client explicitly allows to ignore it.
>We've talked about "you can ignore conversion parameters" 
>parameter before, but this is still something that needs to be 
>standardized (IMHO, in a document separate from IMAP CONVERT) 
>and registered with IANA.

Indeed, this is a choice a client should be able to make. So, we can split this into two, and say that it only occurs when excplicit conversion was not requested, while the excplicit conversion will obviously fail (and give a list of parameters/formats).

>Another note: the CONVERT extension as written would instead 
>fail the conversion.

Yes, I am aware of that - this is a proposal at this point. Might be accepted, might change. If accepted, I guess we will have to update CONVERT respectively.

>> - fail III: contains virus (or, other screening issue) - client gets 
>>no content Finally, whenever the response code is "unsuccessful" or 
>>"fail", we can return the list of supported conversion formats/parms 
>>(whichever makes more sense in that context).
>>
>>What do all You think?
>>  
>>
>I think your list of failure cases is useful, we just need to 
>agree how they need to be handled.

Sounds good. I guess we should wait for feedback from the other timezones as well.