Subject: Re: [imapext] The IMAP APPENDLIMIT Extension - version 04

From: Chris Newman <[email-masked]>
Date: Tue, 03 Mar 2015 17:32:39 -0800

--On February 27, 2015 20:57:05 +0000 Jayantheesh S B <[email-masked]>
wrote:
> Please find the next version (04) of the draft attached with review comments
> addressed.
> 
> Changes in this version:
> 
> 1.       Append-limit-value changed from 32 bits to 64 bits unsigned number

Please use a 63-bit number to align with the new condstore RFC 7162. For those
who write IMAP software in C/C++, there is a technical argument to prefer use
of signed integers:
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.html#Integer_Types
 (see "On Unsigned Integers")
So a 63-bit protocol limit allows the implementer freedom to choose the type.

The document should state that an append limit in excess of UINT32_MAX means
the server accepts larger-than 32-bit-unsigned integer lengths in append
command data literals (this change means this document updates the IMAP base
spec).

Q for list: If the server has a message larger than UINT32_MAX in the store and
a client tries to fetch it, should the server just go ahead and violate the
base-spec's UINT32_MAX limit or should the data be NIL unless the client uses
ENABLE to tell the server it accepts larger-than-uint32-max literals on
messages? If this extension fixes that base spec limitation in addition to
enabling better client diagnostics for overlarge messages, I think that's cool.

Having two different ways to access a per-folder limit is a violation of good
protocol design practices. Specifically 3.1 and 3.2 provide two ways to access
the same appendlimit. I believe the document would be improved by the removal
of section 3.1. Some problems with the current two-mechanism proposal:

* Which limit will the client use if the two limits are different?
* Should a client SELECT a mailbox before doing an APPEND? This is not required
by the base spec and is less efficient. The presence of section 3.1 implies a
client should do this. I do not think a client should do this. Removing section
3.1 removes this issue as a problem.
* The addition of APPENDLIMIT as a response to SELECT even for clients that
don't understand it penalizes all clients with unnecessary network traffic.
While it's not a big response by itself, we're starting to get a
death-of-1000-cuts problem with SELECT responses, so unless it's really
important (and in most cases I think the capability is good enough) I'd rather
not add that extra traffic (and potential battery life cost on mobile devices).
* The document does not state if the APPENDLIMIT response is a MUST/SHOULD/MAY.
If it's a MAY, my server implementation won't send it, in which case why is it
in the spec? If it's a SHOULD, I'll include a product option so admins can
disable it. If it's a MUST, I object to advancing the proposal.

I'm also curious, does anyone intend to implement different per-mailbox append
limits? If nobody speaks up, then perhaps we should just remove all of section
3.

One suggestion for the list to discuss; should we allow the limit to be
expressed in terms of Megabytes or Gigabytes via a suffix? Thus we'd have:

append-limit-value = 1*DIGIT ["G" / "M"]
                     ; M = Megabyte (* 1,000,000)
                     ; G = Gigabyte (* 1,000,000,000)
                     ; Unsigned 63-bit integer after multiplier
                     ; (1 <= n <= 9,223,372,036,854,775,807).

Not much additional code and makes the protocol both more concise and human
readable. Just a suggestion (we have some product config options like this).

FYI, I am interested in implementing this when it approaches rough consensus.

		- Chris

---

From: Jayantheesh S B <[email-masked]>
Date: Wed, 4 Mar 2015 17:11:12 +0000

Chris,

Thanks for your time in reviewing the document. 

Please find my response inline. 

Regards,
Jay

-----Original Message-----
From: Chris Newman [mailto:[email-masked]] 
Sent: Tuesday, March 03, 2015 8:33 PM
To: Jayantheesh S B; [email-masked]
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension - version 04

--On February 27, 2015 20:57:05 +0000 Jayantheesh S B <[email-masked]>
wrote:
> Please find the next version (04) of the draft attached with review 
> comments addressed.
> 
> Changes in this version:
> 

Please use a 63-bit number to align with the new condstore RFC 7162. For those who write IMAP software in C/C++, there is a technical argument to prefer use of signed integers:
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.html#Integer_Types
 (see "On Unsigned Integers")
So a 63-bit protocol limit allows the implementer freedom to choose the type.

The document should state that an append limit in excess of UINT32_MAX means the server accepts larger-than 32-bit-unsigned integer lengths in append command data literals (this change means this document updates the IMAP base spec).

[Jay] Really I am wondering if anyone have the APPENDLIMIT value for individual mail greater than 2^32. Because, I see most of the providers like Gmail, restricts supporting the attachment handling with Max of ~25 MB.
I see in real time use case, supporting more than 512 MB of APPENDLIMIT per mail is practically not the going to happen. Appending mail with attachment more than 512MB of data is not realistic. Moreover, this value is not directly
 related to physical mailbox limit. So, am thinking to restrict the APPENDLIMIT value to 32 bits. Please share if you have any comments. 

Q for list: If the server has a message larger than UINT32_MAX in the store and a client tries to fetch it, should the server just go ahead and violate the base-spec's UINT32_MAX limit or should the data be NIL unless the client uses ENABLE to tell the server it accepts larger-than-uint32-max literals on messages? If this extension fixes that base spec limitation in addition to enabling better client diagnostics for overlarge messages, I think that's cool.

Having two different ways to access a per-folder limit is a violation of good protocol design practices. Specifically 3.1 and 3.2 provide two ways to access the same appendlimit. I believe the document would be improved by the removal of section 3.1. Some problems with the current two-mechanism proposal:

[Jay] Agree with you. But APPEND command can be sent in both Authenticated as well as Selected state, that's why added APPENDLIMIT response in both LIST as well as SELECT.  Moreover in Capabilities, if APPENDLIMIT  string without any value is received indicates, server has per mailbox limit. In this case, client has to rely on the LIST or SELECT response to get the per mailbox specific limit. In case, if APPENDLIMIT is not published as part of capabilities indicates, this extension is not supported in the IMAP server, so client can assume it won't receive the APPENDLIMIT as part of LIST and SELECT. Even if server sends in this case, client can ignore. 
Regarding the per mailbox specific limit,  based on the earlier discussion review comments section 3 has been added. But if everybody feels, its unnecessary to have the mailbox specific limit for this use case. We can re-consider it.

* Which limit will the client use if the two limits are different?
* Should a client SELECT a mailbox before doing an APPEND? This is not required by the base spec and is less efficient. The presence of section 3.1 implies a client should do this. I do not think a client should do this. Removing section
3.1 removes this issue as a problem.
* The addition of APPENDLIMIT as a response to SELECT even for clients that don't understand it penalizes all clients with unnecessary network traffic.
While it's not a big response by itself, we're starting to get a death-of-1000-cuts problem with SELECT responses, so unless it's really important (and in most cases I think the capability is good enough) I'd rather not add that extra traffic (and potential battery life cost on mobile devices).
* The document does not state if the APPENDLIMIT response is a MUST/SHOULD/MAY.
If it's a MAY, my server implementation won't send it, in which case why is it in the spec? If it's a SHOULD, I'll include a product option so admins can disable it. If it's a MUST, I object to advancing the proposal.

[Jay] I will update your comments in the next version of the draft.

I'm also curious, does anyone intend to implement different per-mailbox append limits? If nobody speaks up, then perhaps we should just remove all of section 3.

One suggestion for the list to discuss; should we allow the limit to be expressed in terms of Megabytes or Gigabytes via a suffix? Thus we'd have:

Not much additional code and makes the protocol both more concise and human readable. Just a suggestion (we have some product config options like this).

		- Chris

---

From: Michael M Slusarz <[email-masked]>
Date: Fri, 06 Mar 2015 14:27:02 -0700

> enabling better client diagnostics for overlarge messages, I think  

I agree that the max limit should be extended.  However, it doesn't  
seem like the APPENDLIMIT document is the (best) place to do that  
since burying a change in a document devoted to APPEND behavior that  
affects a general IMAP limit, e.g. changing FETCH behavior, doesn't  
seem to be the model of clarity.

> of section 3.1. Some problems with the current two-mechanism proposal:

+1.  This is what I had written about previously.

> 3.1 removes this issue as a problem.

More important... this may be a non-trivial value to calculate for the server.

> not add that extra traffic (and potential battery life cost on  

To me the most common use case for APPEND is sent-mail saving.   
Generally this will occur in one of two situations:

1.) Replying/Forwarding to an existing message. In these cases, it is  
likely that at the time of sending that the client will still be in  
the same mailbox - most people tend to write a message in one sitting  
so they won't change their mailbox - and this mailbox will NOT be the  
sent-mail mailbox.
2.) Composing a New Message. The client IMAP connection may be in  
either a selected or non-selected state depending on their previous  
actions in the client UI, but odds are high that they are unlikely to  
be in the Sent mailbox.

In either of these cases it is going to be a MUCH less expensive  
operation to grab the APPENDLIMIT with a STATUS(-like) call followed  
by an APPEND rather than SELECTing the mailbox and then APPENDing.   
Not only will the client code have to do the normal state  
initalization of opening a mailbox, which probably isn't necessary  
from a UI perspective since the user doesn't care about the  
contents/status of the sent-mail mailbox, but this initialization  
could involve other delays if the server is doing things like  
on-demand new message tasks within that mailbox (e.g. header caching,  
search indexing).

TLDR: APPENDs to a sent mail mailbox are the most common real-world  
APPEND action, and these APPENDs rarely occur when in selected state  
within the sent mail mailbox.

> disable it. If it's a MUST, I object to advancing the proposal.

Agree.  It can be argued that this would actually make things slightly  
slower for the vast majority of clients, with no (or microscopic) gain  
to APPENDLIMIT aware clients.

> limits? If nobody speaks up, then perhaps we should just remove all  

To me it seems like a fairly common use case that public and/or shared  
namespaces might have a different append "quota" than a private  
namespace.  At least enough of a possibility that it makes sense to  
allow different per mailbox limits.

Earlier it has been mentioned that QUOTA might also change the value.   
There's been discussion of whether this should change/factor into the  
APPENDLIMIT value returned.  If it doesn't, and thus APPENDLIMIT does  
not necessarily reflect the actual value of the current append limit  
on the server, I continue to question how useful APPENDLIMIT would be  
as compared with the current solution of doing a non-LITERAL+ APPEND.

Speaking of this latter option ... discussion of it is completely  
absent from the APPENDLIMIT draft.  At a minimum, the APPENDLIMIT  
draft needs to make clear that the extension only really effects  
behavior when using the combination of APPEND + LITERAL+.  There has  
never been an issue with sending unnecessary data if just using  
APPEND.  I think a discussion, or at least a prominent reference to  
RFC 4549 [4.2.2.3], is necessary to make this distinction clear.

Part of me is wondering whether APPENDLIMIT is even worth it and if  
the better solution is instead that use of APPEND w/LITERAL+ should be  
explicitly discouraged in the LITERAL+ update that Alexey has been  
working on.

> readable. Just a suggestion (we have some product config options like this).

The append limit value is being machine parsed, and this doesn't save  
that much in terms of transfer bandwidth, so I don't see much benefit  
in this.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 09 Mar 2015 12:00:09 +0000

Hi,
After thinking more about this topic (I might have changed my position 
on a few minor issues):

> seem to be the model of clarity.
I agree.
> +1.  This is what I had written about previously.
Not really a violation, because the data is the same whether returned by 
STATUS or SELECT
> More important... this may be a non-trivial value to calculate for the 
(Depending on how accurate a server might want to make this information. 
This is a quality of implementation issue.)
> within the sent mail mailbox.
Ok, I don't have a strong preference one way or another, but I find your 
argument convincing and I can live without APPENDLIMIT be returned on 
SELECT.
> to APPENDLIMIT aware clients.
Right.

  [snip]

---

From: Jayantheesh S B <[email-masked]>
Date: Wed, 18 Mar 2015 19:47:13 +0000

All,

Can you please share your views regarding the below item to come to a consensus?

1.  APPENDLIMIT value should be UINT32_MAX or UINT64_MAX ?

2.  APPENDLIMIT value in SELECT/STATUS response: Whether APPENDLIMIT is needed in both SELECT and STATUS response? 

Please share your thoughts.

Regards,
Jay

-----Original Message-----
From: Alexey Melnikov [mailto:[email-masked]] 
Sent: Monday, March 09, 2015 8:00 AM
To: Michael M Slusarz; [email-masked]
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension - version 04

Hi,
After thinking more about this topic (I might have changed my position on a few minor issues):

On 06/03/2015 21:27, Michael M Slusarz wrote:
>> Q for list: If the server has a message larger than UINT32_MAX in the 
>> store and a client tries to fetch it, should the server just go ahead 
>> and violate the base-spec's UINT32_MAX limit or should the data be 
>> NIL unless the client uses ENABLE to tell the server it accepts 
>> larger-than-uint32-max literals on messages? If this extension fixes 
>> that base spec limitation in addition to enabling better client 
>> diagnostics for overlarge messages, I think that's cool.
>
I agree.
>
>> Having two different ways to access a per-folder limit is a violation 
>> of good protocol design practices. Specifically 3.1 and 3.2 provide 
>> two ways to access the same appendlimit. I believe the document would 
>> be improved by the removal of section 3.1. Some problems with the 
>> current two-mechanism proposal:
>
Not really a violation, because the data is the same whether returned by STATUS or SELECT
>> * Should a client SELECT a mailbox before doing an APPEND? This is 
>> not required by the base spec and is less efficient. The presence of 
>> section 3.1 implies a client should do this. I do not think a client 
>> should do this.
>> Removing section
>
>> * The addition of APPENDLIMIT as a response to SELECT even for 
>> clients that don't understand it penalizes all clients with 
>> unnecessary network traffic.
>> death-of-1000-cuts problem with SELECT responses, so unless it's 
>> really important (and in most cases I think the capability is good 
>> enough) I'd rather not add that extra traffic (and potential battery 
>> life cost on mobile devices).
>
Ok, I don't have a strong preference one way or another, but I find your argument convincing and I can live without APPENDLIMIT be returned on SELECT.
>> If it's a MAY, my server implementation won't send it, in which case 
>> why is it in the spec? If it's a SHOULD, I'll include a product 
>> option so admins can disable it. If it's a MUST, I object to 
>> advancing the proposal.
>
Right.

  [snip]

---

From: Pete Maclean <[email-masked]>
Date: Wed, 18 Mar 2015 18:02:04 -0400

>1.  APPENDLIMIT value should be UINT32_MAX or UINT64_MAX ?

UINT64_MAX

>2.  APPENDLIMIT value in SELECT/STATUS response: Whether APPENDLIMIT 
>is needed in both SELECT

I don't feel strongly about this but, considering the recent 
arguments on the matter, I lean toward having it only in STATUS.

Pete Maclean

>and STATUS response?
At 03:47 PM 3/18/2015, Jayantheesh S B wrote:
>2.  APPENDLIMIT value in SELECT/STATUS response: Whether APPENDLIMIT 
>is needed in both SELECT and STATUS response?
>
>Ok, I don't have a strong preference one way or another, but I find 
>your argument convincing and I can live without APPENDLIMIT be 
>returned on SELECT.
>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 19 Mar 2015 17:18:51 +1100

> >1.  APPENDLIMIT value should be UINT32_MAX or UINT64_MAX ?

Most of the world is going with half that in new standards, for
compatibility with languages that offer a signed 64 bit numeric value
only (aka - java 7 and below).  It makes mapping the values through said
languages a lot easier, and 63 bits should be enough for anyone (or
something).

Alternatively, there's 53 bits, which is the maximum that javascript can
represent:

http://stackoverflow.com/questions/307179/what-is-javascripts-highest-integer-value-that-a-number-can-go-to-without-losin

It's not going to matter in practice, because that's still fricking huge.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Pete Maclean <[email-masked]>
Date: Fri, 20 Mar 2015 12:10:55 -0400

Thanks, Bron.  I am aware of Java's lack of unsigned integers having 
had to work around it before.  (I find it a curious lack although I 
have no idea of the rationale behind it.)  So, yes, I come to agree 
that UINT64_MAX would be a poor choice, never mind an unnecessarily 
large one.  So how about a compromise of, say, UINT48_MAX?

Pete

---

From: Michael M Slusarz <[email-masked]>
Date: Fri, 20 Mar 2015 10:20:21 -0600

> large one.  So how about a compromise of, say, UINT48_MAX?

I think we discussed this earlier in this (or a related) thread.

But it probably makes the most sense to use an unsigned positive  
63-bit value, as Bron recommended, to remain consistent with other  
IMAP extensions, e.g. CONDSTORE:

http://tools.ietf.org/html/rfc7162#section-3.1

michael

---

From: Dave Cridland <[email-masked]>
Date: Fri, 20 Mar 2015 16:28:12 +0000

Ugh. Hit reply instead of reply to all.

> 1.  APPENDLIMIT value should be UINT32_MAX or UINT64_MAX ?
UINT32_MAX.

That's not to say another document couldn't raise this, but I strongly
object to hiding this fundamental limit increase in an unrelated spec.

> 2.  APPENDLIMIT value in SELECT/STATUS response: Whether APPENDLIMIT is
> needed in both SELECT and STATUS response?
>
>
I think that APPENDLIMIT would probably benefit from not being
mailbox-specific, but if the consensus is that this is needed, then put it
in both. Also probably a response-code on a NO to APPEND.

The draft needs examples for APPENDs, too.

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 20 Mar 2015 19:34:16 +0000

I understand Dave's rationale, given that currently the max message size 
is 4GB

I think however this limit also needs to be looked at.  Last time I 
raised it I didn't get anywhere.  However as time goes on, the chances 
of 4GB being insufficient for a message increases.

We already support 64 bit sizes in our server, and probably so do 
others.

------ Original Message ------
From: "Dave Cridland" <[email-masked]>
To: "Jayantheesh S B" <[email-masked]>
Cc: "Alexey Melnikov" <[email-masked]>; "[email-masked]" 
<[email-masked]>; "Michael M Slusarz" <[email-masked]>
Sent: 21/03/2015 5:28:12 a.m.
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension - version 04

>mailbox-specific, but if the consensus is that this is needed, then put 
>it in both. Also probably a response-code on a NO to APPEND.
>