Subject: [imapext] a very simple little imap extension, really quite magnificently simple

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 25 Apr 2014 14:30:20 +0200

Hi,

as some of you will have discovered, opening large mailboxes takes real 
time. So I discussed that with Brandon and wrote a draft, really short, 
perhaps it's actually shorter than the boilerplate. The gist: "If a client 
runs ENABLE UIDONLY, then that client cannot later use any MSNs in any 
commands, and if syntax forces the server needs to send an MSN, then it 
just sends a phony number." There are a couple of details.

Attached. Comments most welcome.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 25 Apr 2014 22:42:48 +1000

> just sends a phony number." There are a couple of details.

I'm sold on the idea.  MSNs are a wart.  They require the server to calculate
and maintain state that the client may not actually need.

I would also have been happy with a different type of SELECT command, but
entire session ENABLE command makes sense too.

It's a pity you can't use 0 as the MSN, but I don't see a problem with just
using 1, rather than some random 4-5 digit number.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 25 Apr 2014 15:46:57 +0200

On Friday, 25 April 2014 14:30:20 CEST, Arnt Gulbrandsen wrote:
>    Opening a large mailbox in IMAP can take mailbox; 30 seconds is

"...can take a long time"?

>    This extension provides a way to avoid having to number the messages
>    consecutively.

Are there any other mail servers out there which actually have a problem 
with sequence numbers, besides GMail? Just wondering what is more probable 
-- GMail adding $code which makes this faster because it's a pain for their 
clients, or $developers updating N clients to make use of this extension.

>    command, and that it MUST NOT expect to receive an EXISTS response.

How is the client supposed to notice that a new mail has arrived, then? I 
see it addressed later in the doc, but it startled me when I read this.

>    While the client will still receive MSNs, it MUST NOT expect them to
>    mean anything. (RFC editor: Remove this parenthesis. It seems to be
>    easier to add support for UIDONLY by disregarding parsed MSNs than to
>    change both the parser and the layer above, so I chose to leave dummy
>    MSNs in.)

I dislike this a lot. If the MSNs do not mean anything, just send a BAD and 
possibly even terminate the connection. Otherwise the garbage which is sent 
will only lead to further confusion of a non-compliant client.

>    These rules apply from the time the server has received an ENABLE
>    command that enables UIDONLY until the TCP connection is closed.

"until the IMAP session terminates". There's no requirement for IMAP to run 
over TCP. In fact, I used to use IMAP-over-SSH-with-PREAUTH on a daily 
basis for many years, and only stopped when they cut down the SSH access. I 
expect to use that in future as well.

>    The server MUST send the UID data item in all FETCH responses,
>    including untagged FETCH responses.

Which means that you effectively require QRESYNC and its VANISHED. Without 
that option it won't work. I see that you talk about that later in the 
text, but it should probably be a hard dep on 5162-bis.

Given that this extension disables EXISTS, you need soemthing like ARRIVED 
[1]. This change on its own will complicate this dratf a lot.

Also, what about STATUS and querying the offset of first message having a 
certain property through that? Or even STATUS EXISTS?

>    The server MUST respond with BAD to any command that uses an MSN,

Add a requirement for [CLIENTBUG], please. We have these response codes for 
a reason.

>    including a UID SEARCH command that uses MSNs. This search command
>    would retrieve the UID of the last message:

"...but the reference to the "last message" is specified as a MSN."

>    The server MUST NOT send EXISTS responses at any time (not even
>    during SELECT/EXAMINE processing), and MUST send UIDNEXT when it
>    would have sent EXISTS.

And the client doesn't know how many messages are in the mailbox anymore. 
That's a rather radical change. I don't like it at all, it's a nice 
safeguard to detect broken sync and this extension takes it away.

> For example:
>
>       C: a idle
>       S: +
>       S: * OK [UIDNEXT 10240902] next UID

This *will* lead to race conditions. Basically, this extension makes an 
unspoken assumption on UIDs being assigned in a consecutive manner, without 
any gaps. I bet there is at least one server implementaiton which can sleep 
and will notice that two messages have arrived and the first of them got 
deleted. How do you convey that?

>    The server MUST send VANISHED responses (see [RFC5162], section 3.6)
>    where it would ordinarily send EXPUNGED responses. ([RFC5162] defines
>    a large extension called Quick Resync, of which VANISHED is a small
>    part. Note that neither the server nor the client need support or use
>    Quick Resync in general.)

I don't like mandating VANISHED without also requiring the rest of the 
QRESYNC functionality. I for one cannot imagine how to sync a mailbox at 
all in at least semi efficient manner when I cannot rely on UIDNEXT/EXISTS 
combination and without QRESYNC's availability at the same time.

>    This document is believed not to have any security implications.

This extension, as-is, opens up a few opportunities for client and server 
to go out of sync, and potentially could lead to loss of data. When these 
problems are mitigated, the doc IMHO still needs a brief section saying how 
that was done and what bugs and gotchas are expected when an implementor 
botches up something.

>    [RFC5162]   Melnikov, A. and D. Cridland, "IMAP4 Extensions for Quick
>               Mailbox Resynchronization", RFC 5162, March 2008.

Would be cool to have a reference to 5162-bis when it gets approved.

Cheers,
Jan

[1] http://tools.ietf.org/html/draft-kundrat-qresync-arrived-00

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 25 Apr 2014 18:57:38 +0200

On Friday, April 25, 2014 3:46:57 PM CEST, Jan Kundrát wrote:
> Are there any other mail servers out there which actually have 
> a problem with sequence numbers, besides GMail? Just wondering 
> what is more probable -- GMail adding $code which makes this 
> faster because it's a pain for their clients, or $developers 
> updating N clients to make use of this extension.

I expect it hits everyone who tries to have n million messages in a single 
mailbox.

Storing MSNs persistently so you don't have to recompute at session start 
(and at the same time supporting simultaneous sessions) was simply too 
difficult for me.

> How is the client supposed to notice that a new mail has 
> arrived, then? I see it addressed later in the doc, but it 
> startled me when I read this.

So you learned it twenty lines later ;)

> I dislike this a lot. If the MSNs do not mean anything, just 
> send a BAD and possibly even terminate the connection. Otherwise 
> the garbage which is sent will only lead to further confusion of 
> a non-compliant client.

The client cannot send BAD.

The server does send BAD when it receives a BAD from the client.

> Which means that you effectively require QRESYNC and its 
> VANISHED. Without that option it won't work. I see that you talk 
> about that later in the text, but it should probably be a hard 
> dep on 5162-bis.

Disagree violently. 5162 is MUCH too complex to require all of it.

> Given that this extension disables EXISTS, you need soemthing 
> like ARRIVED [1]. This change on its own will complicate this 
> dratf a lot.

Why? What's wrong with UIDNEXT?

> Also, what about STATUS and querying the offset of first 
> message having a certain property through that? Or even STATUS 
> EXISTS?

I don't think that matters.

If a client issues STATUS EXISTS, it kills any performance benefit it got 
by issuing ENABLE UIDONLY. That's a problem for the client, but not an 
interop problem, so nothing that requires rules here.

> Add a requirement for [CLIENTBUG], please. We have these 
> response codes for a reason.

Do any other RFCs require CLIENTBUG? (Not suggest, or show in examples. 
Require.)

> And the client doesn't know how many messages are in the 
> mailbox anymore. That's a rather radical change. I don't like it 
> at all, it's a nice safeguard to detect broken sync and this 
> extension takes it away.

Not knowing how many messages there are is a necessary effect of this. It's 
not an accidental problem, it's necessary.

The point of this is to stop counting messages. If you stop counting, you 
don't know (at least not precisely) how many you have. It's as simple as 
that.

> This *will* lead to race conditions. Basically, this extension 
> makes an unspoken assumption on UIDs being assigned in a 
> consecutive manner, without any gaps.

Why do you think so?

> I bet there is at least 
> one server implementaiton which can sleep and will notice that 
> two messages have arrived and the first of them got deleted. How 
> do you convey that?

* VANISHED 24134789
* OK [UIDNEXT 24135791]

> I don't like mandating VANISHED without also requiring the rest 
> of the QRESYNC functionality. I for one cannot imagine how to 
> sync a mailbox at all in at least semi efficient manner when I 
> cannot rely on UIDNEXT/EXISTS combination and without QRESYNC's 
> availability at the same time.

I can show you code that does that, OK?

> This extension, as-is, opens up a few opportunities for client 
> and server to go out of sync, and potentially could lead to loss 
> of data.

No, if you lose sync the worst that happens is that you send uid store 
+flags \deleted for something that's already deleted.

Remember: With this extension there are no MSNs any more. There's no way to 
store +flags on the wrong message.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 25 Apr 2014 19:07:30 +0200

> using 1, rather than some random 4-5 digit number.

I don't see any advantage in 1. The strange number is recognizable; if you 
see that foo == 8214 in the debugger you have a fair chance of remembering 
that you recently implemented RFC8214 and realise that the problem you're 
chasing is related to that. Ditto google for "imap 8214".

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 25 Apr 2014 19:28:59 +0200

On Friday, 25 April 2014 18:57:38 CEST, Arnt Gulbrandsen wrote:
> Storing MSNs persistently so you don't have to recompute at 
> session start (and at the same time supporting simultaneous 
> sessions) was simply too difficult for me.

Given that you already have to track UIDs, I find it hard that it is 
expensive to take a look at the size of their set.

> The server does send BAD when it receives a BAD from the client.

The draft also says that the server should return a strange number when it 
would otherwise return a MSN. I object to that. In my opinion the server 
should signal this in a drastic way, including terminating the session if 
everything else fails. Sending back a mock-up MSN is about as evil as one 
could go in my book.

> Why? What's wrong with UIDNEXT?

The fact that you can have gaps in UID range. The draft assumes that UIDs 
are allocated by adding one the the previous UID. Your whole scheme falls 
apart when this requirement does not hold. That requirement is not 
documented anywhere in the draft.

If the UIDs are not allocated consequently, and the highest UID right now 
is 12345, what does the following mean:

	* OK [UIDNEXT 12500] guess it

> If a client issues STATUS EXISTS, it kills any performance 
> benefit it got by issuing ENABLE UIDONLY. That's a problem for 
> the client, but not an interop problem, so nothing that requires 
> rules here.

So this is an extension whose benefit is immediately neutralized by a 
client which wants to show the user the total number of messages. To me, 
that sounds like a very basic operation.

> Do any other RFCs require CLIENTBUG? (Not suggest, or show in 
> examples. Require.)

I'm not sure how that is relevant. This proposal is about designing an 
extension which radically changes how IMAP operates. Why not at least 
*mandate* a machine readable way of saying "hey client, you've just shot 
yourself in the foot"?

>> mailbox anymore. That's a rather radical change. I don't like 
>> it at all, it's a nice safeguard to detect broken sync and this 
>> extension takes it away.
>
> Not knowing how many messages there are is a necessary effect 
> of this. It's not an accidental problem, it's necessary.
>
> The point of this is to stop counting messages. If you stop 
> counting, you don't know (at least not precisely) how many you 
> have. It's as simple as that.

OK, at this point I cannot imagine how such a client is supposed to work 
*at all*. Let's start with a blank state. The client knows nothing about 
the mailbox. It opens it and presumably wants to show a list of messages to 
the user. What is the client supposed to do now?

What if it wants to show just a subset of the mailbox, perhaps last 1k 
items (actually it doesn't have to be 1k, anything between 500 and 3000 is 
probably fine -- it needs *something*, but not the full 4M mailbox). How to 
achieve that?

What happens when a mail arrives?

How is a resynchronization after being offline for a while going to look 
like?

My answers and comments to the rest of your mail are very much dependant on 
how you propose the client to behave. I can imagine that I have all sorts 
of assumptions because I know how my client works. You might be coming from 
a different background. So I'm interested in seeing how the proposed mode 
of operation is supposed to actually work. I just don't get it now.

Cheers,
Jan

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 25 Apr 2014 22:45:21 +0200

On Friday, April 25, 2014 7:28:59 PM CEST, Jan Kundrát wrote:
> Given that you already have to track UIDs, I find it hard that 
> it is expensive to take a look at the size of their set.

But I don't have to track UIDs. I just have to track the uidnext value so I 
can allocate new UIDs.

> The draft also says that the server should return a strange 
> number when it would otherwise return a MSN. I object to that.

OK, how do you think an unsolicited fetch response should look?

a) Not have a number after the leading asterisk.
b) Have a number, but it should not be strange.

> In my opinion the server should signal this in a drastic way, 
> including terminating the session if everything else fails. 
> Sending back a mock-up MSN is about as evil as one could go in 
> my book.

We're talking about unsolicited fetch responses here. Perhaps other things, 
there are dozens of IMAP extensions and I at least do not know all of them 
entirely. But practically speaking, unsolicited fetch responses.

> The fact that you can have gaps in UID range. The draft assumes 
> that UIDs are allocated by adding one the the previous UID. Your 
> whole scheme falls apart when this requirement does not hold. 
> That requirement is not documented anywhere in the draft.

I really do not understand this.

> If the UIDs are not allocated consequently, and the highest UID 
> right now is 12345, what does the following mean:
>

"There are <=5 new messages. If you want them, you can send uid fetch 
12345:* and get the details you choose you want about all the new 
messages".

> So this is an extension whose benefit is immediately 
> neutralized by a client which wants to show the user the total 
> number of messages. To me, that sounds like a very basic 
> operation.

Oh? I once asked a few colleagues at a googleapps shop about that, and none 
of them had any idea about their total number of messages. "Oh, lots", 
"I've never deleted anything", that was the sort of answer I got. None told 
me any number. One looked at his screen and told me how many per cent of 
his quota he used.

So I don't think that operation is basic or important for users of _large_ 
mailboxes.

> I'm not sure how that is relevant. This proposal is about 
> designing an extension which radically changes how IMAP 
> operates. Why not at least *mandate* a machine readable way of 
> saying "hey client, you've just shot yourself in the foot"?

I can tell you why I didn't add anything like that in 5530. 5530, as you 
may have noticed, only lists codes, it doesn't say that you have to use any 
code. If two codes apply, it doesn't say which one you must choose.

That's because when I surveyed servers for 5530, I found a few instances 
where two response codes applied. Two errors occured, and it wasn't clear 
that one response code ought to win over the other. So I decided to not 
issue a rule on that. A server can let the first response code win, the 
last one win, it doesn't matter.

I think that also applies in this case. Suppose there's a command such as 
"uid search charset iso-8859-42 1 body foo". The charset specification is 
reasonable, but the 8859 committee hasn't issued number 42 yet, so that 
causes one response code. The presence of the MSN causes another. The 
syntax only permits one, so the server has to choose which one to send. Do 
you think that in this case, the MSN-related response code ought to win 
over the charset-related code? Do you think it ought to win over EVERY 
other code in ALL cases?

I think it doesn't matter very much, so let the server do whatever comes 
easiest.

> OK, at this point I cannot imagine how such a client is 
> supposed to work *at all*. Let's start with a blank state. The 
> client knows nothing about the mailbox. It opens it and 
> presumably wants to show a list of messages to the user. What is 
> the client supposed to do now?

Something "uid search unseen", "uid search or unseen within 604800", or 
some variation.

Most clients, of course, do "fetch 1:* ..." instead, which is okay, but 
that approach means that this extension is not interesting for them.

> What if it wants to show just a subset of the mailbox, perhaps 
> last 1k items (actually it doesn't have to be 1k, anything 
> between 500 and 3000 is probably fine -- it needs *something*, 
> but not the full 4M mailbox). How to achieve that?

"Show today's mail" is trivial, "show all the unread mail" is trivial, 
"show the last five hundred messages" is difficult.

> What happens when a mail arrives?

Same as before. Think about it.

Before, you got an EXISTS for two new messages. But then someone expunges 
one before the server processes your fetch, and you get one message back. 
With UIDONLY, you get a UIDNEXT indicating that two new UIDs have been 
allocated. You issue a fetch, and get <=2 messages. There's no difference 
in principle.

> How is a resynchronization after being offline for a while 
> going to look like?

Same as before.

I'll send you a session transcript offlist. I may not get it done tonight, 
though.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 26 Apr 2014 08:42:07 +1000

> (and at the same time supporting simultaneous sessions) was simply too 

Same for me in Cyrus.  We recompute MSNs at SELECT time.  We used to keep
them just as offsets in the cyrus.index file and rewrite/rename on every
expunge, but that has its own pain points (every single expunge causes a
full index rewrite), and it meant that sessions working from on old index
file had outdated information.  CONDSTORE became totally bogus.

So now we just write a flag for EXPUNGED and repack occasionally.

> So you learned it twenty lines later ;)

Well, yeah - EXISTS on the other hand.  We can keep track of EXISTS in Cyrus
just fine without calculating MSNs, because we keep a counter in the index
so that STATUS EXISTS doesn't suck - because most clients request this on a
ton of mailboxes frequently.

I'm not sure why not having MSNs necessitates removing EXISTS.  Knowing how
many messages are in the folder is, as pointed out by others, a very common
thing in clients.  So much so that we added a ton of code to also keep that
figure easy to fetch when we merge messages into conversations.  Both EXISTS
and UNSEEN counts are used in our UI for every folder, for display and for
calculating the size of the scroll area in the mailbox view.

> The server does send BAD when it receives a BAD from the client.

I think the confusion here was things like

* 1 FETCH (UID 500 FLAGS (\Seen))
* 2 FETCH (UID 501 FLAGS ($Special))

... honestly though, now that I look at it.  Is there any reason not to
just use the UID everywhere that MSN appears?  I still really dislike the
use of magic numbers.  Either that or a string like NOMSN.  Anyone sending
ENABLE UIDONLY will already be aware that they need to change their code,
and the nice thing about it really not being a number is that the client
parser will either be correct or fail hard.

> Disagree violently. 5162 is MUCH too complex to require all of it.

I didn't find it so hard once CONDSTORE existed.  The hardest part is
keeping track of the low-water-mark so you know if you can respond with
full information or not.  We use a field called DELETEDMODSEQ for this.

But you don't need to do that.  You can do a no-information version of
it which always responds to a SELECT QRESYNC as if it's forgotten all
the history of the mailbox.  It's not too complex then.

> Why? What's wrong with UIDNEXT?

You still need to capture every create and expunge event on the entire
mailbox so you can send VANISHED responses.  At least in Cyrus, that's
not really architecturally easy in the case that you've slept for a bit
and the EXISTS count has changed by one but the UIDNEXT has jumped by 5.
You need to scan back through the earlier file looking for which other
message disappeared.

It got a lot easier with CONDSTORE, because you just query for every record
which changed since the MODSEQ of the last sync.  That's how our internals
work now - everything is built on MODSEQ tracking rather than positional
mathematics.

> not an accidental problem, it's necessary.

It's really not if you still support sending EXISTS counts.  I'm surprised
that they are expensive, while UIDs are being allocated.  Unlike MSNs, they
don't change between connections, so you can just keep a global cache per
mailbox.  Since you only allow UID commands, you can resync them immediately
rather than having to avoid telling clients about expunges for a bit.

> don't know (at least not precisely) how many you have. It's as simple as 

I thought the point was to avoid having to enumerate messages to give them
sequential IDs such that you can reference to any particular offset in the
set.  That's quite different from not counting at all.  If you implement
quotas, you're already updating a central value when any message is added
or removed.  Updating another central value at the same time doesn't strike
me as hard (in Cyrus we now actually have a quota on number of messages in
the mailbox as a settable field - along with a quota on number of sub-
mailboxes and quotas on number and size of annotations... all the resource
limits we used to hard-code on a per-user basis)

So I do have a problem with losing EXISTS.  It's purely the using MSNs as
a way of accessing a particular "distance" through the mailbox that I'd
like to lose.

> I can show you code that does that, OK?

Real examples are always good :)

> store +flags on the wrong message.

Agree here.  I just can't see many clients getting the benefits of this without
an EXISTS method.  Users do like being able to see how many messages they have
in the UI.

(aside: I'd love to see an RFC or errata saying that it's cool to call STATUS on
the currently selected mailbox, and you see the same counts that you would see if
you queried that data from the mailbox.  It seems to be what the good servers are
already doing - and it matches user expectations and client author expectations.
We take advantage of it with a LIST RETURN STATUS that gets all the folder counts
and the folder list in a single call for our web UI)

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Andrew Sutherland <[email-masked]>
Date: Fri, 25 Apr 2014 21:39:17 -0400

On 04/25/2014 01:28 PM, Jan Kundrát wrote:
>> The point of this is to stop counting messages. If you stop counting, 
>> you don't know (at least not precisely) how many you have. It's as 
>> simple as that.
>
> OK, at this point I cannot imagine how such a client is supposed to 
> work *at all*. Let's start with a blank state. The client knows 
> nothing about the mailbox. It opens it and presumably wants to show a 
> list of messages to the user. What is the client supposed to do now?
>
> items (actually it doesn't have to be 1k, anything between 500 and 
> 3000 is probably fine -- it needs *something*, but not the full 4M 
> mailbox). How to achieve that?

The Firefox OS Gaia e-mail client uses date-based synchronization. 
Although we optimize for small folders, in a big folder we start by 
doing a "SEARCH NOT DELETED SINCE (3 days ago)".  If we get too many 
back, we bisect our way down until we're talking only about a single 
day.  We sync the headers.  Then if we still want more messages, we move 
our time window back in time.  We use various heuristics to grow/shrink 
the time window as we go for efficiency.  We declare a folder entirely 
synchronized when we get to 1990.

This is a nice approach to clients that only sync a subset of what's on 
the server since while there is usually a correlation between UID and 
INTERNALDATE, thanks to message moves and APPEND, it's not a sure thing.

(Note: the awkwardness about SINCE/BEFORE and timezones and interactions 
with INTERNALDATE are not without their issues.  We have a longstanding 
embarassing bug about this at https://bugzil.la/886534 that still 
remains to be addressed. There's additional high-level details and 
rationale at 
https://wiki.mozilla.org/Gaia/Email/Implementation/MailSynchronization 
if anyone is interested.)

Andrew

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 28 Apr 2014 11:52:03 +0300

On 25.4.2014, at 15.30, Arnt Gulbrandsen <[email-masked]> wrote:

> as some of you will have discovered, opening large mailboxes takes real time. So I discussed that with Brandon and wrote a draft, really short, perhaps it's actually shorter than the boilerplate. The gist: "If a client runs ENABLE UIDONLY, then that client cannot later use any MSNs in any commands, and if syntax forces the server needs to send an MSN, then it just sends a phony number." There are a couple of details.

I don't really understand the main point of this. Is this only about reducing the amount of work for some server implementations? Clients won't get any direct benefits from this, right?

As for servers having to do MSN <-> UID mapping when mailbox is opened, that's not really true either. You can do it lazily only as needed. As Bron mentioned you can efficiently keep track of the total number of messages in mailbox. After that, clients will typically work in one of two ways:

a) Either they FETCH 1:* FLAGS, in which case you're pretty much building the MSN <-> UID mapping for free while returning the results, just with the cost of some memory.

b) They're only accessing somewhat recent messages, in which case you can only build the MSN <-> UID mapping for the recent messages and ignore the rest until they're needed (and they most likely won't be).

I've been planning on implementing b) for Dovecot, but haven't yet. So maybe there is something in it that I haven't thought about that would cause trouble, but currently the main problem I see with it is that it would require some extra development. (Maybe SORT+PARTIAL could be problematic.)

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 28 Apr 2014 11:08:26 +0200

> I'm not sure why not having MSNs necessitates removing EXISTS.

It depends on server implementation. If the server computes EXISTS from the 
MSN-UID map, then dropping that map and keeping EXISTS is real work.

> Knowing how
> thing in clients.

Yes, just like breaking horribly on large mailboxes.

I posit that when the process of optimising for large mailbox access 
involves removing EXISTS, and that therefore, it's not worth optimising 
_this_ extension for clients that display EXISTS. Clients that really want 
it can always send STATUS EXISTS.

AFAICS, the clients I've tried that work well with a large mailbox do not 
display EXISTS. They don't display the mailbox at all, but rather some 
subset, such as "the unseen and recent mail". Mailbox for iphone is an 
example. If there is any total message count anywhere in the UI I haven't 
found it.

I'd be happy to make EXISTS optional and say that the server must allow 
issuing STATUS on the selected mailbox.

> parser will either be correct or fail hard.

You mean like this?

 * 500 FETCH (UID 500 FLAGS (\Seen))

Compared to using a constant for the MSN, this has the advantage that MSN 
math almost continues to work, and the disadvantage that MSN math almost 
continues to work.

Anyone who does "uid fetch x-25, x" will see the code continue to work 
provided the UIDs are mostly contiguous, which is often the case for large 
mailboxes. So do you prefer to flush out bugs quickly or allow buggy code 
to work after a fashion? My preference is to flush out bugs, but I do not 
feel strongly about this issue.

> the history of the mailbox.  It's not too complex then.

Ask me what I think about such shams sometime.

> You need to scan back through the earlier file looking for which other

I don't see what you're trying to say here.

I can see that in your case, it's as easy to generated ARRIVED+VANISHED, 
but what does that have to do with whether clients need ARRIVED?

> work now - everything is built on MODSEQ tracking rather than positional

Yes, and?

>> The point of this is to stop counting messages. If you stop counting, 
you 
> set.  That's quite different from not counting at all.

Strictly speaking, the point is to avoid O(n) behaviour in SELECT.

I understand now that it's possible to do the counting at other times and 
thereby keep EXISTS while speeding up SELECT. But as I said above, I don't 
think the need for displaying mailbox size warrants doing that. (The cost 
is zero if you've already done the work, of course.)

> a way of accessing a particular "distance" through the mailbox that I'd

Well, the client can issue STATUS (EXISTS) if it wants...

> Real examples are always good :)

I can't post that code, it's much too large for the list. But I'll add a 
commented session transcript as an appendix to the -01 draft.

> an EXISTS method.  Users do like being able to see how many 

I've seen many complaints about gmail, but not that you can't see how many 
messages you have, so I suspect that users' desire to see that count 
disappears when the number grows incomprehensibly large.

> and the folder list in a single call for our web UI)

I'd be happy to add such a requirement here.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 28 Apr 2014 11:17:16 +0200

On Monday, April 28, 2014 10:52:03 AM CEST, Timo Sirainen wrote:
> I don't really understand the main point of this. Is this only 
> about reducing the amount of work for some server 
> implementations? Clients won't get any direct benefits from 
> this, right?

The ability to open 10M mailboxes fast. Is that an advantage for the 
client, or merely for the user?

> As for servers having to do MSN <-> UID mapping when mailbox is 
> opened, that's not really true either. You can do it lazily only 
> as needed. As Bron mentioned you can efficiently keep track of 
> the total number of messages in mailbox. After that, clients 
> will typically work in one of two ways:
>
> a) Either they FETCH 1:* FLAGS, in which case you're pretty 
> much building the MSN <-> UID mapping for free while returning 
> the results, just with the cost of some memory.

This one is irrelevant. A client that does FETCH 1:* won't bother about a 
fast SELECT.

> b) They're only accessing somewhat recent messages, in which 
> case you can only build the MSN <-> UID mapping for the recent 
> messages and ignore the rest until they're needed (and they most 
> likely won't be).

I can't tell how do do this.

Assume you have about two million messages in a mailbox, and you're about 
to send an unsolicited flag response for one that's neither the first or 
the last. How do you do it?

> I've been planning on implementing b) for Dovecot, but haven't 
> yet. So maybe there is something in it that I haven't thought 
> about that would cause trouble, but currently the main problem I 
> see with it is that it would require some extra development. 
> (Maybe SORT+PARTIAL could be problematic.)

Abhijit and I tried it and gave up after half a day. So yes, maybe it's 
possible, but after running into a few snags we gave up, and neither of us 
is stupid.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 28 Apr 2014 11:20:12 +0200

> I can't tell how do do this.

Oh, I see. You keep the total count persistently, and build the MSN-UID map 
in memory downwards. Right?

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 28 Apr 2014 12:35:09 +0300

On 28.4.2014, at 12.17, Arnt Gulbrandsen <[email-masked]> wrote:

> Assume you have about two million messages in a mailbox, and you're about to send an unsolicited flag response for one that's neither the first or the last. How do you do it?

Here's a relatively easy way: Have two index mappings, one containing "old" messages and one containing "new" messages (every once in a while moving some messages from new to old). For both you know the first and the last UID and also the message count. As long as client is accessing only the messages in the "new" index, you'll never need to load the "old" index into memory or calculate its MSN <-> UID mapping, only the "new" MSN <-> UID mapping. So the "new" mapping would usually be somewhere between 100-10000 mails, which would likely contain all the mails that most people/clients will access.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 28 Apr 2014 20:34:23 +1000

> MSN-UID map, then dropping that map and keeping EXISTS is real work.

True - as with anything, it's up to the server how it tracks things.  Of course,
if it does that, then STATUS is going to be a really expensive command.  Lots of
clients call STATUS too.  You're shooting yourself in the foot not keeping an
exists count unless you're really sure exactly who your clients are.

> Ask me what I think about such shams sometime.

Honestly, you have to suck that up at some point, unless you never forget
anything ever.  Hence the use of DELETEDMODSEQ in Cyrus.

> I don't see what you're trying to say here.

I'm saying that in the Cyrus case, just calculating which messages were
VANISHED (if you didn't do it yourself) is O(n).  Once you start optimising
your standard for one particular implementation, you wind up with weirdness
like not requiring EXISTS.  I'd much rather not require VANISHED, particularly
since you don't know if the client knows about this UID, so you can wind up
sending a lot of VANISHED information that the client doesn't care about.

> Strictly speaking, the point is to avoid O(n) behaviour in SELECT.

Which you only have for EXISTS if you don't keep a count of how many items
are in your mailbox.  As I've already said before, you already need to do
housekeeping when you receive or delete messages, otherwise you can't do
quotas.  You haven't answered that.

> is zero if you've already done the work, of course.)

The cost of just about anything is zero if you've done the work.  The problem
with MSNs is that they are explicitly per connection, so you have no choice
but to do the O(n) work for every connection.  Even there, there are tricks
you could probably do with with clever data structures to reduce the cost to
O(number of disjoin expunges).

> disappears when the number grows incomprehensibly large.

Could be.  Could be that you don't get the same complaints as other people.

http://webapps.stackexchange.com/questions/6262/is-there-a-way-to-display-the-number-of-messages-for-any-label-in-the-gmail-labe

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 28 Apr 2014 13:33:45 +0200

> exists count unless you're really sure exactly who your clients are.

O(n) is okay as long as n is small ;) I mean, if a client has ha habit of 
doing FETCH 1:* and STATUS (EXISTS), the STATUS will always be less 
painful.

The problem only happens when clients that are optimised for small 
mailboxes and used with large ones. Fixing that require client change: 
Cache well, don't retrieve what you don't need, don't use MSNs, don't ever 
look at the entire mailbox, etc.

Btw, the SE link you posted is like ESEARCH (RETURN COUNT) KEYWORD TODO in 
IMAP terms, it's not like STATUS/EXISTS. I agree that the size of such 
subsets is something people want to know. You may notice that all of the 
numbers in the question's example are small. People do care about numbers 
as long as they're within human range.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 28 Apr 2014 22:00:34 +1000

> doing FETCH 1:* and STATUS (EXISTS), the STATUS will always be less 

We work on a theory of 1 million emails in a mailbox... that's the sizing
data we used 5 years ago when we were deciding what was reasonable
performance.  Might need to revise that upwards in a couple of years.

> look at the entire mailbox, etc.

Yeah - though I have to say Timo's "count down" idea is pretty nice.

> as long as they're within human range.

So I'm going to throw this discussion over at Neil as well, our frontend
developer, and see what we can do about making our interface happy about
a 4 million+ message folder (see http://jmap.io/ - working on improving
that and making our server actually talk it is next on our list after the
calendaring features are done).

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Jan Kundrát <[email-masked]>
Date: Mon, 28 Apr 2014 18:18:15 +0200

> a) Not have a number after the leading asterisk.

Yes, that's the option which I much prefer over using made-up numbers.

> "There are <=5 new messages. If you want them, you can send uid 
> fetch 12345:* and get the details you choose you want about all 
> the new messages".

Fair enough. With <=5 new messages, a client could very well just request 
all the ENVELOPEs and INTERNALDATEs and other informaiton it needs, and it 
wouldn't be a big deal. However, what if the UIDNEXT jumped by some "big 
enough" number -- say, a 1000? Unlike a change in EXISTS, the client 
doesn't know what really happened. It only knows an upper bound on the 
number of new arrivals. It could still mean that there's no new arrival at 
all, or that a thousand of new messages arrived. It would be unwise to ask 
for a thousand of envelopes at once.

> Oh? I once asked a few colleagues at a googleapps shop about 
> that, and none of them had any idea about their total number of 
> messages. "Oh, lots", "I've never deleted anything", that was 
> the sort of answer I got. None told me any number. One looked at 
> his screen and told me how many per cent of his quota he used.

I wonder whether they don't know because they don't care, or whether it's 
because because their only UI doesn't offer that information, and thus they 
become trained not to care about that. GMail's UI doesn't offer a way of 
sorting the messages, yet people still do stuff like sorting by recipient 
or by mail size. "Not supported by GMail" doesn't equal "not useful".

> I can tell you why I didn't add anything like that in 5530. 
> 5530, as you may have noticed, only lists codes, it doesn't say 
> that you have to use any code. If two codes apply, it doesn't 
> say which one you must choose.
>
> That's because when I surveyed servers for 5530, I found a few 
> instances where two response codes applied. Two errors occured, 
> and it wasn't clear that one response code ought to win over the 
> other. So I decided to not issue a rule on that. A server can 
> let the first response code win, the last one win, it doesn't 
> matter.
>
> I think that also applies in this case. Suppose there's a 
> command such as "uid search charset iso-8859-42 1 body foo". The 
> charset specification is reasonable, but the 8859 committee 
> hasn't issued number 42 yet, so that causes one response code. 
> The presence of the MSN causes another. The syntax only permits 
> one, so the server has to choose which one to send. Do you think 
> that in this case, the MSN-related response code ought to win 
> over the charset-related code? Do you think it ought to win over 
> EVERY other code in ALL cases?
>
> I think it doesn't matter very much, so let the server do 
> whatever comes easiest.

I think that "using a command which is explicitly forbidden in this 
situation" should indeed trump any other error state, yes. If only because 
it's trivial to check this condition, and also because it's an easy way of 
telling the client developer that they screwed up -- something which I find 
very valuable as a client developer.

>> presumably wants to show a list of messages to the user. What 
>> is the client supposed to do now?
>
> Something "uid search unseen", "uid search or unseen within 
> 604800", or some variation.
>
> Most clients, of course, do "fetch 1:* ..." instead, which is 
> okay, but that approach means that this extension is not 
> interesting for them.

As a (desktop) client developer, I want to show my user what e-mails she 
has in her mailbox(es). The UI framework I use needs to know the amount of 
items to be shown in a ListView/TreeView, otherwise I have to resort to 
ugly bits like resizing the scrollbar handle when she scrolls too close to 
the current boundary.

So OK, I have an empty state, know nothing about the number of messages in 
a mailbox. I just know that my viewport measures 832px, and my code can 
calculate that I can fit 40 messages in there (these numbers are made-up). 
I have no idea what messages are these, so I can make a rough guess that a 
week of messages might approximate what I need to show on the screen, plus 
some extra room for the scrollbar. So I request them:

  a01 UID SEARCH SINCE ...

Three possibilities might happen:

- I got a reasonable number of messages, say, 50 - 500. In this case 
everything is good; I can show a list of messages to the user.

- I got 10k UIDs back because the user opened a LKML archive. Oops. Anyway, 
at least I don't have to go asking for more UIDs when they scroll back.

- I got just two UIDs back, perhaps because it's a low traffic mailbox. 
This means that I can only show two mails on my big screen. That's wrong -- 
just the week ago the mailbox might have had 200 messages delivered in 
there. I have to expand the window I sync, which means that the client has 
to wait for multiple network round trips. I might save some bandwidth, but 
I pay a rather big cost in latency.

The client will probably want to do some extra preload, if only to know 
whether to include some platform-specific UI equivalent of "load more 
messages" -- be it either a button, or something which actives upon show, 
or just some scrollbar indicator.

> "Show today's mail" is trivial, "show all the unread mail" is 
> trivial, "show the last five hundred messages" is difficult.

And that's exactly the point -- taking away EXISTS and MSNs makes it 
impossible to page through the result set based on the number of messages. 
The ability of UIs to show stuff to the user depens on the number of items 
to be shown, not on the time interval covered by the N items, IMHO.

> Before, you got an EXISTS for two new messages. But then 
> someone expunges one before the server processes your fetch, and 
> you get one message back. With UIDONLY, you get a UIDNEXT 
> indicating that two new UIDs have been allocated. You issue a 
> fetch, and get <=2 messages. There's no difference in principle.

Apart to what I said at the beginning of the mail, this change also takes 
away the possibility to double-check that both sides are synced. 
Previously, a change of EXISTS without an appropriate EXPUNGE means that 
someone screwed up, and it is easy to detect this and give up. When you 
remove EXISTS, the notification changes to "Some number of new messages 
might have arrived. The is no more than N new messages.", which is a step 
backwards IMHO. (Yes, I'm still counting messages in my client and in my 
mental model.)

> I'll send you a session transcript offlist. I may not get it 
> done tonight, though.

I think it should be a part of the RFC. I don't think I would be the only 
one asking these questions.

With kind regards,
Jan

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 28 Apr 2014 23:58:49 +0000

I'm finding it hard to get my head around 1M messages in a mailbox.

This obviously isn't a person's mailbox, unless they have their email 
address on every spammer's list and haven't deleted a message (including 
spam) since 1967.

Why mess with the spec to handle a use-case that is meaningless?

Unless this is some kind of use-IMAP-as-a-database-for-some-app kind of 
scenario?

------ Original Message ------
From: "Bron Gondwana" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 29/04/2014 12:00:34 a.m.
Subject: Re: [imapext] a very simple little imap extension, really quite 
magnificently simple

>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Brandon Long <[email-masked]>
Date: Mon, 28 Apr 2014 17:08:15 -0700

We have many many mailboxes over 1M, and I'm sure that's partially due to
our "virtual" folders and the fact that there exists the "all mail" folder
which is everything in the users entire account.

In fact, we have accounts in the 10-50M range, though yes, the likelihood
of those being "useful" accounts is arguable.  They range from legal mail
dumps to source accounts for support systems to who knows what.

My work account here at google has 1.6M messages in it.

And such accounts are just growing in size.

Brandon

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 29 Apr 2014 00:12:05 +0000

maybe it's time for some quota enforcement or forced archiving.

How long would it take a person to make head or tail of 1M messages.  
E.g. even only to decide what to keep or not.

It is easier to just keep everything just in case, but there has to be 
some sanity somewhere...

------ Original Message ------
From: "Brandon Long" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Bron Gondwana" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 29/04/2014 12:08:15 p.m.
Subject: Re: [imapext] a very simple little imap extension, really quite 
magnificently simple

>We have many many mailboxes over 1M, and I'm sure that's partially due 
>to our "virtual" folders and the fact that there exists the "all mail" 
>folder which is everything in the users entire account.
>
>In fact, we have accounts in the 10-50M range, though yes, the 
>likelihood of those being "useful" accounts is arguable.  They range 
>from legal mail dumps to source accounts for support systems to who 
>knows what.
>
>>address on every spammer's list and haven't deleted a message 
>>(including spam) since 1967.
>>
>>Unless this is some kind of use-IMAP-as-a-database-for-some-app kind 
>>of scenario?
>>
>>
>>
>>
>>Subject: Re: [imapext] a very simple little imap extension, really 
>>quite magnificently simple
>>

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 29 Apr 2014 10:18:42 +1000

With good search (something that Google has) you don't need to make
head or tail of everything.  You can search for what you need when you
need it.  The use-case makes sense to me.

Now - personally I archive messages per-year, so each folder contains
fewer than that.  With cross-folder searching, it doesn't cost any more
to find messages, but I can page through each folder a little faster.

(I also filter each mailing list into a separate folder though... so I
think I'm some kind of weird dinosaur)

Bron.

On Tue, Apr 29, 2014, at 10:12 AM, Adrien de Croy wrote:

------ Original Message ------

From: "Brandon Long" <[1][email-masked]>

To: "Adrien de Croy" <[2][email-masked]>

Cc: "Bron Gondwana" <[3][email-masked]>; "[email-masked]"
<[4][email-masked]>

Brandon

On Mon, Apr 28, 2014 at 4:58 PM, Adrien de Croy <[5][email-masked]>
wrote:

------ Original Message ------
From: "Bron Gondwana" <[6][email-masked]>
To: "[7][email-masked]" <[8][email-masked]>
Sent: 29/04/2014 12:00:34 a.m.
Subject: Re: [imapext] a very simple little imap extension, really

 On Monday, April 28, 2014 12:34:23 PM CEST, Bron Gondwana wrote:

 > True - as with anything, it's up to the server how it tracks

 > things. Of course,

 > if it does that, then STATUS is going to be a really expensive

 > command. Lots of

 > clients call STATUS too. You're shooting yourself in the foot

 > not keeping an

 > exists count unless you're really sure exactly who your clients are.

 O(n) is okay as long as n is small ;) I mean, if a client has ha habit
of

 painful.

We work on a theory of 1 million emails in a mailbox... that's the
sizing

 Cache well, don't retrieve what you don't need, don't use MSNs, don't
ever

 Btw, the SE link you posted is like ESEARCH (RETURN COUNT) KEYWORD
TODO in

 subsets is something people want to know. You may notice that all of
the

 numbers in the question's example are small. People do care about
numbers

So I'm going to throw this discussion over at Neil as well, our
frontend

developer, and see what we can do about making our interface happy
about

a 4 million+ message folder (see [9]http://jmap.io/ - working on
improving

that and making our server actually talk it is next on our list after
the

Bron.

--

  Bron Gondwana

  [10][email-masked]

imapext mailing list

[11][email-masked]

[12]https://www.ietf.org/mailman/listinfo/imapext

_______________________________________________
imapext mailing list
[13][email-masked]
[14]https://www.ietf.org/mailman/listinfo/imapext

imapext mailing list

[15][email-masked]

[16]https://www.ietf.org/mailman/listinfo/imapext

--
Bron Gondwana
[email-masked]

References

1. mailto:[email-masked]
2. mailto:[email-masked]
3. mailto:[email-masked]
4. mailto:[email-masked]
5. mailto:[email-masked]
6. mailto:[email-masked]
7. mailto:[email-masked]
8. mailto:[email-masked]
9. http://jmap.io/
  10. mailto:[email-masked]
  11. mailto:[email-masked]
  12. https://www.ietf.org/mailman/listinfo/imapext
  13. mailto:[email-masked]
  14. https://www.ietf.org/mailman/listinfo/imapext
  15. mailto:[email-masked]
  16. https://www.ietf.org/mailman/listinfo/imapext

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 28 Apr 2014 17:19:54 -0700

> And such accounts are just growing in size.

I have to ask: do you store all your computer files in a single directory?

If not, why not.

--lyndon

P.S.  I'm not asking rhetorically.

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 29 Apr 2014 11:16:05 +1000

> or by mail size. "Not supported by GMail" doesn't equal "not useful".

Yes indeed :)

> or just some scrollbar indicator.

This is exactly what we do with our web interface, and it's why we implemented
something similar to what's in the JMAP spec:

http://www.jmap.io/#getmessagelist

Which does offset and limit - so you can actually get a range within a search/sort
rather than only a range by MSN, which isn't very useful if you want your messages
sorted as well (particularly in a folder which has had messages moved in from
various locations, so the sort order isn't consistent with the UID/MSNs)

You'll note that our own API also includes a way to say "give me N messages in a
window around this UID within the sorted output" - obviously if you're half way
down the list, you need to materialise it at least enough to calculate the offset
of that named messageid - and if it's missing, that means loading the entire
mailbox.  That's pretty rare though - only happens if you're paging through at
the same time as you expunge a message.

I think this is a much more useful abstraction than MSNs, because MSNs don't give
you any sorting ability.  On the negative side, it means that messages can both
appear and disappear further back in the list - unlike MSNs where new messages only
ever appear at the top.  On the positive side, we support searches across multiple
folders with full sorting and searching and limiting, causing low latency and low
bandwidth use to get exactly what the user wants onto their screens.

> to be shown, not on the time interval covered by the N items, IMHO.

The last 500 messages is less interesting if they can only be the last 500
messages appended to this mailbox, regardless of moves between mailboxes,
out-of-order delivery, etc.  In theory it's mostly OK, but you can miss
messages that way if you do something like copy the entire contents of one
folder into another in batches and they're out of order.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Brandon Long <[email-masked]>
Date: Mon, 28 Apr 2014 22:04:34 -0700

No, I don't store all of my files in a single directory, but neither is my
mail.

I have probably a dozen labels, but one of those labels is "all mail",
which all Gmail accounts have, and that's exposed as a folder.  And that is
becoming more commonly used by some IMAP clients, since it allows for a
more "Gmail-like" experience to treat the mailbox as a single bag of
messages with different labels (keywords) on the messages.

But really, folders are rarely that useful when one has this much mail.  If
you want a file-system example, the Google codebase is basically a single
repository with a deep hierarchical structure.. and I use our search index
over that data more often than trying to find something in the structure.

I'm unclear what you're really asking, however.  Yes, Gmail has several
design decisions which mean that we may be hitting these limits earlier
than others, and I certainly don't want to force IMAP to change just to
support us.  But I think that the current MSN design implies an O(n) design
for IMAP, and n is just going to keep increasing for everyone.

I think that this is a useful discussion for that reason.

Do we want to argue whether or not users should have that many messages?
 That seems like a losing argument, users do what users do.  Is it really
worth users' effort to decide what to keep from 10 years ago?  That seems
like a waste of effort on their part, so its not a surprise that they
choose not to.

There are clearly other options available to us, but I think this is a
useful thought exercise to see if it would be possible to just remove the
impediment entirely, instead of trying to work around it.

Brandon

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 29 Apr 2014 09:43:17 +0200

I may not feel strongly about the number. But I do feel strongly about this 
mode of decision-making:

> Yes, that's the option which I much prefer over using made-up numbers.

"I prefer". Not even the shadow of a rationale.

Can we please try to use arguments?

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 29 Apr 2014 08:00:12 -0700

Hi Arnt,

--On April 25, 2014 at 7:07:30 PM +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

> I don't see any advantage in 1. The strange number is recognizable; if
> you see that foo == 8214 in the debugger you have a fair chance of
> remembering that you recently implemented RFC8214 and realise that the
> problem you're chasing is related to that. Ditto google for "imap 8214".

Since the use of this extension is an explicit agreement between client and 
server, why not modify the sequence syntax to allow zero for this one 
specific case (or some other indicator like ".")? Then we can stop arguing 
about what special number to use.

-- 
Cyrus Daboo

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 29 Apr 2014 18:03:50 +0200

> Can we please try to use arguments?

The main argument is clarity, of course. Using a made-up number (or a 
random one, a magic constant, or whatever which happens to be a number) can 
be interpretted as an actual number, and will lead to confusion. I cannot 
explain this better, sorry -- to me, this is completely intuitive. Changing 
the syntax to use anything but a number prevents accidental 
misinterpretation. I'm surprised that I'm asked to provide a rationale for 
that.

What is your rationale for using a made-up numeric constant instead of 
using something like "NOMSN", or even just "* FETCH"?

With kind regards,
Jan

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 29 Apr 2014 18:59:12 +0200

On Tuesday, April 29, 2014 5:00:12 PM CEST, Cyrus Daboo wrote:
> Since the use of this extension is an explicit agreement 
> between client and server, why not modify the sequence syntax to 
> allow zero for this one specific case (or some other indicator 
> like ".")? Then we can stop arguing about what special number to 
> use.

Now that you say it, why have a rule at all? The client is not permitted to 
care about the number, so telling the server what to send is superfluous.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 29 Apr 2014 10:49:35 -0700

Hi Arnt,

--On April 29, 2014 at 6:59:12 PM +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

> Now that you say it, why have a rule at all? The client is not permitted
> to care about the number, so telling the server what to send is
> superfluous.

Well just from a debugging standpoint where you may only have been given a 
partial view of what is happening, being able to recognise that this state 
is present or not would be handy.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 29 Apr 2014 22:21:12 +0200

On Tuesday, April 29, 2014 6:03:50 PM CEST, Jan Kundrát wrote:
> The main argument is clarity, of course. Using a made-up number 
> (or a random one, a magic constant, or whatever which happens to 
> be a number) can be interpretted as an actual number, and will 
> lead to confusion. I cannot explain this better, sorry -- to me, 
> this is completely intuitive. Changing the syntax to use 
> anything but a number prevents accidental misinterpretation. I'm 
> surprised that I'm asked to provide a rationale for that.
>
> What is your rationale for using a made-up numeric constant 
> instead of using something like "NOMSN", or even just "* FETCH"?

Simplicity. Ease of implementation. Testability.

This extension is an optimisation. For this reason, it's reasonable to 
expect that clients do not change their user interface based on its 
presence. So if you want to implement this without making a (partly) 
separate user interface, you parse and ignore MSNs sent by unextended 
servers.

So, since clients then know to parse and ignore, I want to leverage that 
ability. This means that supporting the extension requires zero changes to 
the client parser, and zero extra unit tests.

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 29 Apr 2014 23:58:22 +0200

On Tuesday, 29 April 2014 22:21:12 CEST, Arnt Gulbrandsen wrote:
> This extension is an optimisation. For this reason, it's 
> reasonable to expect that clients do not change their user 
> interface based on its presence. So if you want to implement 
> this without making a (partly) separate user interface, you 
> parse and ignore MSNs sent by unextended servers.

Wait -- what has the user interface to do with the way how MSNs are parsed 
or handled? There is plenty of MUAs which can show POP3, IMAP4 and local 
mailbox/maildir/... accounts using the same UI. Surely a change not to use 
MSNs does not require any change to the UI.

> So, since clients then know to parse and ignore, I want to 
> leverage that ability. This means that supporting the extension 
> requires zero changes to the client parser, and zero extra unit 
> tests.

You probably mean zero additional unit tests for the parser, but definitely 
some additional unit tests for that part of code which speaks IMAP and 
models the remote server's state. That's true.

On the other hand, you now have a protocol which contains superfluous 
tokens with no useful value; in fact, these tokens look like something 
which is well known to a client developer, yet actually forbidden to be 
used in this situation. I consider this to be a very confusing feature of a 
protocol. Based on the reactions on this ML, anybody who commented except 
you agrees with that point of view.

> Simplicity.

I'll grant you that if you refer to the fact that the IMAP parser doesn't 
have to be changed. That's true. It's also true that the amount of changes 
needed in the other parts of a client is at least one order of magnitude 
higher.

> Ease of implementation.

No matter which of these two responses formats were used, both would 
require roughly the same amount of work in my client.

I'm willing to believe that there's at least one client out there where 
your proposal would be simpler to implement. However, having made-up data 
in a response is such an antipattern in my opinion that it completely 
trumps this possible benefit.

> Testability.

The above is why I pretty violently disagree with "testability" being one 
of the reasons in favor of retaining these fake MSNs. For the reasons 
stated above, I actually consider the inclusion of fake MSNs to actively 
harm testability. You have the same responses on the wire which now require 
completely different processing. Something which previously mandated a 
lookup based on the MSN must now be handled by the now-made-mandatory UID, 
and the MSN must be suddenly ignored. In Trojita, this would mean a change 
in the way FETCH responses are handled (after they are already parsed), and 
I surely would add multiple test cases for that.

With kind regards,
Jan

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 30 Apr 2014 00:17:50 +0000

why even send the number then if it must be discarded?

Adrien

------ Original Message ------
From: "Arnt Gulbrandsen" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 30/04/2014 4:59:12 a.m.
Subject: Re: [imapext] a very simple little imap extension, really quite 
magnificently simple

>>Since the use of this extension is an explicit agreement between 
>>client and server, why not modify the sequence syntax to allow zero 
>>for this one specific case (or some other indicator like ".")? Then we 
>>can stop arguing about what special number to use.
>
>Now that you say it, why have a rule at all? The client is not 
>permitted to care about the number, so telling the server what to send 
>is superfluous.
>
>Arnt
>

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 29 Apr 2014 22:48:38 -0600

Quoting Arnt Gulbrandsen <[email-masked]>:

>> The main argument is clarity, of course. Using a made-up number (or  
>> a random one, a magic constant, or whatever which happens to be a  
>> number) can be interpretted as an actual number, and will lead to  
>> confusion. I cannot explain this better, sorry -- to me, this is  
>> completely intuitive. Changing the syntax to use anything but a  
>> number prevents accidental misinterpretation. I'm surprised that  
>> I'm asked to provide a rationale for that.
>>
>> What is your rationale for using a made-up numeric constant instead  
>> of using something like "NOMSN", or even just "* FETCH"?
>
> This extension is an optimisation. For this reason, it's reasonable  
> to expect that clients do not change their user interface based on  
> its presence. So if you want to implement this without making a  
> (partly) separate user interface, you parse and ignore MSNs sent by  
> unextended servers.
>
> So, since clients then know to parse and ignore, I want to leverage  
> that ability. This means that supporting the extension requires zero  
> changes to the client parser, and zero extra unit tests.

I agree with this.  Especially since, as written, the RFC is focused  
on server optimization not client optimization.  A client that is  
optimized to not use MSNs is already configured to ignore these values  
anyway, so you're not asking anything of client authors.

I don't see the need for yet another FETCH response, when the desire  
is not to add to the return some piece of data that was available  
before but instead to mask what already exists.  That's akin to  
creating an extension that identifies which ENVELOPE fields to return.

Keeps documentation nice and tidy also, with a single trivial ABNF  
change (adding the extension name).

'1' should be used instead of "RFC Value" for the placeholder.  (Would  
be nice to use something like 0, but that would require modification  
of ABNF which, as I stated above, doesn't seem useful.)  I see no  
discernable difference in ease of debugging for something like this:

* 1 FETCH (UID 100 [...])
* 1 FETCH (UID 101 [...])
* 1 FETCH (UID 102 [...])

versus:

* 8000 FETCH (UID 100 [...])
* 8000 FETCH (UID 101 [...])
* 8000 FETCH (UID 102 [...])

I would assume a developer isn't going to necessarily remember that  
they implemented "RFC 8000" 6 months after the fact, so seeing "8000"  
is much less important than the fact they are seeing duplicate  
MSN<->UID mappings.

Might as well use a single-digit number to reduce network size as much  
as possible.  Depending on buffering and # of messages returned, might  
save a packet or two of data.  '1' is as arbitrary as any number.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 29 Apr 2014 23:00:34 -0600

> like a waste of effort on their part, so its not a surprise that they

I'm not even seeing why the # of messages in a mailbox is terribly  
relevant for this discussion.

If you have 10,000 users with 100 messages each, I would guess you are  
still going to get a slight performance advantage from indicating to  
the server you don't need MSNs.  So it's a huge win for giant  
mailboxes and a small win for smaller mailboxes.  Yay!

Worst case scenario: if a server thinks it is more trouble than its  
worth, it can continue to send MSNs as it currently does and the  
client will continue to ignore those MSNs as it currently does and we  
are at the status quo, with the exception of a single 'ENABLE NOUID'  
command.

Maybe that answers the "what MSN to return" question - a server should  
be able to send whatever it wants as a MSN.  Maybe it is too difficult  
in a server's code to change the MSN to a certain defined value  
without major refactoring, so it should feel free to send whatever it  
wants in the interim although recommended goal is to send the smallest  
payload possible.

michael

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 29 Apr 2014 23:03:32 -0600

Quoting Michael M Slusarz <[email-masked]>:

> client will continue to ignore those MSNs as it currently does and  
> we are at the status quo, with the exception of a single 'ENABLE  
> NOUID' command.

"ENABLE NOUID" should obviously be "ENABLE NOMSN".

michael

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 30 Apr 2014 09:25:21 +0200

On Wednesday, April 30, 2014 7:00:34 AM CEST, Michael M Slusarz wrote:
> Maybe that answers the "what MSN to return" question - a server 
> should be able to send whatever it wants as a MSN.

I changed to this, for a different reason. Suppose the document said "the 
server MUST send blah" and a server actually sends something different. 
Then nothing bad happens. Therefore, the MUST is pointless, so I deleted 
it.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 30 Apr 2014 09:33:28 +0200

> why even send the number then if it must be discarded?

I answered this in the draft itself and twice so far on the list, but here 
goes again:

An implementing client has to parse-and-ignore if it wants to work with 
unextended servers and present the same UI independent of whether the 
server offers UIDONLY. Therefore, if extended servers send nonsense 
numbers, the client needs do _nothing_ conditionally on whether the server 
supports UIDONLY. The parser layer requires no change, there isn't a single 
if(serverHas("uidonly")) anywhere, and this extension requires no extra 
unit tests.

Arnt

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 30 Apr 2014 09:34:04 +0000

I don't think that's a good reason.

If we adopted that for all extensions where would we end up?  Isn't it 
just laziness on the part of the coder?  We leave the MSN in the 
messages and it becomes garbage on the wire.  Since when do we start 
deliberately sending garbage as part of protocol?

There are plenty of extensions which modify parsing.   Imagine if we'd 
taken this approach?  Half the bits on the wire might be garbage.  Or 
maybe we never really wanted to strip something out before so this is 
the first time for this.

------ Original Message ------
From: "Arnt Gulbrandsen" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 30/04/2014 7:33:28 p.m.
Subject: Re: [imapext] a very simple little imap extension, really quite 
magnificently simple

>I answered this in the draft itself and twice so far on the list, but 
>here goes again:
>
>numbers, the client needs do _nothing_ conditionally on whether the 
>server supports UIDONLY. The parser layer requires no change, there 
>isn't a single if(serverHas("uidonly")) anywhere, and this extension 
>requires no extra unit tests.
>
>Arnt
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 30 Apr 2014 11:48:10 +0200

On Wednesday, April 30, 2014 11:34:04 AM CEST, Adrien de Croy wrote:
> If we adopted that for all extensions where would we end up?  
> Isn't it just laziness on the part of the coder?  We leave the 
> MSN in the messages and it becomes garbage on the wire.  Since 
> when do we start deliberately sending garbage as part of 
> protocol?

Decades ago?

Most IMAP servers send a capability list before the one that matters when 
the client uses starttls. Many email messages include a timezone name in 
the date header field as well as the numeric timezone, decades after the 
change to numbers. Many IMAP clients send STATUS commands so often that 
most of them return the exact same information as before. And so on. There 
are lots of little examples of waste.

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Wed, 30 Apr 2014 12:26:04 +0200

On Wednesday, 30 April 2014 11:48:10 CEST, Arnt Gulbrandsen wrote:
> Most IMAP servers send a capability list before the one that 
> matters when the client uses starttls.

The reason for that is that the server might not know whether the client is 
going to issue STARTTLS. In this case, it's an optimization which saves one 
round trip for a client which needs CAPABILITY, but doesn't want STARTTLS. 
It's also needed to see whether the server actually speaks IMAP4rev1.

Unlike this proposal, the CAPABILITY does not contain conflicting 
information. Duplicate != conflicting.

> Many email messages 
> include a timezone name in the date header field as well as the 
> numeric timezone, decades after the change to numbers.

The syntax does not mandate sending the TZ name, so it is a different 
situation. For example. the message you sent includes just the numeric 
offset.

Also note that the TZ name and the numeric offset usually match. There was 
no mandate to use "UTC" and add a numeric offset of -0545 in there.

> Many IMAP 
> clients send STATUS commands so often that most of them return 
> the exact same information as before.

Which is because many IMAP servers do not support NOTIFY. The returned 
information is indeed a repetition of what was sent before, but the client 
cannot know that in advance, and there is no conflicting information in 
what is sent.

Cheers,
Jan